Your AI Agent Isn't the Bottleneck. Your Scheduler Is.

We run thousands of medical charts through agentic coding workflows every day. Each chart belongs to one of dozens of organizations sharing the same pipeline, and processing it kicks off a workflow that runs anywhere from a few minutes to a few hours. A single org runs several different workflows, and they cost wildly different amounts of compute, so the scheduler's whole job is to decide, fairly across all those orgs, which chart runs next.

The model doing the actual coding is its own deep, hard problem. This post is about a different layer: scheduling. Because even with a perfect agent, what decides whether the system stays up is what runs next, for whom, at what cost, and how you watch it. That turned out to be the hard part, and it is the part nobody writes about.

The problem, in one example

You have finite capacity. Say you can run some number of workflows at once. Work keeps arriving, faster than it drains. So every time a slot frees up, you have to answer one question: what runs next?

The obvious answer is "fill the free capacity with whatever is next in line." Watch it break.

Org A has 1,000 charts queued, Org B has 100. Three slots free up. Whatever-is-next hands all three to Org A, because Org A dominates the line. A second later, three more to Org A. Org B sits behind a thousand charts that are not theirs, for hours. The biggest tenant wins every slot purely because it has the most work waiting. Volume becomes priority by accident.

"Fine," you say, "don't go by queue position, go by share. Split the free capacity in proportion to how much each org has waiting." Now the arithmetic turns on you. With capacity C to fill and org i holding q_i of a total Q charts, its slot count is:

slots_i = round(C × q_i / Q)

Plug in C = 3, Q = 1100:

• Org A: round(3 × 1000/1100) = round(2.73) = 3
• Org B: round(3 × 100/1100) = round(0.27) = 0

Org B is mathematically entitled to 0.27 of a slot, and you cannot dispatch a quarter of a workflow. It rounds to zero. Assignment is integral — whole workflows, no fractions — so the small org's fair share keeps vanishing in the rounding.

And it is not a one-off. Org B earns its first whole slot only when round(C × q_B / Q) ≥ 1, that is, when its share crosses 0.5 / C ≈ 16.7%. Org B sits at 100/1100 ≈ 9.1%, well under the line, and it stays under until Org A drains enough to shrink the total: q_B / Q ≥ 1/6 needs Q ≤ 600, i.e. Org A down to ~500 charts. At three dispatched per round, that is roughly (1000 − 500) / 3 ≈ 167 rounds of pure Org A before Org B gets a single slot. The small tenant doesn't just lose, it waits 167 turns to stop losing.

There is a second, quieter failure hiding in here. A customer asks, "when will my charts be done?" With on-the-fly assignment you cannot answer. That ETA — a completion estimate, needed per chart and per org — is impossible to compute when the decision of what runs next is only made at the instant a slot opens. There is no future to read, because the future has not been decided yet.

Unfair scheduling plus un-answerable ETAs. That pair is the entire reason we built what we built.

Why the obvious solutions can't fix it

Lambda

The simplest instinct: event lands, a Lambda runs it. For short, stateless, uniform tasks this is great and you should use it.

It is a non-starter here before we even get to fairness: Lambda has a hard 15-minute ceiling, and our workflows routinely run longer. Hour-long jobs are impossible, full stop. (It also gives you no cross-tenant fairness and nothing to inspect.)

A service pulling from a queue

So you reach for the natural fix: a long-lived service that pulls work off a queue and runs it. Now jobs can run for hours. Problem solved?

No, and this is the crux of the whole post. A queue can never be fair. A queue only ever shows you its front, but fairness is a property of the entire backlog. To schedule Org B fairly against Org A you have to look at all 1,100 charts at once and decide how to interleave them; a consumer pulling from the head cannot, so it reproduces the exact 1,000-versus-100 starvation from above. You have fixed runtime and changed nothing about fairness. And for the same reason you still can't compute ETAs: there is no materialized future to read.

The realization everything else is built on: fairness cannot be decided at dispatch time. It has to be precomputed across the whole backlog, ahead of time. Dispatch should be dumb; the intelligence lives in a precomputed ordering.

SQS, Kinesis, Kafka

Streaming buys you throughput, replay, and ordering, and it is the right tool when you genuinely have a high-volume event stream. It does not solve this.

Fairness in a stream is only as granular as your partition key. Partition-per-org sounds clean until one org is ten times the others and your hot partitions starve the cold ones. You still get no precomputed fair ordering, no per-chart ETA, and no way to reorder or pause work live — a heavy streaming dependency for a problem that was never about stream throughput. It was about fairly ordering a few thousand long jobs.

The missing primitive in all three is the same: a precomputed, fair, inspectable ordering of the entire backlog. That is what we built.

The core idea: precompute a fair order

Our scheduler is an in-memory queue, but the queue is not the interesting part. The interesting part is that it continuously precomputes a fair global ordering of the whole backlog and keeps it materialized, so dispatch is just a constant-time slice off the front. All the intelligence is in the precompute — which is exactly why dispatch can be fair and why it can predict the future.

Weighted round-robin interleaving

The ordering is built by interleaving across orgs instead of draining one at a time. Group the scheduled events by org, oldest first within each group. Then cycle through the orgs, taking up to weight events from each per round, sorting each round's pick by age before appending. Repeat until every group is drained.

Run the 1,000-versus-100 case through this and the starvation is gone: every round gives Org B a turn, so when three slots free up they interleave across orgs instead of all landing on Org A. Each org's weight controls how many turns it gets, so a bigger tenant can legitimately get more throughput, but it can never wall off a smaller one.

A small worked example makes the mechanics concrete. Seven charts arrive in creation order, each tagged with its org — A·1 B·1 C·2 D·1 E·2 F·3 G·1 (chart·org). Org 1 has weight 2; Orgs 2 and 3 have weight 1.

Group by org, oldest first — Org 1: A, B, D, G; Org 2: C, E; Org 3: F — then interleave round by round, taking up to weight from each:

• Round 1: Org 1 takes two (A, B), Org 2 one (C), Org 3 one (F).
• Round 2: Org 1 takes two (D, G), Org 2 one (E), Org 3 is empty.

Sorting each round by age, the queue after precompute is:

arrival:        A  B  C  D  E  F  G
after precompute: A  B  C  F  D  E  G

The difference looks small but it is the whole point: Org 1 never gets more than two in a row before Orgs 2 and 3 get a turn. Org 3's lone chart F jumps ahead of Org 1's D and Org 2's E, because round 1 belongs to everyone, not just the org with the deepest backlog.

Capacity as a cost budget, not a count

A flat "run N at once" breaks on duration. Cheap charts finish in minutes and cycle out; the hour-long ones don't. Run long enough and the slots silently fill with heavyweight workflows that just sit there, while cheap work that could have cycled through ten times waits behind them. Set N low and you waste capacity on easy work; set it high and a cluster of heavy workflows landing together overruns your real resources.

So capacity is a cost budget, not a count. Each workflow carries a cost — a property of the (org, workflow) pair, default 1 — and consumes that much budget while it runs, so a cost-3 workflow occupies the space of three charts. The budget naturally caps how many heavy workflows run at once without throttling the cheap ones. Dispatch walks the precomputed order and runs charts while each fits the remaining budget, stopping at the first that does not (head-of-line, to preserve the fair order) rather than skipping to a cheaper one. No preemption: a cost change never kills in-flight work; dispatch just pauses until it drains.

Scheduling decides what runs; something else has to actually run it for an hour without losing it. We use a durable workflow engine (Temporal), which keeps a long-running workflow alive and resumable across deploys and crashes. One detail worth calling out: rather than trust "done" callbacks (which get dropped or arrive after a restart), we periodically ask the engine which workflows are genuinely still running and reconcile our queue against that. The execution engine, not a message we hope we received, is the source of truth for what is in flight.

When will my charts be done? (ETAs)

Now we can answer the question on-the-fly assignment couldn't. The naive formula — pending × average_duration / capacity — is wrong, because workflows have different durations and different costs, and some are already half-finished. We compute it as a lane simulation: a fast-forward of the precomputed queue draining through capacity.

Model capacity as budget lanes (the cost budget from the last section). Each lane holds one number: how many seconds from now until it next goes free. We need a duration estimate per chart, which is the historical average processing time for that exact (org, workflow), falling back to the org's average. Then:

1. Seed the lanes with what's already running. An in-flight chart's remaining time is started + average − now. A cost-N chart is holding N lanes, so push its remaining time into N of them. Its own ETA is just that remaining time.
2. Pad to exactly budget lanes (a free lane is 0, "free now") and sort them ascending.
3. Walk the precomputed queue in order. A cost-c chart needs c lanes free at the same moment, so it cannot start until the c-th-earliest lane frees up: start = lanes[c − 1]. It finishes at start + average. Mark those c lanes busy until then, re-sort, and move to the next chart. That finish time is the chart's ETA.

Walk the whole queue and every chart has a projected finish. The rollups come for free: an org's clear-time is the latest ETA among its charts; the global clear-time is the latest ETA of all.

It is cost-aware — a cost-3 chart waits for three lanes to align, exactly as it will at dispatch, so a handful of heavy workflows realistically push everyone behind them instead of counting as one cheap unit (with every cost 1, this collapses to the textbook multi-server queue). And it is only possible because the order is fixed in advance: you are fast-forwarding a known sequence through known capacity. On-the-fly assignment has no sequence to fast-forward.

The honest catch: this is the expensive pass, so it does not run on every change. It runs on a fixed few-second timer (and immediately when capacity, weights, or costs change), while the cheap counters and per-org clear-times — read off the last simulation's numbers — update live on every queue event. ETAs tick down smoothly in between.

The features that make it operable

Precompute gives you fairness and ETAs. Materializing the order as a real, first-class object gives you everything else, because now there is something to look at and tug on. When each job costs real money and runs for an hour, you want a cockpit, not a black box.

• High priority. Flag a chart to jump a FIFO tier ahead of all normal work. The "this one is on fire" lever.
• Pause and resume. Suspend specific scheduled charts without removing them, then bring them back.
• Hold. Set an org's weight to zero and the whole tenant is parked: its charts stay queued and visible but never dispatch, and every new chart for that org is auto-held on arrival. A clean kill switch for a misbehaving or non-paying tenant, with no per-chart babysitting. In-flight work is untouched.
• Weights. Tune an org's share of each round to throttle a bursting customer without stopping them.
• Workflow cost. Set the cost for a specific (org, workflow) to make a heavy workflow reserve proportional budget, capping its concurrency without touching the same workflow for other orgs. The total budget itself is a dial too — raise it when resources allow, lower it to protect downstream systems.
• Realtime dashboard. Because the order is materialized, every state transition streams over WebSocket and is watched live — inspect, reorder, hold an org, change a weight or cost, all in real time. With thousands of expensive, long-running jobs this is not a nicety; it is how you operate the system, and it only exists because the schedule is a real object instead of logic that ran once and vanished.

The part nobody owns

Agentic systems rarely fall over because the model was wrong. They fall over because nobody owned the scheduler — and the scheduler they didn't own was a plain queue that could never have been fair in the first place.

Fairness is not a setting you flip at dispatch time. It is something you precompute across the whole backlog, ahead of time, on purpose. Do that, and fairness, ETAs, and live control stop being features you fight for and become things you get for free. As agents get cheaper and longer-running, that scheduling layer stops being plumbing around the intelligence. Increasingly, it is the product.