Reference · Mechanics · Part 4 of 6
Part 2 gave you the ceiling: ~30 requests/second. This part is about what happens when the world pushes past it — and why the failure doesn't just get "a bit slower," it spirals.
The confusion that trips everyone up here is that two completely different numbers both get called "requests per second." Untangle them and the rest of this page falls into place.
There is a rate at which the node can serve work, and a rate at which work arrives. They have nothing to do with each other, and the entire drama is a comparison between them.
| Rate | What it is | Who sets it | Value here |
|---|---|---|---|
| Drain | The service rate — the most the node can possibly serve. | The bug: 3 workers, 2 calls each. Nothing external can raise it. | ~30 req/s |
| Tap | The arrival rate — how much load is pushed at the node. | The outside world: clients and the load balancer. Nothing to do with the pool. | 28 or 35 req/s |
So when we talked about "28 req/s" and "35 req/s," those were both tap values — arrival rates I chose to sit either side of the fixed ~30 drain. The only question that matters is: is the tap faster or slower than the drain?
Think of a sink. The drain empties 30 litres a minute — fixed, you can't change it.
Tap at 28 L/min (below the drain): water leaves as fast as it arrives. The level sits near the bottom and stays there. Stable forever.
Tap at 35 L/min (above the drain): every minute, 5 more litres arrive than can leave. The level rises — 5, 10, 15, 20 litres — and it never stops rising, until it overflows onto the floor. There is no water level at which 35 in / 30 out balances. It can't settle.
Requests behave exactly like the water. Below the drain, the queue stays near empty and latency is just the service time. Above it, the backlog grows every single second:
arrivals 35/s, drain 30/s → +5 requests pile up every second
after 10s: 50 waiting
after 60s: 300 waiting
after 2m : 600 waiting → latency now measured in minutes
after ... : it never stops — the backlog has no upper bound
arrivals 28/s, drain 30/s → queue ≈ 0 the whole time (stable)This is why 28 is healthy and 35 is doomed. It isn't "35 is a little worse than 28." They are on opposite sides of a cliff. Below the drain, the queue has a stable resting point (near zero). Above the drain, the queue has no upper bound at all — it only grows. A 25 % bump in load doesn't cost you 25 % more latency; it moves you from "fine forever" to "collapses eventually." The ceiling from Part 2 is the edge of that cliff.
If the backlog merely grew at a steady +5/s you might notice and react. What makes it accelerate is a positive feedback loop — the slowness feeds more load back into the system:
Notice the shape of it. Over-ceiling makes the node slower; slower triggers thread exhaustion, node evictions, and retries; and every one of those pushes the arrival rate even higher. It doesn't converge to a "slow but stable" state — it spirals downward, faster and faster. That spiral is what "runaway starvation" means. (In the literature this shape is called a metastable failure: the system looks fine right up until a small nudge tips it into a self-sustaining collapse that persists even after the original nudge is gone.)
The loadp.py runs earlier were closed-loop: each client waits for its response before firing the next request. That means arrivals can never truly outrun the drain — the number of requests in flight is capped at the client count. That's why those tests get slow (a 17-second tail) but stay bounded. Real production traffic is open-loop: users and load balancers keep firing at a fixed rate whether or not the node is keeping up. Runaway is an open-loop phenomenon — you need a tap that ignores the drain to make the sink overflow.
Overload is not "a bit slower." When the tap (arrivals) exceeds the drain (~30 req/s service rate), the backlog is unbounded — there is no stable resting point above the ceiling.
And it accelerates: thread exhaustion, node evictions, and client retries all feed more load back in, turning a small overshoot into a self-sustaining spiral.
L = λW that fixes the drain also says queue length and wait time grow without bound once the arrival rate exceeds the service rate. Two terms worth searching to go deeper on the spiral itself: "retry storm" and "metastable failures".
fjp_common_queued climb without bound and latency march toward minutes — then drop it below the ceiling and watch the same graph flatten to zero. Seeing the backlog go vertical vs stay flat is usually when "runaway" stops being a word and becomes a picture.