Reference · Mechanics · Part 3 of 6
Over the ceiling, most requests stay fast while a handful rot for fifteen seconds — and the average quietly hides it. Here is exactly why.
In Part 2 you saw where the ~30 req/s ceiling comes from. Now push past it and watch what happens to the requests. The surprise is that the pain is not spread evenly: it lands on a small, unlucky few, and the number most dashboards lead with — the mean — is built to look away from them.
Split any response into the work itself plus the waiting to be picked up:
response_time = service_time + queue_wait
(~50 ms, fixed) (grows with the backlog)Service time never changes here — the stubbed downstream always answers in 50 ms. Below the ceiling there is no backlog, so queue_wait ≈ 0 and every request is fast. Push over the ceiling and requests start piling up waiting for a free worker. That growing queue_wait is the entire story of the tail — and it lands on the unlucky requests first, not on everyone equally.
This is 396 real requests against the broken/4 app at 20 concurrent clients. Not percentiles — the actual spread:
response time count % of all
100-200 ms 252 63.6% ████████████████████████████████████████
200-500 ms 3 0.8%
0.5-1 s 7 1.8% █
1-2 s 129 32.6% ████████████████████
15-25 s 5 1.3% █
p50 = 123 ms mean = 775 ms p99 = 15,178 msRead it slowly. Almost nobody took 775 ms — the mean. 64% were served in about 150 ms, another 33% waited around 1.5 s, and 5 requests out of 396 sat for 15 to 25 seconds. The distribution isn't a single hump around the average; it's a big fast pile, a second lump, and a thin, far-flung tail. The "average" is a value virtually no one experienced.
The instinct is that a queue is orderly: take a number, first-come-first-served, everyone waits roughly the same. If that were true here, all 20 in-flight requests would wait a similar middling time — a narrow distribution, no 15-second victims. But that is not how this queue behaves.
Picture a worker who always grabs the next task off the top of a pile — the most recently dropped one. New tasks keep landing on top and get grabbed almost instantly. But a sheet that slips to the bottom just sits there, buried under every newcomer, for fifteen seconds while hundreds of fresher tasks are lifted off the top above it.
That's a stack (LIFO — last in, first out), and it's the opposite of a fair line:
Fair line (FIFO): everyone waits about the same → one narrow hump, no tail.
Pile (LIFO): most items (recent) fly through, a few (that sank) are stranded → most fast, a few catastrophic. That is exactly the shape above.
The common pool was designed for CPU-bound divide-and-conquer work — tasks that split into subtasks and run flat-out, never waiting on anything. For that job, grabbing the freshest task first is a feature: the data it needs is still warm in the CPU cache. The pool has no fairness mechanism because it never expected a task to sit around waiting — waiting simply isn't part of the world it was built for.
Now commit the bug and hand it blocking I/O instead. Suddenly tasks do sit and wait (for the HTTP response), and "always grab the newest" curdles into "a few requests sink to the bottom and rot." The precise internal ordering is an implementation detail that varies by JDK — but the observable, reproducible fact is simple: this queue is unfair, and it favours newcomers. That unfairness is the tail.
/orders request needs both of its tasks — service-a and service-b — to finish before it can return. If either one sinks to the bottom of the pile, the whole request is stuck. The fan-out doesn't just double the work; it exposes every request to the tail twice, which is why the stranded fraction is bigger than you'd guess from a single-call service.
The mean of 775 ms is the average of a fast pile and a stuck pile — a middle that almost nobody actually lived through. If you watch p50 (123 ms) or even the mean (775 ms), the service looks merely "a bit slow." Nothing pages. Meanwhile five users just waited fifteen seconds, and in the real world they've already given up and left.
The honest diagnostic is not any single number — it's the gap between p99 and p50. That gap is the backlog made visible. When p99 pulls away from p50 while throughput stays flat, you are looking at a queue filling up, no matter how calm the average looks. (This is the whole point of Lesson 2, where you predict the mean perfectly and are still blindsided by the tail.)
| Offered load | Throughput | p50 | mean | p99 |
|---|---|---|---|---|
| c = 1 (baseline) | 16 req/s | 61 ms | 62 ms | 76 ms |
| c = 4 (at the knee) | 25 req/s | 165 ms | 158 ms | 248 ms |
| c = 20 (5× the load) | 25 req/s | 255 ms | 773 ms | 17,521 ms |
Throughput barely moves from c=4 to c=20 — it's pinned at the ceiling. The extra load had nowhere to go but the queue, and the queue dumped it onto the tail: p99 leapt from 248 ms to over 17 seconds while the median crept up modestly. Flat throughput, exploding p99: that pairing is the fingerprint.
Under a starved pool the queue is a pile, not a fair line — so pain concentrates on a few requests. Watch the gap between p99 and p50; distrust the mean. The tail is where your users actually suffer, and the average is engineered to hide it.