Reference · Mechanics · Contents
A short book on one bug: why a Java service caps at ~30 req/s with the CPU idle, the connection pool "healthy," and — eventually — the health check failing and nodes crash-looping. Read it in order, or jump to the part that's fuzzy.
Every page is self-contained, built from the real numbers and analogies we worked through, and designed to be re-read later. The whole mechanism is a chain: a tiny pool → a hard ceiling → an unfair queue → a runaway spiral → a rolling, fleet-wide outage. Each part is one link in that chain.
One missing argument sends async work to a pool of 3 threads shared by the entire JVM (Part 1). Because each request needs 2 of them and each call takes 50 ms, arithmetic — not any setting — caps the service at ~30 requests/second (Part 2). Push past that and work piles up in an unfair queue, so most requests are fast but a few rot for 15 seconds while the average hides it (Part 3). If requests arrive faster than 30/s, the backlog never drains — it runs away (Part 4). Because load is uneven, this hits some nodes and not others, disguising a universal bug as a per-node incident (Part 5). And when a node's front-end threads are all stuck, even the health check can't get answered, so the orchestrator kills healthy-but-starved nodes and the failure walks across the fleet (Part 6).
The starved resource is invisible: there's no "30" to grep for, no metric on it by default, and every resource you'd normally blame looks healthy — which is exactly why this bug hides until it's an outage.
CompletableFuture javadoc (the default-executor rule) ·
ForkJoinPool javadoc (sizing & no compensation) ·
Little's Law (the ceiling & the mean).