Reference · Mechanics · Part 5 of 6

Blast radius: why only certain nodes fall over

The same broken code is running on every node — so why does the outage hit some of them and leave the rest looking perfectly healthy? This is the part that fools almost everyone during an incident.

When you first meet this bug in production, one thing feels deeply wrong. You have ten identical nodes, built from one image, running one binary. A few are crash-looping. The rest are calm. Your instinct screams: if this were a code bug, they'd all be broken — so it must be something about those specific machines.

That instinct is half right, and the wrong half is what sends you chasing ghosts for hours.

The instinct you should keep

You are correct that the bug is everywhere. The missing executor argument — the single line that sends async work to the shared 3-thread pool — is compiled into every node. There is no node without it. Nobody deployed a "bad build" to a subset of the fleet.

The pivot: having the bug ≠ triggering the bug

Here is the move that resolves the paradox. Carrying the bug and setting it off are two different events. The defect only bites a node once that node's incoming request rate crosses the ~30 req/s cliff we derived in Part 4. Below the cliff, the queue stays near empty and the node behaves flawlessly. Above it, the backlog runs away.

So the failure isn't bug. The failure is bug + enough load. And load, it turns out, is never handed out evenly.

Same image. Same JVM. Same everything. A gap of a few requests per second is the entire difference between "fine" and "on fire" — because, as Part 4 showed, this is a cliff, not a slope. A tiny difference in load lands two identical nodes on opposite sides of it.

Three reasons the load isn't even

1. Load balancers don't split traffic perfectly

"Round-robin across ten nodes" sounds like ten equal tenths. In reality it almost never is:

• Sticky sessions pin a user to one node for their whole visit.
• Long-lived / keep-alive connections mean a node that grabbed the connection keeps serving it, regardless of the "balance."
• Hash-based routing can send a hot key or tenant repeatedly to the same node.
• A handful of heavy clients — a batch job, a scraper, one big customer — can concentrate on whichever node they landed on.

The result: some nodes sit at 28 req/s while others sit at 35. On a linear system that would just be "a bit busier." On a cliff, it's the line between two completely different fates.

2. The failure rolls — it spreads itself

This is the piece that turns "one unlucky node" into "a fleet-wide incident." The moment a node tips over and starts failing its health check (see Part 6), the load balancer reacts — and its reaction pours fuel on the fire:

Node A tips over  →  health check fails  →  load balancer pulls A
        →  A's traffic is redistributed to B, C, D
        →  now B, C, D each get MORE load
        →  whichever was closest to the cliff tips next
        →  its traffic redistributes again ...
        →  the failure walks across the fleet

Pulling a node doesn't reduce the work — it just relocates it onto the survivors, shoving the next-closest node over its own cliff. And it doesn't stop there: a killed node restarts, comes up briefly looking healthy, rejoins the pool, immediately gets hammered by the redistributed load, and fails again. So nodes don't just fail — they flap in and out.

3. No two nodes are in the same state

Even with perfectly even average load, identical nodes behave differently moment to moment:

• They restarted at different times — one has a warm JVM with JIT-compiled hot paths and an empty queue; another is cold, or is carrying a deep backlog from just before its last restart.
• Garbage-collection pauses land at different instants, briefly stalling one node's workers while its neighbours run clean.

Identical code and identical average load still produce different instantaneous behaviour — enough to decide which node happens to cross the cliff first.

The diagnosis trap

Where hours disappear You pull up a healthy node and it's pristine — CPU idle, memory fine, the connection pool sitting almost empty. You pull up a crash-looping node running the same image. Your brain runs the syllogism: "if it were the code, they'd all be broken — these aren't — therefore it's not the code." So you go hunting the node: a bad host, a noisy neighbour, a flaky NIC, an AZ problem, the load balancer itself. Every one of those is a dead end, because the real cause is the shared line of code — and uneven load is disguising a universal bug as a per-node incident.

The tell you can train yourself to notice: the "broken" nodes have idle CPU and an idle connection pool, exactly like the healthy ones. A genuine bad host or resource leak would look different — pegged CPU, exhausted memory, real errors. Identical-looking-but-starved is the fingerprint of this bug, not of a hardware problem.