Reference · Mechanics · Part 6 of 6
This is the moment a quiet latency problem becomes a loud outage — when a process that is genuinely "up" starts failing its own health check, and the orchestrator, trying to help, makes everything worse.
Here is the thing to hold onto before anything else: a health check is just another HTTP request. There is nothing magic about /health or /actuator/health. To answer it, the server has to hand it to a worker thread, exactly like it would a real request. As we saw back in Part 1, those worker threads are a finite pool — and if that pool is exhausted, the health check waits in line with everyone else. That single fact is the whole chapter.
/orders request's Tomcat thread blocks on join(), waiting for workers that never free up. These blocked Tomcat threads pile up — and keep piling up — until the connector hits 200 / 200 busy: the entire main thread pool, every thread frozen mid-request.At the instant the probe fails: CPU is near idle, the JVM is alive and responsive to a debugger, the heap is fine, the connection pool is fine, and there is not a single exception in the logs. The process is "up" and simply cannot answer. This is thread exhaustion cascading out of the ForkJoin pool — a blocked-thread problem, not a connection problem and not a resource-limit problem. Don't go hunting for a crash; there isn't one.
We fired heavy load at the app, then sent the same probe — a 2-second timeout — five times to each connector. The difference is the entire argument for how the demo is wired:
| Connector | Shares the app's threads? | Probe result (×5) |
|---|---|---|
:8080 main connector |
Yes — same Tomcat pool as /orders |
~2 s every time; 2 of 5 timed out (HTTP 000). The rest scraped in right at the 2 s edge. |
:8081 management connector |
No — its own thread pool | HTTP 200 in ~10 ms, every single time. |
Read that main-connector row carefully: a real Kubernetes probe usually runs with a 1-second timeout, not two. Against a connector that can't answer in under ~2 seconds, a 1-second probe fails every time. The instance is marked unhealthy on a schedule.
A failing probe isn't a passive signal — the platform acts on it, and both actions dig the hole deeper:
Liveness probe fails → the orchestrator kills and restarts the instance. But a restart doesn't change the code — the same missing executor is still there — so the fresh instance takes load, starves again, and fails its probe again. That's a crash loop / restart storm: pods cycling endlessly, each one healthy at boot and dead within a minute.
Readiness / load-balancer probe fails → the orchestrator pulls the instance out of rotation. Its share of traffic is now redistributed onto the survivors. Those survivors were already near the edge, so the extra load tips them over too — their probes start failing, they get pulled, and the failure walks across the fleet one node at a time. (This is the rolling collapse from Part 5.)
The dashboards and health endpoints exist precisely for moments like this. But if they answer on the same connector that is drowning, they drown with it — the metrics scrape times out, Grafana goes blank, the health endpoint hangs. You go blind at the exact moment you most need to see. The failure erases its own evidence.
Give the actuator its own port, which in Spring Boot means its own connector with its own thread pool. Now the pool serving /orders can be 200/200 frozen, and observability still has threads to spare:
management:
server:
port: 8081 # own port = own connector = own thread pool
# "The actuator survives the very saturation it exists to diagnose."This is exactly why, in the table above, :8081 answered in ~10 ms while :8080 timed out — the management connector's threads were never tied up carrying /orders to the starved pool. The same reasoning applies to your Kubernetes / ECS liveness and readiness probes: point them somewhere with its own capacity, and never share the connector that's drowning.
The opposite mistake is a health check that's too trivial — one that returns UP without ever touching the code path that's actually starved. Now the probe passes cheerfully while real users get 15-second responses. The orchestrator never restarts anything, no alert fires, and the app degrades silently for as long as you'll tolerate it. Failing-under-load and falsely-passing-UP are two faces of the same defect: a health check that doesn't reflect reality. A good check exercises a representative slice of the real work; a great one has its own capacity to answer even when that work is saturated.
A health check needs a thread too — it is not exempt from the starvation it's meant to detect.
Never share the connector that's drowning. Give observability its own port and thread pool so it survives the very incident it exists to diagnose — and make the check meaningful enough to catch a starved-but-"up" process, without being so heavy it can't answer.
../demo/app/src/main/resources/application.yml (the management.server.port setting) and ../demo/README.md for the full setup. The relevant capability is the Spring Boot Actuator management port, which moves the actuator endpoints onto a dedicated connector.
:8080 time out while :8081 answers in milliseconds is the fastest way to feel why this one config line matters.