Reference · Runbook
The compressed lab manual. Predict → run → read the dashboard → prove with jcmd.
Every run starts on paper. From Little's Law (concurrency = throughput × latency), rearranged:
| Symbol | Meaning | Broken (4 cores) |
|---|---|---|
| workers | concurrent worker slots | 3 (procs − 1) |
| latency | one downstream call | 0.05 s |
| calls/req | fan-out per request | 2 |
| → ceiling | hard throughput cap | 3 × 20 ÷ 2 = 30 req/s |
Key moveThe ceiling is set by worker slots, not offered load or CPU. Offering 50 concurrent clients to a 3-worker pool does not raise it — it only lengthens the queue (and p99).
cd demo
docker compose up -d
# WireMock :8090 Prom :9090 Grafana :3000./run-app.sh broken 4 # 4 cores → parallelism 3
./run-app.sh fixed 4 # dedicated 40-thread pool
./run-app.sh broken 2 # 2 cores → parallelism 1
./run-app.sh broken 1 # 1 core → parallelism 0 (!)App :8080 · actuator :8081 · Grafana has no login.
./ramp.sh # 1→5→10→20→50, ~5 min
./load.sh 50 60 # 50 concurrent, 60 s
# hey preferred; curl fallback if absent
brew install hey # for clean numbersjcmd $(jps | grep starvation | cut -d' ' -f1) \
Thread.print | grep -A5 commonPool
# → 3 workers, all park / socketRead0
curl -s localhost:8080/debug/pool | jqGrafana → Thread Starvation — ForkJoin Common Pool. Each row makes the next one land.
| Row | What it shows | The point |
|---|---|---|
| 1 | The symptom. Throughput flatlines ~30 req/s. p50 pinned ~50 ms; p99 climbs linearly. | p99−p50 divergence is the signature of queueing. The mean would hide it. |
| 2 | The trap. CPU near idle. Connection pool 3 leased / 17 idle. Tomcat threads not near max. | Every resource you'd normally blame looks healthy. Parked threads burn no CPU, so autoscaling never fires. |
| 3 | The bottleneck. fjp_common_active pinned at parallelism; fjp_common_queued grows unbounded. | The one saturated resource is the only one with no default metric — that's why this bug hides. |
| 4 | Alerting. All saturation signals on one panel. | Healthy = flat zero. Alert on queue > 0 sustained 2 min, not a high threshold. |
| Task size | availableProcessors | common pool parallelism | Effect |
|---|---|---|---|
| 4 vCPU | 4 | 3 | this demo (~30 req/s) |
| 2 vCPU | 2 | 1 | one worker for the whole JVM |
| 1 vCPU | 1 | 0 | parallelism<2 ⇒ JVM spawns a new thread per task (see note) |
Correction to verifyThe README calls the 1 vCPU case "silently synchronous / runs on the caller." The JDK 21 javadoc says the opposite: when parallelism < 2, supplyAsync creates a new thread per task. We'll test this empirically in the Fargate lesson.
jcmd output, re-run a mode, or explain a metric.