Lesson 2 · Predict → Run → Prove

The signature of queueing

You can predict the mean perfectly and still be blind to what your users feel. This lesson is about learning to distrust it.

In Lesson 1 you found the ceiling: ~25–30 req/s, no matter how much load you offer. Here's the obvious next question — if extra load doesn't become throughput, where does it go? The answer is written in the latency percentiles, and it is invisible to the one number most dashboards lead with.

The knowledge (30 seconds)

Every response time splits in two:

response time = service time  +  queue wait
               (~50 ms, fixed)   (grows with backlog)

Below the ceiling there's no backlog, so latency ≈ service time and everyone's experience is the same. Above the ceiling, requests pile up behind the three workers, and the wait lands on the unlucky requests first. The distribution goes bimodal: some served immediately, some stuck behind the queue. And the mean of a bimodal split is a lie — it reports a middle nobody actually experiences.

p50 — the typical request. Moves modestly.
p99 — the queued, unlucky request. Explodes. This is the one your SLO and your angriest users live on.
mean — averages the fast and the stuck. Looks merely "a bit slow" while the tail is on fire.

The measured data (broken / 4 cores)

Offered load	Throughput	p50	mean	p99
c = 1 (baseline)	16 req/s	61 ms	62 ms	76 ms
c = 4 (at the knee)	25 req/s	165 ms	158 ms	248 ms
c = 20 (5× the load)	25 req/s	255 ms	773 ms	17,521 ms

Closed-loop load, warm JVM. Your numbers will vary a little — the shape is the point.

Predict, before you read the reveal

Going from c=4 to c=20 you offered five times the load. Two predictions:

Still ~25 req/s — flat. Offering 5× the load moved throughput by nothing. So all that extra load went somewhere else. Hold that thought.

W = 20 / 25 = 0.8 s ≈ 800 ms. Measured: 773 ms. Little's Law — your core equation — predicts the mean to within a few percent. You can forecast the mean. That's exactly what makes the next number so dangerous.

The reveal — the mean lies You just predicted the mean (~800 ms) almost exactly. Now look at the table's last cell: p99 = 17,521 ms. Seventeen and a half seconds. The mean you nailed was 22× rosier than what the unlucky 1% actually endured. A dashboard headlining "avg latency 773 ms" would never page anyone — while a slice of your users time out and leave. p99 pulling away from p50/mean is the signature of queueing. The gap is the backlog.

Run it yourself

Drive load while watching the latency panel. Any load tool works — the percentiles come from the server-side histogram, so even the curl fallback shows them on Grafana:

cd ~/Desktop/thread-starvation-course/demo
./run-app.sh broken 4                 # if not already running

python3 loadp.py 4 20                 # knee: p99 close to p50
python3 loadp.py 20 25                # overload: watch p99 detonate
# (or ./load.sh 20 25 — drives load; read percentiles off Grafana)

Open Grafana :3000 → Row 1. Put p50 and p99 on the same panel and watch the gap open as you step the load up. The throughput panel beside it stays flat. That side-by-side — flat throughput, diverging percentiles — is the picture to burn in.

Throughput held flat at 25 req/s from c=4 to c=20, yet p99 leapt from 248 ms to 17,521 ms. Where did the five-fold extra load actually go?

It became queue wait. Service time never changed — the stubs still answer in 50 ms, so p50 barely moved. The extra load had nowhere to go but the ForkJoin submission queue, and that wait lands on the tail: p99 exploded while throughput stayed pinned at the ceiling. Note the two tempting wrong answers: the downstream did not slow (fixed 50 ms), and the connection pool did not fill — it was sitting almost idle. We'll dismantle that "it must be the connection pool" instinct properly in Lesson 3, because it's the single most common misdiagnosis of this bug.

Primary source Little's Law — L = λW. The same equation that predicts the ceiling (Lesson 1) predicts the mean latency here. It says nothing about the tail — which is precisely why the tail needs its own panel. For the deeper "why percentiles, not averages" argument, Gil Tene's talk "How NOT to Measure Latency" is the canonical reference.

Your teacher is in the chat. Paste your c=20 numbers and we'll compare the shape to mine — especially how far your p99 ran. When you're ready, ask for Lesson 3: the trap. Why every resource you'd normally blame — CPU, the connection pool — looks perfectly healthy while the app suffocates, and why a blocked thread is not a blocked connection.

← Lesson 1 Home · Runbook · Glossary Lesson 3: the trap →