CPU saturation and queueing: why latency explodes near 100%
A saturated CPU does not crash — it makes everything wait. Response time stays flat at low load, then shoots up as utilisation approaches 100%.
What causes it
Queueing theory explains the curve: with utilisation ρ, the average number of requests waiting grows like ρ ÷ (1 − ρ). At 50% busy, waiting is modest; at 90% it is nine times higher; at 99% it is ninety-nine times higher. The last few percent of capacity are the most expensive.
How to recognise it
- CPU pinned near 100%
- Queue depth and latency rising together
- Throughput flat at capacity while arrivals keep growing
The math
ρ = λ ÷ (c·μ); waiting grows like ρ ÷ (1 − ρ)
How to fix it
- Add cores or instances
- Reduce CPU time per request
- Cache repeated work
Reproduce it in the simulator
Open the simulator, build a small architecture and use Fault injection in the inspector, or push traffic past a component’s capacity.
Learn more
Part of the guide Load balancing and capacity planning.