Timeouts and retries: how slow dependencies cause outages
A timeout is how long a caller is willing to wait. Without one, a slow dependency holds threads, memory and connections indefinitely; with a badly chosen one, healthy-but-busy calls fail for no reason.
What causes it
Slowness spreads upstream. When a database slows down, every service calling it holds resources longer, which slows those services in turn. Timeouts cut that chain. Retries can then recover transient failures — but retrying an overloaded service multiplies its load, so retries need exponential back-off, jitter and a small cap.
How to recognise it
- 504 Gateway Timeout or “Task timed out”
- Latency clusters right at the timeout value
- Request volume to a dependency rising while user traffic is flat (retry storm)
The math
Retry back-off = base × 2^attempt × random(0.5, 1.5)
Error messages you will see
| Error | Seen in |
|---|---|
504 Gateway Timeout | Users, Router, Server, Database, Cache, Queue and 214 more |
Task timed out | AWS Lambda |
How to fix it
- Fix the slow dependency first
- Set timeouts slightly above normal p99 latency
- Use retries with back-off only for idempotent calls
Reproduce it in the simulator
Open the simulator, build a small architecture and use Fault injection in the inspector, or push traffic past a component’s capacity.
Learn more
Part of the guide Load balancing and capacity planning.