When a node dies: designing for instance failure
Instances die: hardware fails, a deploy goes wrong, a cloud provider retires a host. Good designs assume it will happen and keep serving when it does.
What causes it
A single instance is a single point of failure. When it stops, every request in flight on it fails, and every new request sent to it is refused. Whether users notice depends entirely on what sits in front: a health-checked load balancer routes around the dead node within one decision; a direct connection just fails.
How to recognise it
- Connection refused or reset errors
- A load balancer reporting fewer healthy targets
- Error rate jumps by exactly that node’s share of traffic
How to fix it
- Watch whether traffic fails over
- Restart it and see how long recovery takes
- Add redundancy so one failure is survivable
Reproduce it in the simulator
These templates trigger this failure on purpose:
Learn more
Part of the guide Load balancing and capacity planning.