Availability zone outage: surviving the loss of a data centre
An availability zone is one or more data centres with independent power and networking. Zones do fail, and when one does, everything you placed only in that zone goes with it.
What causes it
Zone failures are correlated failures: your load balancer, app servers and database primary can all disappear at once if they share a zone. Redundancy inside one zone does not help. Surviving a zone outage needs copies in another zone plus something — a health-checked load balancer or DNS — that moves traffic there, and a database replica that can be promoted to primary.
How to recognise it
- Many components fail at the same moment
- Health checks fail for every target in one zone
- Database writes fail until a replica in another zone is promoted
How to fix it
- Spread replicas across zones
- Put a health-checked load balancer or DNS in front
- Use multi-AZ databases with automatic failover
Reproduce it in the simulator
These templates trigger this failure on purpose:
Learn more
Part of the guide Load balancing and capacity planning.