Fault injection and chaos engineering: breaking things on purpose
Fault injection means deliberately making a component fail — killing it, slowing it, or making a share of its requests error — to learn how the rest of the system reacts before a real outage teaches you.
What causes it
Most architectures are only tested when everything works. Injecting a 10% error rate or 200 ms of extra latency shows whether callers retry sensibly, whether timeouts are set, whether a load balancer routes around a sick node, and how far a single failure spreads.
How to recognise it
- A controlled, known share of requests failing
- Upstream components reacting (retries, timeouts, failover)
How to fix it
- Check whether callers retry sensibly
- Watch the error rate upstream
- Clear the fault in the inspector
Reproduce it in the simulator
Open the simulator, build a small architecture and use Fault injection in the inspector, or push traffic past a component’s capacity.
Learn more
Part of the guide Load balancing and capacity planning.