Cluster nodes down: running a distributed database with failures
Distributed databases are built to lose nodes. With replication, the remaining nodes still hold every row — but with less capacity and less redundancy.
What causes it
Each lost node removes its share of throughput and one copy of the data it held. The cluster keeps answering until too few replicas remain for some partitions to meet the requested consistency level.
How to recognise it
- Fewer nodes reporting healthy
- Higher load on surviving nodes
- Occasional unavailable errors on some keys
How to fix it
- Restart the nodes
- Raise the replication factor
Reproduce it in the simulator
Open the simulator, build a small architecture and use Fault injection in the inspector, or push traffic past a component’s capacity.
Learn more
Part of the guide Message queues vs event streams.