Cassandra UnavailableException: consistency levels and node failures
In Cassandra-style databases each row is stored on several replicas, and each request asks for a consistency level — ONE, QUORUM or ALL replicas. If too few replicas are alive, the request fails with UnavailableException.
What causes it
With replication factor 3 and QUORUM, every request needs 2 of its 3 replicas. Lose one node and every partition still has 2 — fine. Lose two nodes out of six and some partitions have only one replica left, so a share of requests fails, even though most of the cluster is up.
How to recognise it
- UnavailableException: Cannot achieve consistency level QUORUM
- Errors on some keys but not others
- Error share jumps each time another node goes down
The math
P(available) = P(at least CL of RF replicas alive) — hypergeometric over nodes down
Error messages you will see
| Error | Seen in |
|---|---|
UnavailableException: Cannot achieve consistency level | Apache Cassandra, ScyllaDB, Amazon Keyspaces |
How to fix it
- Restart the failed nodes
- Use a lower consistency level for reads that tolerate it
- Increase the replication factor
Reproduce it in the simulator
Open the simulator, build a small architecture and use Fault injection in the inspector, or push traffic past a component’s capacity.
Learn more
Part of the guide Message queues vs event streams.