How distributed systems fail
Every failure ServerSloth simulates, with its cause, the error messages you will see in production and how to fix it.
Load balancing and capacity planning
- Out of memory (OOM)Why application servers run out of memory when requests queue faster than they finish, how to spot it, and how to fix it before the OOM killer does.
- When a node diesWhat happens to traffic when a server or database instance dies suddenly, and how health checks, redundancy and failover keep the system up.
- Availability zone outageHow an availability zone outage takes down every component placed in it, and how multi-AZ load balancers, DNS failover and replicas keep you online.
- CrashLoopBackOffWhy Kubernetes pods and auto-scaled replicas crash repeatedly under load, what CrashLoopBackOff means, and how to break the loop.
- Queue full (503)What it means when a server or load balancer rejects requests with 503 because its queue is full, and why that is often the right behaviour.
- “Too many connections”Why PostgreSQL says “sorry, too many clients already” and MySQL returns error 1040, how connection limits work, and how poolers, caches and replicas fix it.
- Throttling and 429 Too Many RequestsHow API gateways, DynamoDB, S3 and third-party APIs throttle traffic with 429 and 503 SlowDown, and how back-off, queues and quotas fix it.
- Serverless concurrency limitsHow serverless concurrency limits work, why λ × duration decides how many instances you need, and how to avoid TooManyRequestsException.
- No healthy targetsWhy load balancers return 503 when every target fails its health check, and how redundancy across zones prevents a total outage.
- Timeouts and retriesWhy timeouts need to be set deliberately, how retries with exponential back-off and jitter help, and how retry storms make overload worse.
- Fault injection and chaos engineeringWhat fault injection is, why teams break their own systems deliberately, and which failures to try first in a system design simulator.
- CPU saturation and queueingWhy response time grows sharply as utilisation approaches 100%, explained with queueing theory, and what to do when CPU is saturated.
- Serverless cold startsWhat a cold start is in AWS Lambda, Cloud Run and Azure Functions, why traffic spikes cause them, and how provisioned concurrency helps.
- Autoscaling lagHow target-tracking autoscaling works, why replicas trail demand by their start-up time, and how to scale earlier.
- Max replicas reachedWhat happens when an auto-scaling group or Kubernetes deployment reaches its maximum replica count and load keeps growing.
- Latency injectionWhy adding latency to one dependency raises memory, connections and errors upstream, and what it teaches about timeouts.
- High load warningsWhich early signals — CPU at 80%, memory at 80%, queues half full — predict an outage, and how to act on them before users notice.
Caching in system design
- Hot partitions and hot keys in DynamoDB, Cassandra and KafkaWhy one popular key can throttle a partitioned database while total capacity sits unused, and how key design, write sharding and caching fix hot partitions.
- Replica lagWhat database replica lag is, why it grows when writes outpace a replica, and how to keep read-after-write consistency with replicas.
- Cache stampedeWhat a cache stampede (thundering herd) is, why a Redis restart or flush can take down your database, and how to prevent it.
- Cache evictionWhy a cache that is too small keeps evicting keys, how to estimate the memory you need, and the difference between allkeys-lru and noeviction in Redis.
Message queues vs event streams
- Disk fullWhy databases and Kafka brokers stop accepting writes when their disk fills, the error messages you will see, and how to prevent it.
- Third-party API failuresHow errors and rate limits from payment, email, auth and AI APIs reach your users, and how retries, circuit breakers and queues contain them.
- Cassandra UnavailableExceptionHow replication factor and consistency level decide whether a Cassandra or ScyllaDB cluster can answer when nodes are down, with the probability math.
- Kafka NOT_ENOUGH_REPLICASWhy Kafka rejects producers with NOT_ENOUGH_REPLICAS when brokers go down, and how replication factor and min.insync.replicas trade durability for availability.
- Database read-onlyWhy a database starts rejecting writes while reads keep working, and the PostgreSQL and MySQL errors that tell you the disk is full.
- Kafka consumer lagWhat consumer lag is, why it grows, why more consumers than partitions do not help, and how to size Kafka consumer groups.
- Queue backlog ageWhy the age of the oldest message in SQS or RabbitMQ is a better alert than queue depth, and how to drain a backlog.
- Under-replicated partitionsWhat under-replicated partitions mean in Kafka, why they appear when a broker fails, and why they are an urgent warning sign.
- Cluster nodes downHow distributed databases keep serving data when some nodes fail, and what capacity and redundancy you lose meanwhile.
- Retention data lossHow Kafka and Kinesis retention works, why lagging consumers lose messages when retention runs out, and how to prevent it.
- Poison messages and dead-letter queuesWhat a poison message is, why one bad message can block a queue, and how dead-letter queues and max receive counts contain it.
- Pub/sub delivery failuresWhy SNS, EventBridge and other pub/sub systems drop deliveries after retries, and why subscribing a queue is safer than a function.