Kafka consumer lag: when consumers fall behind producers
Consumer lag is the number of messages written to a topic that the consumer group has not processed yet. Growing lag means consumers are slower than producers.
What causes it
Lag follows a simple rule: each second it grows by messages produced minus messages consumed. Consumer throughput is capped by parallelism, and in Kafka parallelism is capped by the partition count — a topic with 6 partitions can have at most 6 consumers working at once, so the seventh sits idle.
How to recognise it
- Lag growing steadily, never recovering
- Oldest unconsumed message getting older
- Adding consumers beyond the partition count changes nothing
The math
B(t+1) = max(0, B(t) + produced − consumed); parallelism ≤ partitions
How to fix it
- Add consumers (up to the partition count)
- Add partitions so more consumers can work
- Make each message cheaper to process
Reproduce it in the simulator
These templates trigger this failure on purpose:
Learn more
Part of the guide Message queues vs event streams.