Retention data loss: messages that expire before anyone reads them
Streams keep messages for a fixed retention period. If consumers fall further behind than that, unread messages are deleted — permanently.
What causes it
Retention is a time limit, not a guarantee of delivery. When consumer lag (in time) exceeds the retention period, the oldest unread data ages out. Short retention plus slow consumers is a silent data-loss recipe.
How to recognise it
- Consumer offsets reset or “offset out of range”
- Lag measured in time approaching the retention period
- Gaps in downstream data
The math
Data is lost when oldest unread message age > retention
How to fix it
- Add consumers or partitions
- Increase retention
- Alert on consumer lag early
Reproduce it in the simulator
These templates trigger this failure on purpose:
Learn more
Part of the guide Message queues vs event streams.