Message queues vs event streams: SQS, RabbitMQ, Kafka and consumer lag

Putting a queue or a stream between services lets producers move on while consumers work at their own pace. It turns traffic spikes into backlogs — which is great, as long as the backlog drains.

Why asynchronous messaging

In a synchronous call, the caller waits for the callee and inherits its latency and failures. With a queue, the producer only waits for the broker’s acknowledgement, a few milliseconds. User-facing latency stays low even when the work behind it is slow.

The cost is that work happens later, and you need to watch how much later.

The backlog formula

Every second, the backlog grows by what was published and shrinks by what was consumed. If producers outpace consumers for long enough, the backlog — and the age of the oldest message — grows without limit.

Alert on the age of the oldest message as well as on depth. Depth tells you how much work is waiting; age tells you whether it is getting done in time.

B(t+1) = max(0, B(t) + P(t) − C(t))

Queues vs streams

Queues (Amazon SQS, RabbitMQ, Azure Service Bus): each message is delivered to one consumer and deleted once acknowledged. Consumers scale freely; ordering is weak unless you use FIFO queues.
Streams (Apache Kafka, Amazon Kinesis, Google Pub/Sub, Azure Event Hubs): an ordered, replayable log. Many consumer groups can read the same data independently, and messages are kept for a retention period whether or not they were read.
Pub/sub (Amazon SNS, EventBridge, NATS): each message is pushed to every subscriber — fan-out rather than work distribution.

Partitions cap consumer parallelism

A Kafka topic is split into partitions, and within a consumer group each partition is read by one consumer at a time. Six partitions means at most six messages in progress at once.

Adding a seventh consumer does nothing; it sits idle. To process faster you need more partitions, faster processing per message, or both.

Consumer parallelism = min(partitions, consumers × concurrency)

Retention, poison messages and dead-letter queues

Streams delete data after the retention period. If consumer lag (in time) exceeds retention, unread messages are lost.

Queues redeliver failed messages, which recovers transient errors but loops forever on a message that always fails. A dead-letter queue catches messages after a maximum number of receives so the rest of the queue keeps flowing.

Durability: replication and acknowledgements

Kafka replicates each partition across brokers. With acks=all and min.insync.replicas = 2, a write is confirmed only once two replicas have it. Lose too many brokers and producers are rejected with NOT_ENOUGH_REPLICAS — durability chosen over availability.