Message Queues

Putting a buffer between a producer and a consumer, and what happens when they disagree about pace.

A queue converts an overload into a delay — it never creates capacity.

backpressurequeue depthconsumer lag

Try it

Move the dials — the sentence under the picture changes.

Producer6 msg/tick

Consumer6 msg/tick

Producer and consumer are matched and the queue is empty, so it adds nothing but a hop. Push the producer above the consumer to see backpressure build.

What it is

A queue decouples a producer from a consumer. The producer hands off a message and returns; the consumer picks it up when it can. That buys three things: the producer stops waiting on slow work, a burst gets absorbed instead of rejected, and the consumer can restart without the producer noticing.

What it does not buy is capacity. A queue converts an overload into a delay. If the consumer is permanently slower than the producer, the queue only decides how long it takes for the problem to become visible — and it makes the eventual failure worse, because by then there is a large backlog to work through as well.

Backpressure

Backpressure is the polite word for telling whoever is upstream to slow down. The widget shows the whole story. Producer above consumer means the depth grows, and the system looks healthy the entire time, latency quietly rising, until the queue hits its bound and starts dropping.

Queue depth is the metric worth alerting on, and how fast it is changing matters more than its value: a queue at 40% and growing is a worse position than one at 80% and draining.

When the buffer is full something has to give — drop new messages, drop old ones, or push back on the producer and make it slow down. Pushing back is usually the correct answer, and usually the option nobody built.

Delivery guarantees

At-most-once: fire and forget. Fast, and it loses messages on failure.

At-least-once: the consumer acknowledges after processing, and anything unacknowledged is redelivered. This is the practical default, and it means duplicates are guaranteed — a consumer that crashes after doing the work but before acknowledging will do the work again.

Exactly-once is not something the network delivers; it is something you achieve by making the consumer idempotent — doing the work twice has the same effect as doing it once. Deduplicate on a message id, or design the operation so that repeating it changes nothing. Systems advertising exactly-once are doing this for you within a bounded scope.

Queue or log?

A queue (SQS, RabbitMQ) deletes a message once it has been consumed. Work is distributed across consumers and each message is handled once.

A log (Kafka, Kinesis) keeps an ordered, replayable record and lets each consumer group track its own offset (its position in the stream). Several independent consumers can read the same stream, and you can rewind after a bug. Ordering is per-partition — a lane within a topic; order holds inside one lane — which means your partition key is what decides what "in order" actually guarantees.

Where it goes wrong

Poison messages that fail forever, blocking a partition or being redelivered indefinitely. A dead-letter queue (a side queue for messages that keep failing) after N attempts is the standard escape.
Ordering assumptions. Most queues guarantee ordering far more narrowly than people assume, and a retry reorders things by definition.
Scaling consumers to hide a bug. Adding consumers to drain a growing backlog works right up until the downstream dependency they all share becomes the new bottleneck.

Take this with you

The one idea: a queue turns overload into delay. It does not add capacity; it buys time and absorbs bursts.
In an interview, say at-least-once plus an idempotent consumer, name the partition key as what defines ordering, and mention a dead-letter queue.
At work, alarm on queue depth and on whether it is growing. A queue that is slowly filling is an outage with a delay on it.