Message Queue Basics

Your API was serving 400 requests per second two weeks ago. This morning it caps out at 30 and logs are full of ETIMEDOUT errors from a million push notifications nobody asked for. The core microservice just collapsed because an email worker took 12 seconds to respond on a third-party SMTP endpoint. You do not need a bigger database or a better load balancer. You need to decouple the producer from the consumer, and a message queue is the cheapest way to do it — the single most underrated pattern in system design.

A message queue is a buffer between two parts of a system: the part that generates work (the producer) and the part that does the work (the consumer). The producer writes a message, the queue holds it durably, and the consumer pulls it whenever it has capacity. That single shift removes the most common failure mode in distributed systems: synchronous coupling, where a slow downstream service takes down everything upstream of it.

Why Synchronous Calls Are the Root Problem

In a synchronous architecture, every request waits for the full chain to finish. Order-service calls payment-service which calls ledger-service which calls an email gateway. If any single hop takes 3 seconds, your p95 latency is already at 3 seconds plus overhead. When the email gateway goes down, every order fails even though you only needed to send a confirmation email.

Queues fix this by swallowing the latency. The order service publishes a message and returns 200 in under 50 milliseconds. The email is sent later, possibly seconds or minutes later, by a worker that is allowed to keep retrying. This is why payment gateways like Stripe famously use webhooks and queues instead of blocking your checkout on their internal settlement pipeline. The same decoupling argument explains why resilient backends are built around microservices architecture rather than one monolithic chain.

The Core Concepts You Must Understand First

Before comparing tools, get the vocabulary straight, because every vendor uses slightly different terms for the same ideas.

• Producer / Publisher. The component that enqueues a message. It sends and forgets; it is not blocked by downstream health.
• Consumer / Subscriber. The component that reads messages from the queue and processes them.
• Broker. The server-side software that stores and routes messages. RabbitMQ, Kafka, and Amazon SQS are all brokers.
• Topic vs. Queue. A topic broadcasts a message to many consumers; a queue delivers each message to exactly one consumer. Kafka uses topics with consumer groups; RabbitMQ uses queues with routing keys.
• At-least-once vs. exactly-once delivery. Nearly every real system guarantees at-least-once, meaning your worker may see the same message twice. Your consumer code must be idempotent (safe to run twice) rather than trusting the broker to deduplicate.
• Acknowledgement (ACK). The consumer tells the broker it finished. If it fails or times out, the broker re-delivers the message.
• Dead-letter queue (DLQ). A holding area for messages that repeatedly fail. Without a DLQ, a poison message can be retried forever and stall your whole pipeline.

If you skip the idempotency lesson, you will learn it the hard way: a retried payment webhook creating two charge records, or a retried order job shipping the same cart twice.

Comparing the Major Message Brokers

The tool you pick depends on your throughput ceiling, your delivery semantics, and whether you want to run it yourself or rent it.

As a rule of thumb: use SQS or Redis Streams when you have a small team and just need reliable job execution. Reach for Kafka when you have multiple consumers deriving different products (analytics, search index, ML features) from the same event stream, and you need the ability to replay old events.

How to Pick: A Simple Decision Tree

Rather than cargo-culting whatever your last job used, work through these four questions.

1. Do you need to replay historical events? Yes to replay → Kafka or Pulsar. No → RabbitMQ or SQS.
2. What is your peak throughput? Below ~5,000 messages/sec → RabbitMQ, SQS, or Redis. Above that, or with heavy fan-out → Kafka.
3. Do you want to operate infrastructure? No → managed options (SQS, Pub/Sub, Confluent Cloud). Yes, by choice or for data-residency reasons → self-hosted RabbitMQ or Kafka.
4. Do you need strict ordering + exactly-once semantics? Yes → SQS FIFO or a single-partition Kafka topic. No → at-least-once is fine.

Teams over-engineer this. For a typical web application doing background emails, file processing, and webhook fan-out, a managed SQS queue and three worker containers will carry you to a very large scale before you ever need Kafka. If you are new to backend plumbing, a worker on a Node.js backend, or a queue integrated into your cloud architecture with managed infrastructure, is the fastest path to a working system.

Implementation Gotchas That Trip Up Real Teams

Reading the docs is not enough. These are the failures I have seen repeatedly in production.

• Forgetting idempotency keys. At-least-once delivery means duplicates are normal, not exceptional. Add a unique message ID and store processed IDs in a table or deduplication cache before doing side effects.
• Setting visibility timeouts too low in SQS. If your worker usually takes 30 seconds but you set a visibility timeout of 15 seconds, every slow message becomes a duplicate. Either raise the timeout or make the consumer idempotent, ideally both.
• No dead-letter configuration. A malformed payload from a bad deploy will be retried until your queue backs up and your customer-facing systems stall. Wire a DLQ in from day one.
• Consumers that lack backoff. A crashing consumer piling up unACKed messages causes redelivery storms. Implement exponential backoff and a max retry count.
• Blocking I/O in consumer threads. Long database calls inside a single-threaded consumer serialize your entire work. Use concurrency that matches your downstream capacity.
• Monitoring blind spots. Queue lag (messages waiting) is the single most important metric. Set an alert when lag exceeds your healthy baseline, not when the queue is empty.

The most common production incident in queue systems is not the queue failing. It is the consumer being too slow and the lag growing until retention expires and messages are silently lost. Viewed through the lens of software architecture, the queue is only one moving part in a system where the real pressure point is capacity planning and monitoring.

Retention, Ordering, and Delivery Semantics in Practice

Kafka defaults to retaining messages for 7 days, and you pay storage the whole time. SQS holds messages for up to 14 days by default, then deletes them. RabbitMQ keeps messages until consumed or a queue TTL is set. If your business needs messages for months (compliance, event sourcing), tuned retention in Kafka or Pulsar beats a work queue that deletes on consume.

Ordering is another trap. A topic with 12 partitions only guarantees order within a single partition. If you publish order events without setting a partition key, related events can land in different partitions and arrive out of order. Always set the partition key to the entity you care about (customer ID, order ID, stream ID).

Exactly-once is real but limited. Kafka's exactly-once semantics work only within a single Kafka write-read-write transaction and break the moment you touch an external database or another service. Real-world systems effectively use at-least-once plus idempotent consumers, and that is the honest architecture to plan for.

Patterns Worth Stealing

These four patterns solve most queue problems before they happen.

• Disaster-recovery job queue. Push failed tasks (webhooks, syncs, exports) to a queue with a DLQ and a retry schedule. Human operators drain the DLQ instead of staring at logs.
• Fan-out for analytics. One producer event feeds the user-timeline service, the search indexer, and the analytics pipeline independently. A consumer slow-down no longer blocks the others.
• Backpressure control. Because the queue absorbs spikes, your workers keep a steady, predictable load instead of being crushed by bursts. This smooths out your database CPU too.
• Durable request/response. Even request/response flows benefit from a request queue and a response topic. The caller polls for its correlation ID, so a restart mid-flight does not lose the request.

Costs You Should Actually Budget For

Managed brokers price on volume, and the free tiers are generous but small. SQS gives 1 million requests/month free, which is roughly a few hundred thousand messages once you count polling; beyond that it is $0.40 per million requests. Confluent Cloud's free tier allows 5 MB/s ingress, good enough to prototype, but production Kafka storage and egress bills add up. Google Pub/Sub charges $40 per TiB of delivered bytes after a 10 GB free allowance. If you process billions of messages, self-hosted RabbitMQ on a single VM can be dramatically cheaper than a managed option, at the cost of you owning uptime and version upgrades.

Frequently Asked Questions

When should I use Kafka instead of RabbitMQ or SQS?

Use Kafka when you need to replay historical events, when dozens of consumers read the same stream, or when your throughput exceeds roughly 5,000 messages per second. Use RabbitMQ or SQS for standard job execution and background work at lower volume.

How do I stop duplicate messages from corrupting my data?

Make your consumers idempotent: process each message exactly once by checking a deduplication table keyed on the message ID before performing side effects. At-least-once delivery makes duplicates normal, so treat them as the default and engineer for them.

What is a dead-letter queue and when should I create one?

A DLQ holds messages that exhaust their retry limit so they do not loop forever and block the main queue. Create one for every queue you build, because a malformed payload from a single bad deploy will otherwise stall your entire pipeline.

Can I get exactly-once delivery in practice?

Only within a single broker transaction. Once you touch an external database, a third-party API, or another service, exactly-once is impossible. The industry-standard approach is at-least-once delivery combined with idempotent consumers, and you should build for that.

How much does running a message queue actually cost for a small startup?

Very little. SQS's free tier covers about 1 million requests per month, which handles the background workload of a typical small app. Self-hosted RabbitMQ on one VM costs only the VM. Costs climb only when you hit tens of millions of messages per month.