Queues and Messaging

SECTION 01

Why queues and workers exist

In the back pressure article, the queue shows up as an abstract piece: a bounded space between the API and the workers that keeps input from destroying a slower component downstream. But "queue" isn't one thing. In practice it's implemented by different tools, and each one solves a different part of the problem.

Producer
   ↓
Broker / Queue
   ↓
Consumer / Worker

Redis stores fast in-memory data structures. RabbitMQ routes messages with delivery guarantees. Kafka keeps an event log that multiple independent consumers can read. Celery and BullMQ are libraries that run background tasks using one of these brokers underneath.

This article walks through each of these pieces and ends by showing exactly where back pressure fits in when they work together.

SECTION 02

Redis: the multipurpose piece

Redis is an in-memory data structure store. The simplest way to think about it: a giant, very fast dictionary that also knows how to store lists, sets, hashes, and streams — not just key-value pairs.

Because of that versatility, Redis becomes the base for several different things: cache (see cache strategies), user sessions, rate limiting, distributed locks, and, with a list or a Stream, a simple queue.

LPUSH queue:emails "{ \"to\": \"a@x.com\" }"

BRPOP queue:emails 0

SECTION 03

RabbitMQ: message broker

RabbitMQ is a dedicated message broker: a server whose only job is to receive, route, and deliver messages between producers and consumers, with delivery confirmation.

Producer
   ↓
Exchange (routes by rule)
   ↓
Queue
   ↓
Consumer → ACK

The extra piece compared to a Redis queue is the explicit ACK: the consumer only confirms the message after finishing processing it. If the consumer crashes mid-job, RabbitMQ redelivers the message to another worker — nothing is lost because of an isolated crash.

SECTION 04

Kafka: event streaming

Kafka solves a different problem than RabbitMQ. Instead of a queue that empties as it's consumed, Kafka keeps an event log: every message published to a topic stays retained for a configurable time, and any consumer can read it — more than once, if needed.

Producer
   ↓
Topic (partitions)
   ↓
Consumer A · Consumer B · Consumer C
   (each reads independently)

That changes what's possible: a payment.completed event can be read at the same time by an analytics service, an audit service, and a recommendation service, with none of them depending on each other or "stealing" the message from the rest.

SECTION 05

Celery: Python task queue

Celery is a Python framework for running background tasks with workers. It isn't a broker itself — it needs one (usually Redis or RabbitMQ) to store pending tasks.

@app.task(bind=True, max_retries=3)
def send_email(self, user_id):
    try:
        email_service.send(user_id)
    except Exception as exc:
        raise self.retry(exc=exc, countdown=10)

# in the API:
send_email.delay(user_id)

The .delay(...) call publishes the task to the broker and returns immediately. A Celery worker, running separately from the API, picks up the task and executes it. Retries, periodic scheduling (Celery beat), and long-running jobs are already handled by the framework.

SECTION 06

BullMQ: Node.js task queue

BullMQ plays the same role as Celery, but for the Node/TypeScript ecosystem, and it's built directly on top of Redis.

const queue = new Queue("notifications", { connection: redis });
await queue.add("push", { userId }, { attempts: 3 });

new Worker("notifications", async (job) => {
  await pushService.send(job.data.userId);
}, { connection: redis, concurrency: 20 });

That concurrency setting isn't a minor detail — it's exactly the kind of limit discussed in back pressure: how many jobs this worker processes at once. If 5000 notifications arrive at once, the queue grows, but the worker keeps consuming at the pace it can handle, instead of trying to run everything at the same time.

SECTION 07

Task vs event: command or fact?

With all five tools on the table, it's easier to see a distinction that trips up a lot of people: task and event are not synonyms.

A task is a work instruction — "do this": send_email, generate_teams, resize_image. Usually a single worker executes it, and the result matters to whoever requested it. An event is a fact that already happened — "this happened": payment.completed, match.cancelled. Multiple consumers can react to it independently, and no "owner" needs to know who's listening.

SECTION 08

Putting it together: a mini e-commerce

Placing the pieces side by side in one system makes it clearer where each one fits.

Creating an order is synchronous: the API queries PostgreSQL, confirms stock, and responds — the same consistency path described in back pressure. Notifying the customer, generating the PDF invoice, and refreshing the cache are async: they become jobs in a BullMQ/Redis queue, processed by dedicated workers. Analytics, auditing, and recommendations don't owe anyone an immediate response: they're independent consumers of an event on Kafka.

SECTION 09

Core idea

None of these tools replace the discipline of back pressure — they just supply the material it's built from.

Internet
   ↓
Rate Limit
   ↓
Concurrency Limit
   ↓
Connection Pool
   ↓
Bounded Queue (RabbitMQ / BullMQ / Celery)
   ↓
Workers
   ↓
PostgreSQL · Kafka (events) · Redis (cache)

Swapping Redis for RabbitMQ, or RabbitMQ for Kafka, doesn't remove the question raised in back pressure: what happens when more work comes in than the system can drain? Each tool changes the technical answer — queue size, ACK, retention, replay — but the architectural decision stays the same: accept only as much as you can handle well, and make clear what happens to the excess.