Background Jobs & Queue Design: Getting the Boring Parts Right

Background job systems look simple until retries, ordering, and failure handling show up in production. This post covers the core design decisions queue choice, retry strategy, idempotency, and dead-letter handling that determine whether your job system is reliable or a recurring source of on-call pages.

July 7, 2026System Design

Why Background Jobs Exist

Not every piece of work belongs in the request-response cycle. Sending a confirmation email, generating a report, resizing an uploaded image, syncing inventory across warehouses these shouldn’t make a user wait on the main request. You push them onto a queue, a worker picks them up, and the user gets their response immediately while the actual work happens asynchronously.

The mechanics of “put a job on a queue” are simple. Making that system reliable under real-world failure conditions is where most of the actual engineering effort goes.

Choosing a Queue: The Real Trade-off

The common options Redis-backed queues (Sidekiq, BullMQ), dedicated brokers (RabbitMQ, SQS), or a database-backed queue table aren’t interchangeable defaults. The decision usually comes down to a few questions:

  • Do you need strict ordering? Simple Redis-backed queues often don’t guarantee it out of the box; brokers like RabbitMQ or SQS FIFO queues can, at a throughput cost.
  • Do you need at-least-once or exactly-once delivery? True exactly-once delivery is rare and expensive to guarantee end-to-end; most systems settle for at-least-once and push the responsibility of idempotency onto the consumer instead.
  • How much operational overhead can you take on? A database-backed queue is the least infrastructure to run but the least performant at high volume; a dedicated broker scales better but is one more system to operate, monitor, and keep available.

Idempotency Is Not Optional

At-least-once delivery means every job handler will eventually be invoked more than once for the same logical job a worker crashes after doing the work but before acknowledging it, and the job gets redelivered. If “charge the customer” or “send the email” isn’t idempotent, this isn’t a rare edge case; it’s a guaranteed eventual double-charge or duplicate email.

Practical approach: attach a unique idempotency key to each job (an order ID, a request ID), and check a durable store for “have I already processed this key” before executing the side effect not just before enqueueing it.

Retry Strategy: More Than “Just Retry”

A naive retry loop retry immediately, forever turns a transient failure into a thundering herd that can take down the very dependency you’re retrying against. A better default:

  • Exponential backoff with jitter. Each retry waits longer than the last, with some randomness added so retries from many failed jobs don’t all land at the same instant.
  • A maximum retry count. Infinite retries hide permanently broken jobs instead of surfacing them after N attempts, a job should stop retrying and get flagged, not loop forever silently.
  • Different retry policies for different failure types. A timeout calling a flaky downstream API deserves a retry. A job that fails because of malformed input will fail identically every time retrying it just wastes worker capacity; it should fail fast and get routed for inspection instead.

Dead-Letter Queues Need an Owner

When a job exhausts its retries, it shouldn’t just vanish from the queue silently. Route it to a dead-letter queue and, critically, make sure someone actually looks at it. A dead-letter queue that nobody monitors is just a place where failures go to be forgotten instead of fixed. At minimum, alert on dead-letter queue depth crossing a threshold, and periodically review what’s landing there.

Observability: You Can’t Fix What You Can’t See

The questions you need to answer at 2 a.m. during an incident: how many jobs are queued right now, how long has the oldest one been waiting, what’s the failure rate per job type, and which jobs are stuck retrying. If your queue system can’t answer these without you writing an ad-hoc script, that’s a gap worth closing before you actually need the answer under pressure.

The Underlying Principle

Background job systems fail quietly by default a stuck job or a silent duplicate side effect doesn’t throw an error users see; it just doesn’t happen, or happens twice, somewhere out of view. The design decisions above idempotency, sane retries, monitored dead-letter queues, and real observability exist specifically to make failures loud and visible instead of silent and accumulating.