Background Jobs

1Why Move Work Out of the Request

A request handler should do the minimum needed to answer the client. Anything slow, retryable, or not needed for the response belongs in a background job:

// before: user waits for the email provider, and a provider outage fails signup
app.post("/signup", async (req, res) => {
  const user = await users.create(req.body);
  await mailer.sendWelcome(user.email);           // 800ms, sometimes throws
  res.status(201).json(user);
});

// after: enqueue and return; a worker sends the email with retries
app.post("/signup", async (req, res) => {
  const user = await users.create(req.body);
  await queue.add("send-welcome-email", { userId: user.id }, { attempts: 5, backoff: { type: "exponential", delay: 2000 } });
  res.status(201).json(user);
});

2The Moving Parts

producer (API) ──add job──► queue (Redis / RabbitMQ / SQS / Postgres) ──► worker(s) ──► side effect
                                         ▲                                    │
                                         └──── retry with backoff / DLQ ◄─────┘ on failure

• Producer — enqueues a job: a name + a small JSON payload (IDs, not whole objects).
• Queue / broker — durable storage of pending jobs (distributed-systems/message-brokers/).
• Worker — a separate process (or several) that pulls jobs and runs handlers. Scale workers independently from the API.
• Scheduler — creates jobs on a cron schedule ("0 3 * * *" for a nightly cleanup).

3BullMQ in Practice

import { Queue, Worker } from "bullmq";
const connection = { host: "localhost", port: 6379 };

export const emailQueue = new Queue("email", { connection });

// producer
await emailQueue.add("welcome", { userId: "u_1" }, {
  jobId: "welcome:u_1",          // dedup: the same jobId isn't added twice
  attempts: 5,
  backoff: { type: "exponential", delay: 2000 },
  removeOnComplete: 1000,
});

// worker (separate process)
new Worker("email", async (job) => {
  const user = await users.findById(job.data.userId);   // load fresh data
  if (!user) return;                                    // deleted meanwhile → nothing to do
  await mailer.sendWelcome(user.email);
}, { connection, concurrency: 10 });

// repeatable job
await emailQueue.add("digest", {}, { repeat: { pattern: "0 8 * * MON" } });

NestJS wraps this with @nestjs/bullmq (@Processor("email"), @InjectQueue("email")); the concepts are identical.

4Delivery Semantics: Design for "At Least Once"

Workers can crash after doing the work but before acknowledging the job, so a job may run more than once. Every handler must be idempotent — running it twice has the same effect as once:

distributed-systems/message-brokers/09-delivery_guarantees goes deeper.

5Retries, Backoff and Dead Letters

• Retry transient failures (timeouts, 5xx, rate limits) with exponential backoff + jitter (distributed-systems/fundamentals/06-retry_with_exponential_backoff).
• Don't retry permanent failures (validation errors, 4xx "user not found") — mark the job failed immediately.
• After the last attempt, move the job to a dead-letter queue (or "failed" set) with its error, alert on it, and provide a way to inspect and replay it.

6The Dual-Write Problem and the Outbox

await db.orders.insert(order);             // 1. commit to DB
await queue.add("order-created", { id });  // 2. enqueue — what if the process dies between 1 and 2?

If the process crashes between the two steps, the order exists but the job never does. If you enqueue first, the job may run for an order that was rolled back. The transactional outbox fixes it: write the job into an outbox table in the same DB transaction as the order; a relay process reads the outbox and publishes to the queue, deleting/marking rows after success.

BEGIN;
INSERT INTO orders (...) VALUES (...);
INSERT INTO outbox (id, type, payload) VALUES (gen_random_uuid(), 'order-created', '{"orderId": 81}');
COMMIT;

7Operating Workers

• Graceful shutdown: on SIGTERM, stop taking new jobs, let in-flight jobs finish (with a deadline), then exit.
• Visibility: queue depth, job age (oldest waiting), failure rate, duration — alert when the backlog grows (devops-cloud/observability/05-monitoring_and_alerting).
• Timeouts: a stuck job holds a concurrency slot forever; set per-job timeouts.
• Priorities / separate queues: don't let a 50,000-job bulk import delay password-reset emails.

Under the HoodHow a Redis-Backed Queue Hands Out Jobs▾

BullMQ stores each queue as a handful of Redis structures: a wait list (LIST of job IDs), an active list, a delayed sorted set (scored by run-at timestamp), completed/failed sets, and a HASH per job with its data and attempt count.

A worker fetches with a blocking move — conceptually BLMOVE wait active — so the job ID moves atomically from wait to active: two workers can never take the same job. While processing, the worker holds a lock key for that job with a short TTL and keeps renewing it. If the worker process dies, renewals stop, the lock expires, and a periodic stalled-job check moves the job back to wait — which is exactly why a job can run twice and handlers must be idempotent.

Failures increment the attempt counter; if attempts remain, the job goes into the delayed sorted set with score now + backoff(attempt), and a timer promotes due delayed jobs back to wait. All multi-step transitions run as Lua scripts inside Redis, which execute atomically, so no other client can observe a half-moved job.

⚠️ Common Pitfalls

1. Putting whole objects in the payload — they go stale; enqueue IDs and load fresh data in the worker.
2. Non-idempotent handlers — duplicate emails, double charges after a retry or stalled-job recovery.
3. Retrying permanent errors — wastes capacity and delays the DLQ alert.
4. Enqueue outside the DB transaction — lost or phantom jobs (use an outbox).
5. Running workers inside the API process without limits — CPU-heavy jobs slow down HTTP latency.
6. No monitoring of queue depth — failures pile up silently.
7. setTimeout/in-memory queues for important work — lost on every restart or deploy.

📋 Quick Reference

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: fw_background_jobs_theory.md · generated by tools/build_lessons.py