# Background Jobs

## 1. Why Move Work Out of the Request

A request handler should do the minimum needed to answer the client. Anything slow, retryable, or not needed for the response belongs in a **background job**:

| Work | Why not in the request |
|---|---|
| sending emails / push notifications | third-party latency + failures shouldn't fail the user's action |
| image/video processing, PDF generation | seconds to minutes of CPU |
| calling webhooks | the receiver may be slow or down; needs retries |
| report exports, bulk imports | long-running, would hit HTTP timeouts |
| periodic cleanup, billing runs | not triggered by a request at all (scheduled) |

```ts
// before: user waits for the email provider, and a provider outage fails signup
app.post("/signup", async (req, res) => {
  const user = await users.create(req.body);
  await mailer.sendWelcome(user.email);           // 800ms, sometimes throws
  res.status(201).json(user);
});

// after: enqueue and return; a worker sends the email with retries
app.post("/signup", async (req, res) => {
  const user = await users.create(req.body);
  await queue.add("send-welcome-email", { userId: user.id }, { attempts: 5, backoff: { type: "exponential", delay: 2000 } });
  res.status(201).json(user);
});
```

## 2. The Moving Parts

```
producer (API) ──add job──► queue (Redis / RabbitMQ / SQS / Postgres) ──► worker(s) ──► side effect
                                         ▲                                    │
                                         └──── retry with backoff / DLQ ◄─────┘ on failure
```

- **Producer** — enqueues a job: a name + a small JSON payload (IDs, not whole objects).
- **Queue / broker** — durable storage of pending jobs (`distributed-systems/message-brokers/`).
- **Worker** — a separate process (or several) that pulls jobs and runs handlers. Scale workers independently from the API.
- **Scheduler** — creates jobs on a cron schedule (`"0 3 * * *"` for a nightly cleanup).

## 3. BullMQ in Practice

```ts
import { Queue, Worker } from "bullmq";
const connection = { host: "localhost", port: 6379 };

export const emailQueue = new Queue("email", { connection });

// producer
await emailQueue.add("welcome", { userId: "u_1" }, {
  jobId: "welcome:u_1",          // dedup: the same jobId isn't added twice
  attempts: 5,
  backoff: { type: "exponential", delay: 2000 },
  removeOnComplete: 1000,
});

// worker (separate process)
new Worker("email", async (job) => {
  const user = await users.findById(job.data.userId);   // load fresh data
  if (!user) return;                                    // deleted meanwhile → nothing to do
  await mailer.sendWelcome(user.email);
}, { connection, concurrency: 10 });

// repeatable job
await emailQueue.add("digest", {}, { repeat: { pattern: "0 8 * * MON" } });
```

NestJS wraps this with `@nestjs/bullmq` (`@Processor("email")`, `@InjectQueue("email")`); the concepts are identical.

## 4. Delivery Semantics: Design for "At Least Once"

Workers can crash after doing the work but before acknowledging the job, so a job may run **more than once**. Every handler must be **idempotent** — running it twice has the same effect as once:

| Technique | Example |
|---|---|
| natural idempotency | "set status = shipped" (not "toggle") |
| idempotency key / dedup table | record `job:welcome:u_1 done` in a table with a unique constraint before sending |
| check-then-act on current state | skip if `invoice.paid_at` is already set |
| provider idempotency keys | pass `Idempotency-Key: <jobId>` to Stripe |

`distributed-systems/message-brokers/09-delivery_guarantees` goes deeper.

## 5. Retries, Backoff and Dead Letters

- **Retry transient failures** (timeouts, 5xx, rate limits) with **exponential backoff + jitter** (`distributed-systems/fundamentals/06-retry_with_exponential_backoff`).
- **Don't retry permanent failures** (validation errors, 4xx "user not found") — mark the job failed immediately.
- After the last attempt, move the job to a **dead-letter queue** (or "failed" set) with its error, alert on it, and provide a way to inspect and **replay** it.

## 6. The Dual-Write Problem and the Outbox

```ts
await db.orders.insert(order);             // 1. commit to DB
await queue.add("order-created", { id });  // 2. enqueue — what if the process dies between 1 and 2?
```

If the process crashes between the two steps, the order exists but the job never does. If you enqueue first, the job may run for an order that was rolled back. The **transactional outbox** fixes it: write the job into an `outbox` table **in the same DB transaction** as the order; a relay process reads the outbox and publishes to the queue, deleting/marking rows after success.

```sql
BEGIN;
INSERT INTO orders (...) VALUES (...);
INSERT INTO outbox (id, type, payload) VALUES (gen_random_uuid(), 'order-created', '{"orderId": 81}');
COMMIT;
```

## 7. Operating Workers

- **Graceful shutdown**: on `SIGTERM`, stop taking new jobs, let in-flight jobs finish (with a deadline), then exit.
- **Visibility**: queue depth, job age (oldest waiting), failure rate, duration — alert when the backlog grows (`devops-cloud/observability/05-monitoring_and_alerting`).
- **Timeouts**: a stuck job holds a concurrency slot forever; set per-job timeouts.
- **Priorities / separate queues**: don't let a 50,000-job bulk import delay password-reset emails.

## Under the Hood: How a Redis-Backed Queue Hands Out Jobs

BullMQ stores each queue as a handful of Redis structures: a **wait list** (LIST of job IDs), an **active list**, a **delayed** sorted set (scored by run-at timestamp), **completed/failed** sets, and a HASH per job with its data and attempt count.

A worker fetches with a blocking move — conceptually `BLMOVE wait active` — so the job ID moves atomically from *wait* to *active*: two workers can never take the same job. While processing, the worker holds a **lock key** for that job with a short TTL and keeps renewing it. If the worker process dies, renewals stop, the lock expires, and a periodic **stalled-job check** moves the job back to *wait* — which is exactly why a job can run twice and handlers must be idempotent.

Failures increment the attempt counter; if attempts remain, the job goes into the *delayed* sorted set with score `now + backoff(attempt)`, and a timer promotes due delayed jobs back to *wait*. All multi-step transitions run as **Lua scripts** inside Redis, which execute atomically, so no other client can observe a half-moved job.

![The API enqueues a job into the wait list; a worker atomically moves it to active and holds a renewing lock; success moves it to completed, failure schedules a delayed retry with backoff, and exhausted attempts land in the failed set](diagrams/01-job-lifecycle.svg)

## Common Pitfalls

1. **Putting whole objects in the payload** — they go stale; enqueue IDs and load fresh data in the worker.
2. **Non-idempotent handlers** — duplicate emails, double charges after a retry or stalled-job recovery.
3. **Retrying permanent errors** — wastes capacity and delays the DLQ alert.
4. **Enqueue outside the DB transaction** — lost or phantom jobs (use an outbox).
5. **Running workers inside the API process** without limits — CPU-heavy jobs slow down HTTP latency.
6. **No monitoring of queue depth** — failures pile up silently.
7. **`setTimeout`/in-memory queues for important work** — lost on every restart or deploy.

## Quick Reference

| Need | Use |
|---|---|
| durable job queue (Node) | BullMQ (Redis), pg-boss (Postgres), SQS |
| delay / schedule | `delay`, `repeat: { pattern }` |
| retries | `attempts` + exponential `backoff` |
| dedup | stable `jobId` / idempotency key |
| atomic "write + enqueue" | transactional outbox |
| scale processing | more workers / `concurrency` |
| failed after all retries | dead-letter queue + alert + replay |