# Asynchronous Processing

## 1. What "Asynchronous Processing" Means at the System Level

This topic is about a specific pattern built on top of asynchronous *communication* (previous topic): work that is deliberately deferred, queued, and executed later — separate from the request that triggered it. The classic shape:

```
1. A request arrives (HTTP call, event, upload, etc.)
2. The service does the minimum work needed to accept it, then responds immediately
3. The actual work is handed off — to a background job, a queue, a worker pool
4. A worker picks up the deferred work and executes it, independently of the original caller
```

This is different from the JavaScript-level concept of an async function or a `Promise` (covered fully in `languages/javascript/27-event_loop` and `languages/javascript/29-promises`) — those are the *mechanism* a single process uses to avoid blocking on I/O. This topic is about deferring work *across a system boundary*, often to an entirely different process, possibly minutes or hours later.

## 2. Why Defer Work At All

- **Fast responses.** A user uploading a video shouldn't wait for transcoding to finish before seeing "upload received."
- **Absorbing spikes.** A flash sale generates 50,000 order confirmations in a minute; sending them all synchronously would overwhelm the email provider and the checkout path with it.
- **Resource isolation.** A CPU-heavy report generation job shouldn't run on the same request thread pool that serves login requests.
- **Resilience.** If the email provider is down for ten minutes, queued jobs simply wait — they don't turn into failed checkouts.

## 3. The Building Blocks: Job Queue + Worker Pool

```ts
// producer side: enqueue work instead of doing it inline
async function handleUpload(fileId: string) {
  await storage.save(fileId);
  await jobQueue.enqueue("transcode-video", { fileId });
  return { status: "accepted", fileId };
}

// worker side: a pool of workers pulls jobs and executes them independently
async function runWorker(jobQueue: JobQueue) {
  while (true) {
    const job = await jobQueue.dequeue(); // blocks until work is available
    try {
      await transcodeVideo(job.payload.fileId);
      await jobQueue.ack(job.id);
    } catch (err) {
      await jobQueue.nack(job.id); // re-queue or dead-letter, see Dead Letter Queues
    }
  }
}
```

A **worker pool** runs N workers concurrently pulling from the same queue, so throughput scales by adding workers (bounded by the downstream resource — a database, an external API — not by the queue itself).

![A producer enqueues work and returns immediately; a pool of workers pulls jobs independently, each on its own schedule](diagrams/01-job-queue-worker-pool.svg)

## 4. Fire-and-Forget vs Tracked Async Work

Not all deferred work is equal — it matters whether anyone needs to know the outcome:

| Pattern | Caller cares about the result? | Example |
|---|---|---|
| **Fire-and-forget** | No — best-effort only | Logging an analytics event |
| **Tracked async job** | Yes, eventually — via polling or a callback | Video transcoding status shown in a UI |
| **Async with notification** | Yes, pushed back to the caller | Email sent when a report finishes generating |

A tracked job needs a **job status record** (`pending` → `running` → `completed`/`failed`) that the caller can poll (`GET /jobs/{id}`) or be notified about (webhook, WebSocket, another queue message) — the queue itself usually doesn't retain enough history for this once a job is dequeued.

## 5. Idempotency: The Non-Negotiable Requirement

Because queues generally offer **at-least-once** delivery (a job can be delivered more than once — see the Message Brokers group's Delivery Guarantees topic), every job handler must be **idempotent**: running it twice with the same input must produce the same end state as running it once.

```ts
// NOT idempotent — running twice charges the customer twice
async function chargeCustomer(orderId: string, amount: number) {
  await paymentGateway.charge(amount); // no way to detect "already charged"
}

// idempotent — a unique key lets the downstream system detect duplicates
async function chargeCustomerIdempotent(orderId: string, amount: number) {
  await paymentGateway.charge(amount, { idempotencyKey: `charge-${orderId}` });
  // the gateway itself recognizes a repeated idempotencyKey and returns the
  // ORIGINAL result instead of charging again
}
```

Idempotency has to be designed in at the handler (or the downstream API), not assumed away — "the queue probably won't redeliver" is not a guarantee any real broker makes.

## 6. Ordering and Concurrency

Workers processing jobs concurrently means jobs for the *same logical entity* can run out of order or in parallel unless something prevents it:

```ts
// two jobs for the SAME order running concurrently on different workers
// can race: job A sets status="shipped", job B (an older event) then
// overwrites it back to status="paid"
```

Common fixes: partition work so all jobs for one entity always go to the same worker (this is exactly what Kafka's partitioning does — see the Message Brokers group), include a version/timestamp check before applying an update ("only apply this if it's newer than the current state"), or use a per-entity lock while processing.

## 7. Backpressure and Scaling the Worker Pool

If jobs arrive faster than workers can process them, the queue backs up. This is the subject of the next topic in full, but the shape here: either **scale workers** (add more, bounded by the downstream resource's capacity) or **apply backpressure** upstream (slow down or reject new work) — silently letting an unbounded queue grow is the failure mode that the next topic addresses directly.

## Under the Hood: How a Worker Pool Actually Schedules Concurrent Jobs

### Concurrency is bounded, not unlimited

A worker pool doesn't spawn one worker per job — it runs a **fixed number of concurrent workers** (or a fixed concurrency limit within one process), each looping: dequeue → process → ack → dequeue again.

```ts
async function runPool(jobQueue: JobQueue, concurrency: number) {
  const workers = Array.from({ length: concurrency }, () => runWorker(jobQueue));
  await Promise.all(workers); // all N workers run concurrently, forever
}
```

Within a single Node.js process, "concurrent" workers pulling from an in-memory or network-backed queue are not running in true parallel (Node is single-threaded for JS execution) — they interleave on the event loop, exactly as covered in `languages/javascript/27-event_loop`: each `await jobQueue.dequeue()` yields control back to the event loop, which is what lets multiple in-flight jobs make progress concurrently even on one thread. True parallelism (using multiple CPU cores) requires actual separate processes or worker threads, each running its own pool.

### Why the concurrency number is a real capacity decision, not a knob to maximize

Every job usually calls out to some shared downstream resource — a database connection pool, a rate-limited third-party API, disk I/O. Setting worker concurrency higher than that resource can sustain doesn't increase throughput; it just moves the bottleneck and adds contention (connection pool exhaustion, 429 rate-limit responses, disk thrashing). The correct concurrency for a worker pool is derived from the *downstream* resource's real capacity, not from how many jobs happen to be waiting.

### Why "at-least-once + idempotent" is the actual contract, not a suggestion

A worker that dequeues a job, crashes mid-processing, and never acks leaves the broker no choice but to redeliver that job to another worker once a visibility timeout expires (this mechanism is broker-specific — RabbitMQ's unacked redelivery, Kafka's consumer group rebalance — but the shape is universal, covered in depth in the Message Brokers group). There is no way to distinguish, from the broker's side, "the worker is just slow" from "the worker died after doing half the work" — so **every** at-least-once system will eventually redeliver a job whose side effects already happened once. Idempotency isn't defensive programming for a rare edge case; it's the only way the "deliver at least once" contract can be made safe to build on top of.

## Common Pitfalls

1. **Writing a job handler that assumes exactly-once delivery** — at-least-once is the norm for most brokers; a non-idempotent handler will eventually double-process a job when a worker crashes mid-run or a redelivery races an ack.
2. **Sizing worker concurrency by "how many jobs are in the queue" instead of "what the downstream resource can sustain"** — over-concurrency just relocates the bottleneck to a database connection pool or a rate limit and adds contention.
3. **No job status tracking for work the caller needs an answer about** — a queue is not a source of historical truth once a job is dequeued; tracked jobs need their own status record.
4. **Assuming jobs for the same entity process in order just because they were enqueued in order** — concurrent workers can reorder them unless the system explicitly partitions by entity or checks versions.
5. **Treating "enqueued successfully" as "will definitely run soon"** — a growing backlog with no backpressure or autoscaling can silently delay jobs by hours; queue depth needs monitoring, not just queue existence.

## Quick Reference

```
Pattern: accept fast -> enqueue -> respond -> worker pool processes independently later

Fire-and-forget:      caller never checks the outcome
Tracked async job:    caller polls a status record
Async + notification: system pushes the outcome back (webhook/WS/another event)

Idempotency: REQUIRED — at-least-once delivery means every handler can run more than once
Concurrency: bounded by the downstream resource's real capacity, not queue depth
Ordering:    not guaranteed across concurrent workers unless partitioned by entity or versioned
Cross-ref:   in-process async mechanics -> languages/javascript/27-event_loop, 29-promises
             unbounded backlog growth   -> next topic, Backpressure
```