Asynchronous Processing

1What "Asynchronous Processing" Means at the System Level

This topic is about a specific pattern built on top of asynchronous communication (previous topic): work that is deliberately deferred, queued, and executed later — separate from the request that triggered it. The classic shape:

1. A request arrives (HTTP call, event, upload, etc.)
2. The service does the minimum work needed to accept it, then responds immediately
3. The actual work is handed off — to a background job, a queue, a worker pool
4. A worker picks up the deferred work and executes it, independently of the original caller

This is different from the JavaScript-level concept of an async function or a Promise (covered fully in languages/javascript/27-event_loop and languages/javascript/29-promises) — those are the mechanism a single process uses to avoid blocking on I/O. This topic is about deferring work across a system boundary, often to an entirely different process, possibly minutes or hours later.

2Why Defer Work At All

• Fast responses. A user uploading a video shouldn't wait for transcoding to finish before seeing "upload received."
• Absorbing spikes. A flash sale generates 50,000 order confirmations in a minute; sending them all synchronously would overwhelm the email provider and the checkout path with it.
• Resource isolation. A CPU-heavy report generation job shouldn't run on the same request thread pool that serves login requests.
• Resilience. If the email provider is down for ten minutes, queued jobs simply wait — they don't turn into failed checkouts.

3The Building Blocks: Job Queue + Worker Pool

// producer side: enqueue work instead of doing it inline
async function handleUpload(fileId: string) {
  await storage.save(fileId);
  await jobQueue.enqueue("transcode-video", { fileId });
  return { status: "accepted", fileId };
}

// worker side: a pool of workers pulls jobs and executes them independently
async function runWorker(jobQueue: JobQueue) {
  while (true) {
    const job = await jobQueue.dequeue(); // blocks until work is available
    try {
      await transcodeVideo(job.payload.fileId);
      await jobQueue.ack(job.id);
    } catch (err) {
      await jobQueue.nack(job.id); // re-queue or dead-letter, see Dead Letter Queues
    }
  }
}

A worker pool runs N workers concurrently pulling from the same queue, so throughput scales by adding workers (bounded by the downstream resource — a database, an external API — not by the queue itself).

4Fire-and-Forget vs Tracked Async Work

Not all deferred work is equal — it matters whether anyone needs to know the outcome:

A tracked job needs a job status record (pending → running → completed/failed) that the caller can poll (GET /jobs/{id}) or be notified about (webhook, WebSocket, another queue message) — the queue itself usually doesn't retain enough history for this once a job is dequeued.

5Idempotency: The Non-Negotiable Requirement

Because queues generally offer at-least-once delivery (a job can be delivered more than once — see the Message Brokers group's Delivery Guarantees topic), every job handler must be idempotent: running it twice with the same input must produce the same end state as running it once.

// NOT idempotent — running twice charges the customer twice
async function chargeCustomer(orderId: string, amount: number) {
  await paymentGateway.charge(amount); // no way to detect "already charged"
}

// idempotent — a unique key lets the downstream system detect duplicates
async function chargeCustomerIdempotent(orderId: string, amount: number) {
  await paymentGateway.charge(amount, { idempotencyKey: `charge-${orderId}` });
  // the gateway itself recognizes a repeated idempotencyKey and returns the
  // ORIGINAL result instead of charging again
}

Idempotency has to be designed in at the handler (or the downstream API), not assumed away — "the queue probably won't redeliver" is not a guarantee any real broker makes.

async function sendReceipt(job) {
  await email.send(job.userEmail, receipt(job.orderId));
}
// redelivered after a crash → customer gets 2 receipts

async function sendReceipt(job) {
  const first = await db.insertIgnore('sent_receipts', job.orderId);
  if (!first) return;          // already handled
  await email.send(job.userEmail, receipt(job.orderId),
                   { idempotencyKey: `receipt-${job.orderId}` });
}

6Ordering and Concurrency

Workers processing jobs concurrently means jobs for the same logical entity can run out of order or in parallel unless something prevents it:

// two jobs for the SAME order running concurrently on different workers
// can race: job A sets status="shipped", job B (an older event) then
// overwrites it back to status="paid"

Common fixes: partition work so all jobs for one entity always go to the same worker (this is exactly what Kafka's partitioning does — see the Message Brokers group), include a version/timestamp check before applying an update ("only apply this if it's newer than the current state"), or use a per-entity lock while processing.

7Backpressure and Scaling the Worker Pool

If jobs arrive faster than workers can process them, the queue backs up. This is the subject of the next topic in full, but the shape here: either scale workers (add more, bounded by the downstream resource's capacity) or apply backpressure upstream (slow down or reject new work) — silently letting an unbounded queue grow is the failure mode that the next topic addresses directly.

async function runPool(jobQueue: JobQueue, concurrency: number) {
  const workers = Array.from({ length: concurrency }, () => runWorker(jobQueue));
  await Promise.all(workers); // all N workers run concurrently, forever
}

⚠️ Common Pitfalls

1. Writing a job handler that assumes exactly-once delivery — at-least-once is the norm for most brokers; a non-idempotent handler will eventually double-process a job when a worker crashes mid-run or a redelivery races an ack.
2. Sizing worker concurrency by "how many jobs are in the queue" instead of "what the downstream resource can sustain" — over-concurrency just relocates the bottleneck to a database connection pool or a rate limit and adds contention.
3. No job status tracking for work the caller needs an answer about — a queue is not a source of historical truth once a job is dequeued; tracked jobs need their own status record.
4. Assuming jobs for the same entity process in order just because they were enqueued in order — concurrent workers can reorder them unless the system explicitly partitions by entity or checks versions.
5. Treating "enqueued successfully" as "will definitely run soon" — a growing backlog with no backpressure or autoscaling can silently delay jobs by hours; queue depth needs monitoring, not just queue existence.

📋 Quick Reference

Pattern: accept fast -> enqueue -> respond -> worker pool processes independently later

Fire-and-forget:      caller never checks the outcome
Tracked async job:    caller polls a status record
Async + notification: system pushes the outcome back (webhook/WS/another event)

Idempotency: REQUIRED — at-least-once delivery means every handler can run more than once
Concurrency: bounded by the downstream resource's real capacity, not queue depth
Ordering:    not guaranteed across concurrent workers unless partitioned by entity or versioned
Cross-ref:   in-process async mechanics -> languages/javascript/27-event_loop, 29-promises
             unbounded backlog growth   -> next topic, Backpressure

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: dist_asynchronous_processing_theory.md · generated by tools/build_lessons.py