# Streams — Part 2: Backpressure and Internals

## 1. The Problem Backpressure Solves

If a Readable stream produces data faster than a Writable stream can consume it (e.g. reading a fast local file while writing over a slow network connection), and nothing throttles the producer, data piles up in memory without bound — the exact problem streaming was supposed to avoid in the first place. **Backpressure** is the mechanism that throttles a fast source to match a slow destination.

## 2. `highWaterMark` — the Internal Buffer Threshold

```js
const fs = require("fs");

const readable = fs.createReadStream("data.txt", {
  highWaterMark: 64 * 1024, // 64KB — default for byte streams
});
```

Every stream maintains an internal buffer. `highWaterMark` is the size (in bytes for normal streams, in **number of objects** for object-mode streams, default `16`) at which the stream's internal buffer is considered "full" — the point at which the stream signals the producer to slow down. It's a threshold, not a hard cap — a little overflow past it is normal.

| Stream kind | Default `highWaterMark` |
|---|---|
| Byte stream (Buffer/string) | 64KB (16KB before Node 18 for some stream kinds — check your version) |
| Object-mode stream | 16 objects |

## 3. Paused vs Flowing Mode (Readable Streams)

```js
const fs = require("fs");
const readable = fs.createReadStream("data.txt");

// FLOWING mode: data pushed automatically, as fast as it's available
readable.on("data", (chunk) => console.log("flowing:", chunk.length));

// vs PAUSED mode: you must explicitly pull data
const readable2 = fs.createReadStream("data.txt");
readable2.on("readable", () => {
  let chunk;
  while ((chunk = readable2.read()) !== null) {
    console.log("paused-mode pull:", chunk.length);
  }
});
```

| | Paused mode | Flowing mode |
|---|---|---|
| Default state | every Readable starts here | entered by adding a `'data'` listener, or calling `.pipe()`/`.resume()` |
| How data arrives | you call `.read()` explicitly to pull it | pushed to you automatically via `'data'` events |
| Where data sits meanwhile | in the internal buffer, waiting to be pulled | delivered as soon as it's read internally |
| Natural throttling | yes — you control the pace by choosing when to call `.read()` | no — the source pushes at its own pace unless something pauses it |

A stream can be switched back to paused mode with `.pause()`, and flowing mode is exactly the state `.pipe()` needs to actively manage, since nothing else is naturally slowing the source down once it's flowing.

## 4. What `.pipe()` Actually Does Under the Hood

`.pipe()` is not magic — it's a fairly small, well-defined orchestration of the primitives above:

```
1. src.on('data', chunk => {
     const ok = dest.write(chunk);       // write() returns a boolean
     if (!ok) {
       src.pause();                       // internal buffer is over highWaterMark — STOP the source
     }
   });

2. dest.on('drain', () => {
     src.resume();                        // buffer has drained back down — SAFE to resume
   });

3. src.on('end', () => dest.end());        // no more data — close the destination too
```

`write()`'s return value is the actual backpressure signal: `true` means "keep sending, I can take more," `false` means "my internal buffer just exceeded my `highWaterMark` — stop until I tell you otherwise." The `'drain'` event fires once the writable's internal buffer has emptied back out, which is `.pipe()`'s cue to call `src.resume()`.

![Readable writes into Writable; write() returns false past highWaterMark, pipe() pauses the source until 'drain' fires, then resumes](diagrams/01-backpressure-cycle.svg)

## 5. Manual Backpressure (What `.pipe()` Saves You From Writing)

```js
const fs = require("fs");

function copyWithBackpressure(srcPath, destPath, onDone) {
  const src = fs.createReadStream(srcPath);
  const dest = fs.createWriteStream(destPath);

  src.on("data", (chunk) => {
    const canKeepGoing = dest.write(chunk);
    if (!canKeepGoing) {
      src.pause(); // destination's buffer is full — stop pulling more
      dest.once("drain", () => {
        src.resume(); // destination caught up — safe to continue
      });
    }
  });

  src.on("end", () => dest.end());
  dest.on("finish", onDone);
}
```

This is functionally what `.pipe()` does internally — writing it manually makes the pause/drain/resume cycle concrete instead of hidden behind one method call.

## 6. Building a Custom Readable/Writable Stream

```js
const { Readable, Writable } = require("stream");

class CounterStream extends Readable {
  constructor(max, options) {
    super(options);
    this.current = 0;
    this.max = max;
  }
  _read() {
    // called whenever the internal consumer wants more data
    if (this.current >= this.max) {
      this.push(null); // signal end-of-stream
      return;
    }
    this.push(String(this.current++));
  }
}

class LoggingWritable extends Writable {
  _write(chunk, encoding, callback) {
    console.log("received:", chunk.toString());
    callback(); // MUST be called to signal this write is done — drives backpressure timing
  }
}

new CounterStream(5).pipe(new LoggingWritable());
```

`_read()` and `_write()` are the required low-level implementations; the base `Readable`/`Writable` classes handle buffering, `highWaterMark` tracking, and event emission around them. Calling `callback()` late in `_write()` is itself a valid (if unusual) way to intentionally slow down a producer — the next write won't be considered "flushed" until then.

## Under the Hood: Backpressure Is the Real Point of Streams

The paused/flowing state machine and `highWaterMark` aren't incidental implementation details — they're the actual mechanism that makes streaming safe under mismatched producer/consumer speeds. Without it, "streaming" would just mean "chunking with no throttling," which degrades to the same unbounded-memory-growth problem streams exist to prevent, just spread out over more event-loop turns instead of happening all at once. The `write() → false → pause() → 'drain' → resume()` cycle in section 4 is the entire mechanism; every higher-level stream utility (`.pipe()`, `stream.pipeline()`, async iteration over a Readable) is built on exactly that same feedback loop underneath.

## Common Pitfalls

1. **Attaching a `'data'` listener and then later needing paused-mode behavior** — once a stream starts flowing, switching back requires an explicit `.pause()`; mixing paradigms carelessly causes missed or duplicated chunks.
2. **Writing to a destination in a loop without checking `write()`'s return value** — this is exactly the backpressure violation `.pipe()` exists to prevent; manual loops need the same check.
3. **Forgetting `callback()` in a custom `_write()`/`_transform()` implementation** — the stream will stall forever waiting for it, since it's the internal signal that a chunk has been fully handled.
4. **Not handling `'error'` on both ends of a pipe** — `.pipe()` does NOT automatically forward errors from source to destination or vice versa; unhandled stream errors crash the process. Prefer `stream.pipeline()` (from `require('stream')`), which handles error propagation and cleanup automatically.
5. **Assuming `highWaterMark` is a hard cap** — it's a threshold that triggers backpressure signaling, not a strict ceiling; a stream can briefly hold slightly more.

## Quick Reference

```
Readable / Writable / Duplex / Transform → the four stream types
highWaterMark    → internal buffer threshold (bytes, or object count in object mode)
paused mode       → pull manually via .read()
flowing mode        → pushed automatically via 'data' events
write() returns false → destination's buffer exceeded highWaterMark, PAUSE the source
'drain' event           → buffer emptied, safe to RESUME
.pipe()                   → automates the write()/pause()/drain()/resume() cycle
stream.pipeline()           → pipe() + automatic error propagation and cleanup
```