# Buffers ## 1. What a Buffer Actually Is A `Buffer` is a fixed-length chunk of raw binary data — a sequence of bytes (integers 0-255), with no concept of encoding until you explicitly ask for one. JavaScript strings are UTF-16 text; a `Buffer` is neither text nor any particular format — it's the same kind of raw memory you'd manipulate in C, exposed to JS. ```js const buf = Buffer.from("hi"); console.log(buf); // — the raw bytes of "hi" in UTF-8 console.log(buf.length); // 2 — byte length, NOT necessarily character count console.log(buf[0]); // 104 — indexing gives you the raw byte value ``` `Buffer` predates JavaScript's own `TypedArray`/`ArrayBuffer` support; today `Buffer` is implemented as a subclass of `Uint8Array`, so every `TypedArray` method (`.slice()`, `.map()` via array-like iteration, etc.) works on it too, plus Node adds buffer-specific conveniences (`.toString()`, `.write()`, numeric read/write helpers). ## 2. Creating Buffers ```js Buffer.alloc(4); // — zero-filled, SAFE Buffer.alloc(4, 1); // — zero-filled then filled with 1 Buffer.allocUnsafe(4); // — UNINITIALIZED memory, FAST but may contain old data Buffer.from("hi", "utf8"); // from a string, with an explicit (or default) encoding Buffer.from([1, 2, 3]); // from an array of byte values Buffer.from(existingBuffer); // COPIES the bytes into a new buffer ``` | Method | Zeroed? | Speed | When to use | |---|---|---|---| | `Buffer.alloc(n)` | yes | slower (has to zero memory) | default choice — safe by default | | `Buffer.allocUnsafe(n)` | **no** | faster | hot paths where you immediately overwrite every byte yourself | | `Buffer.from(...)` | n/a — copies source data | — | wrapping existing data (string, array, another buffer) | ## 3. Reading and Writing Numeric Data Buffers store raw bytes, but binary protocols (network packets, file formats) usually encode multi-byte integers in a specific byte order (endianness). Node's numeric read/write methods handle that explicitly: ```js const buf = Buffer.alloc(4); buf.writeUInt32BE(1, 0); // big-endian: most significant byte first console.log(buf); // buf.writeUInt32LE(1, 0); // little-endian: least significant byte first console.log(buf); // console.log(buf.readUInt32LE(0)); // 1 — read it back out ``` `BE`/`LE` suffixes mean big-endian/little-endian; picking the wrong one against a spec (a file format, a network protocol) silently produces a different number, not an error. ## 4. Strings and Encodings ```js const buf = Buffer.from("café", "utf8"); console.log(buf.length); // 5 — "é" takes 2 bytes in UTF-8, not 1 character = 1 byte console.log(buf.toString("utf8")); // "café" console.log(buf.toString("hex")); // "636166c3a9" console.log(buf.toString("base64")); // "Y2Fmw6k=" Buffer.byteLength("café", "utf8"); // 5 — byte length WITHOUT allocating a buffer ``` `"café".length` (a JS string) is `4` (character count); `buf.length` (a Buffer of that same string) is `5` (byte count) — this mismatch between "characters" and "bytes" is the source of a whole category of buffer bugs with multi-byte characters (see Common Pitfalls). | Encoding | Use case | |---|---| | `"utf8"` (default) | text | | `"hex"` | human-readable representation of raw bytes, 2 hex chars per byte | | `"base64"` | binary-to-text for transport (email, JSON payloads, URLs with `base64url`) | | `"latin1"`/`"binary"` | 1 byte per character, legacy binary-as-string interop | | `"ascii"` | 7-bit text, strips the high bit | ## 5. Slicing: Views, Not Copies ```js const original = Buffer.from("hello world"); const view = original.subarray(0, 5); // "hello" — but NOT a copy view[0] = 72; // 'H' console.log(original.toString()); // "Hello world" — the ORIGINAL changed too ``` `.subarray()` (and the deprecated but still-common `.slice()`, which behaves the same way on Buffers) returns a **view into the same underlying memory**, not a new copy. Writing through the view mutates the original buffer's bytes directly. This is a deliberate performance choice — avoiding a copy — but it means slices are not safe to hand out if the recipient shouldn't be able to affect the source. ```js const copy = Buffer.alloc(5); original.copy(copy, 0, 0, 5); // an ACTUAL copy, into a separate allocation ``` ## 6. Concatenation and Comparison ```js const a = Buffer.from("foo"); const b = Buffer.from("bar"); Buffer.concat([a, b]); // — new allocation, copies both in Buffer.concat([a, b], 5); // explicit total length — truncates/allocates exactly 5 bytes a.equals(b); // false — byte-for-byte comparison Buffer.compare(a, b); // -1, 0, or 1 — for sorting buffers ``` `Buffer.concat` always allocates a brand-new buffer and copies every input into it — unlike `.subarray()`, this one IS a real copy, which matters when building up data incrementally from many small chunks (e.g. reassembling stream chunks) since each concat is O(total size). ![An 8KB pool slab gets carved into three small allocUnsafe buffers via subarray views; a 4th, larger request exceeds the pool and gets its own dedicated allocation](diagrams/01-buffer-pool.svg) ## Under the Hood: Buffers Live Outside the V8 Heap, and Small Ones Share a Pool ### Buffer memory is allocated natively, not on V8's GC'd JS heap A JS object like `{}` or `[1,2,3]` lives on V8's managed heap, sized and garbage-collected by V8 itself — historically capped at a few GB by default. `Buffer` data is different: its actual bytes are allocated as native memory outside that heap (backed by a `Uint8Array`'s underlying `ArrayBuffer`, allocated via Node's own C++ allocator), with only a small JS-visible handle object living on the V8 heap. This is *why* buffers can hold large amounts of binary data (file contents, network payloads) without inflating the V8 heap size or being constrained by heap size limits the way ordinary JS objects are — V8's GC still tracks the handle and eventually frees the native memory once nothing references it, but the bulk allocation itself sits outside GC-managed space. ### `Buffer.poolSize` — small buffers share an 8KB pre-allocated slab ```js console.log(Buffer.poolSize); // 8192 — 8KB, the default pool size ``` For efficiency, `Buffer.allocUnsafe(size)` (and internally, `Buffer.from(string)` for small strings) doesn't ask the OS/allocator for a fresh allocation every time for small buffers. Instead, Node keeps one pre-allocated 8KB "slab," and hands out `.subarray()`-style views into unused regions of that slab for any request smaller than half the pool size (4KB by default). Once the current slab's remaining space is used up, Node allocates a brand-new 8KB slab and continues from there. ```js const a = Buffer.allocUnsafe(10); // carved from the current pool slab const b = Buffer.allocUnsafe(10); // carved from right after `a` in the SAME slab // a and b's underlying memory is adjacent, from the same 8KB block ``` This is exactly why `Buffer.allocUnsafe` is fast (no allocator call for the common case) but "unsafe": the slab is reused memory that previously belonged to other buffers, so a freshly carved-out region can contain **leftover bytes from whatever was there before** until you overwrite it. `Buffer.alloc()` is the same idea but explicitly zeroes the memory first, trading a bit of speed for safety. Requests at or above half the pool size skip the pool entirely and get their own dedicated allocation (`Buffer.allocUnsafeSlow` forces this path explicitly, useful when you need a buffer that does NOT share memory with anything else, e.g. before handing it to native code that will hold onto it independently). ### `ArrayBuffer`/`Uint8Array` common ground Because `Buffer` extends `Uint8Array`, anything that accepts a `Uint8Array`/`ArrayBuffer` (the `crypto` module, `TextEncoder`/`TextDecoder`, `fetch` bodies, worker `postMessage` transfers) accepts a `Buffer` directly — they're genuinely the same underlying kind of object, not just similar APIs. ## Common Pitfalls 1. **Using `Buffer.allocUnsafe()` and forgetting to overwrite every byte** — leftover data from a previous, unrelated buffer's slab region can leak into your buffer; only use it when you're about to fill the entire thing yourself. 2. **Slicing a sensitive buffer and handing the slice to untrusted code** — `.subarray()`/`.slice()` shares memory with the original; mutations through the slice affect the source, and reads through it can expose bytes beyond what you intended if the slice bounds are wrong. 3. **Confusing string `.length` (characters/UTF-16 code units) with `Buffer.length` (bytes)** — multi-byte UTF-8 characters (emoji, accented letters, CJK text) make these numbers diverge; always use `Buffer.byteLength(str)` when you need a byte count from a string. 4. **Picking the wrong endianness (`BE` vs `LE`)** when reading/writing multi-byte integers against a spec that mandates one — produces a different (wrong) number silently, no error thrown. 5. **Building up a large buffer via repeated `Buffer.concat` in a loop** — each call copies everything into a fresh allocation; for many chunks this is O(n²) total copying. Collect chunks in an array and `concat` once, or use a stream. ## Quick Reference ``` Buffer.alloc(n) → zero-filled, safe, slightly slower Buffer.allocUnsafe(n) → uninitialized, fast, may contain stale data Buffer.from(str, enc) → copy bytes from a string/array/another buffer buf.length → byte count (NOT character count for multi-byte text) buf.toString(enc) → decode bytes to a string Buffer.byteLength(str, enc) → byte count of a string WITHOUT allocating buf.subarray(start, end) → VIEW into same memory (mutations affect original) buf.copy(target, ...) → actual copy into a separate allocation Buffer.concat([...]) → new allocation, copies all inputs in Buffer.poolSize → 8192 (8KB) — shared slab size for small allocUnsafe buffers readUInt32BE/LE, writeUInt32BE/LE → explicit-endianness numeric access ```