File Uploads

1How a Browser Sends a File

An HTML form with enctype="multipart/form-data" (or fetch with a FormData body) sends a multipart request: one body, split into parts by a random boundary string, each part with its own headers.

POST /avatars HTTP/1.1
Content-Type: multipart/form-data; boundary=----X9f2
Content-Length: 48211

------X9f2
Content-Disposition: form-data; name="userId"

42
------X9f2
Content-Disposition: form-data; name="file"; filename="me.png"
Content-Type: image/png

<binary bytes...>
------X9f2--

A regular JSON body parser can't read this — you need a multipart parser (multer / busboy in Express, FileInterceptor in NestJS, request.formData() in Next.js route handlers).

2Memory vs Disk vs Stream

import multer from "multer";

// in-memory: req.file.buffer — simple, but every concurrent upload sits in RAM
const memory = multer({ storage: multer.memoryStorage(), limits: { fileSize: 2 * 1024 * 1024 } });

// on disk: streamed to a temp file — RAM stays flat
const disk = multer({ dest: "/tmp/uploads", limits: { fileSize: 50 * 1024 * 1024, files: 1 } });

app.post("/avatars", requireAuth, memory.single("file"), async (req, res) => {
  // req.file: { originalname, mimetype, size, buffer }
});

3Presigned Uploads

For large files the best upload is the one that never passes through your API:

1. client → API:  POST /uploads { filename, contentType, size }
2. API validates (auth, type, size), creates a DB record "pending",
   returns a short-lived presigned PUT URL for key uploads/<uuid>
3. client → S3:   PUT <presigned URL> with the bytes
4. S3 → API (event) or client → API: "upload complete" → verify object, mark "ready"

The presigned URL embeds a signature over the bucket, key, method, content type and expiry — S3 rejects anything that doesn't match. See devops-cloud/aws/04-s3.

4Validate Like the File Is Hostile

Every field of an upload is attacker-controlled: filename, the part's Content-Type, and the bytes.

const SIGNATURES: Record<string, number[]> = {
  "image/png": [0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a],
  "image/jpeg": [0xff, 0xd8, 0xff],
  "application/pdf": [0x25, 0x50, 0x44, 0x46], // %PDF
};
function sniff(buf: Buffer): string | undefined {
  return Object.entries(SIGNATURES).find(([, sig]) => sig.every((b, i) => buf[i] === b))?.[0];
}

5Storing and Referencing Files

Store the bytes in object storage (S3, GCS, Azure Blob) — not in the database, and not on an app server's local disk (which disappears on redeploy and isn't shared between instances). Store metadata in the database:

CREATE TABLE files (
  id UUID PRIMARY KEY,
  owner_id UUID NOT NULL REFERENCES users(id),
  storage_key TEXT NOT NULL,        -- uploads/2024/06/3f1c...png
  original_name TEXT NOT NULL,      -- for display only
  content_type TEXT NOT NULL,       -- the SNIFFED type
  size_bytes BIGINT NOT NULL,
  sha256 TEXT NOT NULL,             -- dedup / integrity
  status TEXT NOT NULL DEFAULT 'pending',  -- pending → ready | rejected
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

Heavy post-processing (thumbnails, video transcoding, virus scanning) belongs in a background job (16-background_jobs/), not in the request.

Under the HoodStreaming Multipart Parsing▾

A multipart parser like busboy never needs the whole body in memory. It reads the request stream chunk by chunk and runs a state machine that searches for the boundary sequence (\r\n--<boundary>) — using a Boyer–Moore–Horspool style search so it can scan large chunks quickly and handle a boundary split across two chunks. Between boundaries it parses the part headers, then emits a readable file stream for that part. You pipe that stream wherever it should go (disk, S3 multipart upload, a hash function).

Backpressure makes this safe: if the destination is slower than the client, the destination's write() returns false, the pipe pauses the file stream, busboy stops reading the request socket, TCP's receive window fills, and the client's upload slows down — instead of your process buffering gigabytes. Size limits are enforced as bytes flow: when fileSize is exceeded, the parser truncates the stream and flags it, so you can abort without ever having read the whole file.

⚠️ Common Pitfalls

1. No size limit — one request can exhaust memory or disk.
2. Trusting file.mimetype or the extension — both are client-supplied.
3. Using originalname in the storage path — path traversal and collisions.
4. Saving to local disk on a multi-instance deployment — files "disappear" depending on which instance serves the request.
5. Serving uploads from your main domain with an inline content type — uploaded HTML/SVG executes as your site (stored XSS).
6. Processing synchronously in the request — timeouts on large files; move to a job.
7. Forgetting cleanup — temp files and abandoned "pending" uploads accumulate.

📋 Quick Reference

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: fw_file_uploads_theory.md · generated by tools/build_lessons.py