# Deployment Strategies

## 1. Why Strategy Matters

Most production incidents are caused by **changes**. A deployment strategy decides how a new version replaces the old one — and therefore how many users a bad release can hurt, how fast you notice, and how fast you can undo it.

| Strategy | How | Downtime | Rollback speed | Extra capacity | Blast radius |
|---|---|---|---|---|---|
| **recreate** | stop all old, start all new | yes | slow (redeploy) | none | 100% |
| **rolling** | replace instances a few at a time | no | medium (roll back again) | small (surge) | grows during rollout |
| **blue-green** | stand up full new env, switch traffic at once | no | instant (switch back) | 2× | 100% after switch |
| **canary** | send a small % of traffic to new version, increase gradually | no | fast (route back) | small | small (1–5% first) |
| **feature flags / dark launch** | deploy code switched off, enable per user/cohort | no | instant (flip flag) | none | as targeted |
| **shadow / mirroring** | copy live traffic to new version, discard its responses | no | n/a (not serving) | ~2× for that service | zero for users |

## 2. Rolling Updates

```yaml
# Kubernetes Deployment
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 25%        # up to 25% extra pods during the rollout
    maxUnavailable: 0    # never drop below desired capacity
```

Kubernetes starts new pods, waits for **readiness**, then terminates old ones. Requirements: readiness probes that reflect real readiness, graceful shutdown, and **both versions running at once** must be compatible (API and database).

## 3. Blue-Green

Two identical environments; **blue** serves traffic, **green** gets the new version and is tested in place; then the router/LB switches 100% to green. Rollback = switch back (blue is still warm). Costs double capacity during the switch, and **stateful parts are shared** — the database can't be "switched back", so schema changes must be compatible with both colors.

## 4. Canary Releases

```
v2 gets 1% → watch error rate, latency, business metrics for N minutes
         → 5% → 25% → 50% → 100%
any step: metrics worse than v1 baseline → automatic rollback to 0%
```

- Traffic shifting via a service mesh, ingress controller, or LB weighted target groups (Argo Rollouts, Flagger, AWS CodeDeploy).
- **Automated analysis** compares canary vs baseline (not vs a fixed threshold only): error ratio, p99 latency, saturation, and a business KPI (checkout conversion).
- Canary by **cohort** (internal users, one region, one percentage of users with sticky assignment) so a user doesn't flip between versions on every request.

## 5. Feature Flags: Decouple Deploy from Release

Deploying code and **releasing a feature** become separate steps:

```ts
if (flags.isEnabled("new-checkout", { userId, country })) return newCheckout(cart);
return legacyCheckout(cart);
```

Ship dark, enable for employees → 1% → 10% → everyone; kill switch if needed. Costs: flag debt (remove flags after rollout), combinatorial testing, and the flag service becomes critical infrastructure.

## 6. Database Changes: Expand / Contract

Every zero-downtime strategy runs **old and new code at the same time**, so schema changes must be backward compatible:

```
1. expand:   add the new column/table (nullable / with default) — old code ignores it
2. migrate:  deploy code that writes BOTH old and new; backfill existing rows in batches
3. switch:   deploy code that reads the new column
4. contract: once no code uses the old column, drop it (in a later release)
```

Never rename or drop a column in the same release that stops using it.

## 7. Rollback vs Roll Forward

- **Rollback**: return to the previous artifact (fast if the artifact is immutable and the schema is compatible).
- **Roll forward**: ship a fix. Necessary when a migration can't be reversed.
- Decide **before** deploying how you'd undo it — that's what the expand/contract discipline buys.

## Under the Hood: How Weighted Canary Routing Works

At L7 (Envoy/NGINX/ALB), a route has multiple **weighted clusters**: for each new request, the proxy draws a random number (or hashes a sticky key like a user ID or cookie) and picks `v1` with probability 0.95 and `v2` with 0.05. A rollout controller (Argo Rollouts, Flagger) is a loop: update weights → wait for the analysis interval → query the metrics backend (`sum(rate(http_requests_total{version="v2",code=~"5.."}[5m])) / sum(rate(http_requests_total{version="v2"}[5m]))`) → compare with thresholds or the baseline → promote or abort. With pod-count-based canaries (no mesh), traffic share is only approximately `canaryPods / totalPods` and can't go below one pod's share — the reason fine-grained canaries need L7 traffic splitting.

![Rolling replaces pods a few at a time, blue-green switches all traffic between two full environments, and canary shifts traffic from 1 to 100 percent while automated analysis compares error rate and latency against the baseline and rolls back on regression](diagrams/01-strategies.svg)

## Common Pitfalls

1. **Incompatible schema changes** deployed with a zero-downtime strategy.
2. **Readiness probes that lie** — rolling updates send traffic to pods that aren't ready.
3. **Canary without automated analysis** — nobody watches the dashboard at 2 a.m.
4. **Non-sticky canary** — users bounce between versions (broken sessions, inconsistent UI).
5. **Blue-green with a shared, mutated database** treated as if it could be switched back.
6. **Feature flags never removed**.

## Quick Reference

| Need | Use |
|---|---|
| simplest, downtime OK | recreate |
| default for stateless services | rolling (maxUnavailable 0) |
| instant cutover and rollback | blue-green |
| limit blast radius with real traffic | canary + automated analysis |
| release independent of deploy | feature flags |
| test with production traffic safely | shadow traffic |
| schema changes | expand → migrate → switch → contract |