# Storage and StatefulSets

## 1. Volumes

A container's filesystem disappears when the container restarts. **Volumes** attach storage to a Pod:

| Volume | Lifetime | Use |
|---|---|---|
| `emptyDir` | Pod (survives container restarts) | scratch space, sharing files between containers; `medium: Memory` for tmpfs |
| `configMap` / `secret` / `projected` | Pod | config files (`03-configmaps_and_secrets/`), ServiceAccount tokens |
| `hostPath` | node | node agents only — a security risk for apps |
| `persistentVolumeClaim` | independent of the Pod | databases, uploads, anything that must survive rescheduling |

## 2. PersistentVolumes, Claims and StorageClasses

- **PersistentVolume (PV)**: a piece of storage in the cluster (an EBS volume, an EFS share, a Ceph RBD image).
- **PersistentVolumeClaim (PVC)**: a Pod's request for storage ("20 Gi, ReadWriteOnce, class gp3").
- **StorageClass**: how to **dynamically provision** PVs via a **CSI driver**.

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: { name: gp3 }
provisioner: ebs.csi.aws.com
parameters: { type: gp3, encrypted: "true" }
reclaimPolicy: Retain                    # keep the disk when the PVC is deleted
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer  # create the disk in the AZ where the Pod lands
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: uploads }
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: gp3
  resources: { requests: { storage: 20Gi } }
```

**Access modes**: `ReadWriteOnce` (one node — block storage like EBS), `ReadWriteOncePod`, `ReadOnlyMany`, `ReadWriteMany` (many nodes — network file systems like EFS/NFS/CephFS). Block volumes are **zonal**: a Pod using an EBS volume can only run in that volume's AZ.

**Reclaim policy** `Delete` (default for dynamic provisioning) deletes the disk with the PVC — use `Retain` for important data. **VolumeSnapshots** (CSI) give point-in-time copies; tools like **Velero** back up both resources and volumes.

## 3. StatefulSets

Deployments treat Pods as interchangeable cattle. A **StatefulSet** gives each Pod a **stable identity**:

- stable names: `postgres-0`, `postgres-1`, `postgres-2`
- stable DNS via a **headless Service**: `postgres-0.postgres.db.svc.cluster.local`
- **one PVC per Pod** from `volumeClaimTemplates`, re-attached to the same ordinal after rescheduling
- **ordered** start-up, scaling and rolling updates (by default): `-0` before `-1`; updates in reverse order; `partition` for staged rollouts

```yaml
apiVersion: v1
kind: Service
metadata: { name: postgres }
spec:
  clusterIP: None                # headless
  selector: { app: postgres }
  ports: [{ port: 5432 }]
---
apiVersion: apps/v1
kind: StatefulSet
metadata: { name: postgres }
spec:
  serviceName: postgres
  replicas: 3
  selector: { matchLabels: { app: postgres } }
  template:
    metadata: { labels: { app: postgres } }
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: postgres
          image: postgres:17
          ports: [{ containerPort: 5432 }]
          volumeMounts: [{ name: data, mountPath: /var/lib/postgresql/data }]
  volumeClaimTemplates:
    - metadata: { name: data }
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: gp3
        resources: { requests: { storage: 50Gi } }
```

A StatefulSet does **not** make an application clustered — replication, leader election and failover are the app's job. That's why production databases on Kubernetes use **operators** (CloudNativePG, Strimzi for Kafka, Redis operators) that encode backup, failover and upgrade procedures — or a managed service (RDS, Cloud SQL), which is often the better default (`devops-cloud/aws/06-rds_and_databases`).

## 4. Stateful Workload Practices

- **PodDisruptionBudget** (`minAvailable: 2`) so node drains don't take down quorum.
- Anti-affinity / topology spread across zones; remember zonal volumes pin Pods to their AZ.
- Backups tested by restoring; snapshots + logical backups (pg_dump) + WAL archiving.
- Volume expansion: edit the PVC size (if the class allows); StatefulSet templates are immutable — expand each PVC.

## Under the Hood: Dynamic Provisioning and Attachment

When a Pod with a PVC is created, the PVC is `Pending`. With `WaitForFirstConsumer`, the scheduler first picks a node (considering the class's topology), then annotates the PVC with that node; the **external-provisioner** sidecar of the CSI controller sees the PVC and calls the driver's `CreateVolume` (e.g. EBS `CreateVolume` in that AZ), then creates a PV and binds it to the PVC (a one-to-one, exclusive binding). The **attach/detach controller** creates a `VolumeAttachment`, and the CSI controller's external-attacher calls `ControllerPublishVolume` (attaching the EBS disk to the EC2 instance). On the node, the kubelet calls the CSI node plugin's `NodeStageVolume` (format if needed, mount to a global path) and `NodePublishVolume` (bind-mount into the Pod's directory), and only then starts the containers. When the Pod moves to another node, the volume must be detached first — which is why a Pod on a dead node can be stuck until the attachment times out or is force-detached.

![A StatefulSet with a headless Service creates postgres-0, -1 and -2, each with its own PVC from volumeClaimTemplates; a StorageClass and CSI driver dynamically provision zonal disks, which are attached to the node and mounted into each Pod](diagrams/01-statefulset-storage.svg)

## Common Pitfalls

1. **Databases in Deployments** with a shared PVC.
2. **`reclaimPolicy: Delete`** on irreplaceable data.
3. **`Immediate` binding** creating volumes in an AZ without capacity for the Pod.
4. **No PDB** — a cluster upgrade drains all replicas at once.
5. **Assuming StatefulSet = HA database** without an operator.
6. **Never testing restores**.

## Quick Reference

```
volumes: emptyDir · configMap/secret/projected · hostPath (avoid) · PVC
PV ↔ PVC (bind) · StorageClass + CSI (provisioner, reclaimPolicy Retain, WaitForFirstConsumer, expansion)
access: RWO (block, zonal) · RWX (EFS/NFS) · snapshots · Velero
StatefulSet: stable names + headless Service DNS + volumeClaimTemplates + ordered rollout · use operators · PDBs