Storage and StatefulSets

1Volumes

A container's filesystem disappears when the container restarts. Volumes attach storage to a Pod:

2PersistentVolumes, Claims and StorageClasses

• PersistentVolume (PV): a piece of storage in the cluster (an EBS volume, an EFS share, a Ceph RBD image).
• PersistentVolumeClaim (PVC): a Pod's request for storage ("20 Gi, ReadWriteOnce, class gp3").
• StorageClass: how to dynamically provision PVs via a CSI driver.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: { name: gp3 }
provisioner: ebs.csi.aws.com
parameters: { type: gp3, encrypted: "true" }
reclaimPolicy: Retain                    # keep the disk when the PVC is deleted
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer  # create the disk in the AZ where the Pod lands
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: uploads }
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: gp3
  resources: { requests: { storage: 20Gi } }

Access modes: ReadWriteOnce (one node — block storage like EBS), ReadWriteOncePod, ReadOnlyMany, ReadWriteMany (many nodes — network file systems like EFS/NFS/CephFS). Block volumes are zonal: a Pod using an EBS volume can only run in that volume's AZ.

Reclaim policy Delete (default for dynamic provisioning) deletes the disk with the PVC — use Retain for important data. VolumeSnapshots (CSI) give point-in-time copies; tools like Velero back up both resources and volumes.

3StatefulSets

Deployments treat Pods as interchangeable cattle. A StatefulSet gives each Pod a stable identity:

• stable names: postgres-0, postgres-1, postgres-2
• stable DNS via a headless Service: postgres-0.postgres.db.svc.cluster.local
• one PVC per Pod from volumeClaimTemplates, re-attached to the same ordinal after rescheduling
• ordered start-up, scaling and rolling updates (by default): -0 before -1; updates in reverse order; partition for staged rollouts

apiVersion: v1
kind: Service
metadata: { name: postgres }
spec:
  clusterIP: None                # headless
  selector: { app: postgres }
  ports: [{ port: 5432 }]
---
apiVersion: apps/v1
kind: StatefulSet
metadata: { name: postgres }
spec:
  serviceName: postgres
  replicas: 3
  selector: { matchLabels: { app: postgres } }
  template:
    metadata: { labels: { app: postgres } }
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: postgres
          image: postgres:17
          ports: [{ containerPort: 5432 }]
          volumeMounts: [{ name: data, mountPath: /var/lib/postgresql/data }]
  volumeClaimTemplates:
    - metadata: { name: data }
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: gp3
        resources: { requests: { storage: 50Gi } }

A StatefulSet does not make an application clustered — replication, leader election and failover are the app's job. That's why production databases on Kubernetes use operators (CloudNativePG, Strimzi for Kafka, Redis operators) that encode backup, failover and upgrade procedures — or a managed service (RDS, Cloud SQL), which is often the better default (devops-cloud/aws/06-rds_and_databases).

4Stateful Workload Practices

• PodDisruptionBudget (minAvailable: 2) so node drains don't take down quorum.
• Anti-affinity / topology spread across zones; remember zonal volumes pin Pods to their AZ.
• Backups tested by restoring; snapshots + logical backups (pg_dump) + WAL archiving.
• Volume expansion: edit the PVC size (if the class allows); StatefulSet templates are immutable — expand each PVC.

Under the HoodDynamic Provisioning and Attachment▾

When a Pod with a PVC is created, the PVC is Pending. With WaitForFirstConsumer, the scheduler first picks a node (considering the class's topology), then annotates the PVC with that node; the external-provisioner sidecar of the CSI controller sees the PVC and calls the driver's CreateVolume (e.g. EBS CreateVolume in that AZ), then creates a PV and binds it to the PVC (a one-to-one, exclusive binding). The attach/detach controller creates a VolumeAttachment, and the CSI controller's external-attacher calls ControllerPublishVolume (attaching the EBS disk to the EC2 instance). On the node, the kubelet calls the CSI node plugin's NodeStageVolume (format if needed, mount to a global path) and NodePublishVolume (bind-mount into the Pod's directory), and only then starts the containers. When the Pod moves to another node, the volume must be detached first — which is why a Pod on a dead node can be stuck until the attachment times out or is force-detached.

⚠️ Common Pitfalls

1. Databases in Deployments with a shared PVC.
2. reclaimPolicy: Delete on irreplaceable data.
3. Immediate binding creating volumes in an AZ without capacity for the Pod.
4. No PDB — a cluster upgrade drains all replicas at once.
5. Assuming StatefulSet = HA database without an operator.
6. Never testing restores.

📋 Quick Reference

volumes: emptyDir · configMap/secret/projected · hostPath (avoid) · PVC
PV ↔ PVC (bind) · StorageClass + CSI (provisioner, reclaimPolicy Retain, WaitForFirstConsumer, expansion)
access: RWO (block, zonal) · RWX (EFS/NFS) · snapshots · Velero
StatefulSet: stable names + headless Service DNS + volumeClaimTemplates + ordered rollout · use operators · PDBs

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: k8s_storage_and_statefulsets_theory.md · generated by tools/build_lessons.py