Pods and Deployments

1Pods

A Pod is the smallest deployable unit: one or more containers that share a network namespace (one IP, localhost between them), volumes and lifecycle, always scheduled together on one node. Most Pods run one app container, sometimes with sidecars (log shippers, proxies) or init containers (run to completion before the app starts — migrations, waiting for dependencies).

apiVersion: v1
kind: Pod
metadata: { name: api, labels: { app: api } }
spec:
  initContainers:
    - name: wait-for-db
      image: busybox:1.36
      command: ["sh", "-c", "until nc -z postgres 5432; do sleep 2; done"]
  containers:
    - name: api
      image: ghcr.io/petpal/api:1.4.2
      ports: [{ containerPort: 8080, name: http }]
      resources:
        requests: { cpu: 250m, memory: 256Mi }   # used for scheduling
        limits:   { memory: 512Mi }              # OOMKilled above this
      readinessProbe:
        httpGet: { path: /ready, port: http }
        periodSeconds: 5
      livenessProbe:
        httpGet: { path: /health, port: http }
        initialDelaySeconds: 10
        periodSeconds: 10
        failureThreshold: 3
      securityContext:
        runAsNonRoot: true
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities: { drop: ["ALL"] }

Resources

• requests: what the scheduler reserves; the basis for CPU shares and autoscaling percentages.
• limits: CPU limits throttle; memory limits kill (OOMKilled, exit code 137).
• QoS classes: Guaranteed (requests = limits for all), Burstable, BestEffort (none) — BestEffort is evicted first under node pressure.
• Common practice: always set requests; set a memory limit; be careful with CPU limits (throttling hurts latency).

Probes

Liveness must check only the process itself — never dependencies like the database, or a DB blip restarts every Pod.

Pod lifecycle

Phases: Pending → Running → Succeeded/Failed. Container states: Waiting (ContainerCreating, ImagePullBackOff, CrashLoopBackOff), Running, Terminated. Restart policy Always (Deployments), OnFailure/Never (Jobs). On termination Kubernetes sends SIGTERM, waits terminationGracePeriodSeconds (30 s default) and then SIGKILLs — apps must handle SIGTERM by finishing in-flight requests (devops-cloud/docker/ on PID 1 signal handling).

livenessProbe:
  httpGet: { path: /health/deep, port: 8080 }  # pings Postgres
# DB blips for 30 s → every pod fails liveness
# → kubelet restarts ALL pods → outage

livenessProbe:
  httpGet: { path: /livez, port: 8080 }   # event loop alive?
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }  # deps OK? else leave endpoints

2Deployments

A Deployment manages ReplicaSets, which keep N identical Pods running. Changing the Pod template (new image, env, resources) creates a new ReplicaSet and performs a rolling update:

apiVersion: apps/v1
kind: Deployment
metadata: { name: api }
spec:
  replicas: 4
  revisionHistoryLimit: 5
  selector: { matchLabels: { app: api } }
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1          # at most 5 Pods during rollout
      maxUnavailable: 0    # never fewer than 4 ready
  minReadySeconds: 10
  progressDeadlineSeconds: 300
  template:
    metadata: { labels: { app: api } }
    spec:
      containers:
        - name: api
          image: ghcr.io/petpal/api:1.4.3

kubectl set image deploy/api api=ghcr.io/petpal/api:1.4.3   # (prefer editing the file + apply)
kubectl rollout status deploy/api
kubectl rollout history deploy/api
kubectl rollout undo deploy/api --to-revision=3
kubectl scale deploy/api --replicas=6
kubectl rollout restart deploy/api        # recreate Pods (e.g. to pick up a changed Secret)

Recreate strategy kills all old Pods first (downtime; for apps that can't run two versions). Canary and blue/green need extra tooling (Argo Rollouts, Flagger, or two Deployments + traffic splitting — devops-cloud/cicd/).

3Jobs, CronJobs, DaemonSets

• Job: run Pods to completion (completions, parallelism, backoffLimit, activeDeadlineSeconds, ttlSecondsAfterFinished).
• CronJob: Jobs on a schedule (schedule: "0 3 * * *", concurrencyPolicy: Forbid, timeZone).
• DaemonSet: one Pod per (matching) node — node exporters, log agents, CNI.

Under the HoodHow a Rolling Update Proceeds▾

The Deployment controller computes a hash of the Pod template (pod-template-hash label) and creates a new ReplicaSet for it. It then scales the new ReplicaSet up and the old one down in steps bounded by maxSurge (extra Pods allowed above replicas) and maxUnavailable (Pods allowed to be not-ready). With 4 replicas, surge 1, unavailable 0: create 1 new Pod (5 total) → wait until it's Ready for minReadySeconds → scale old to 3 → create another new Pod → … Readiness probes gate every step, so a broken version stalls the rollout with old Pods still serving; if no progress happens within progressDeadlineSeconds, the Deployment condition becomes Progressing=False (CI should fail and roll back). Old ReplicaSets are kept at 0 replicas (up to revisionHistoryLimit) — rollout undo simply scales an old one back up.

⚠️ Common Pitfalls

1. No readiness probe — traffic hits Pods before they're ready; rollouts can't detect broken versions.
2. Liveness probe checking dependencies — cascading restarts.
3. No requests — bad scheduling, noisy neighbours, broken autoscaling.
4. :latest tags — no rollbacks, unpredictable nodes.
5. Ignoring SIGTERM — dropped requests during every deploy.
6. Running as root with a writable filesystem.

📋 Quick Reference

Pod: containers + init + sidecars share IP/volumes · requests (schedule) vs limits (CPU throttle, mem OOMKill)
probes: startup · readiness (endpoints) · liveness (restart; no deps)
Deployment → ReplicaSet → Pods · RollingUpdate maxSurge/maxUnavailable · rollout status/history/undo
Job · CronJob · DaemonSet · SIGTERM → grace period → SIGKILL

1Why Services

Pods are ephemeral: they get new IPs every time they're recreated. A Service gives a stable virtual IP and DNS name in front of a changing set of Pods selected by labels, and load-balances connections across the ready ones.

apiVersion: v1
kind: Service
metadata: { name: api, namespace: petpal }
spec:
  selector: { app: api }        # matches Pod labels
  ports:
    - name: http
      port: 80                  # Service port
      targetPort: http          # container port (name or number)
  type: ClusterIP

Other Pods reach it as http://api (same namespace), http://api.petpal or api.petpal.svc.cluster.local thanks to CoreDNS.

2Service Types

HTTP routing by host/path for many services through one load balancer is done with Ingress or the Gateway API (04-ingress_and_networking/).

3Endpoints and Readiness

The EndpointSlice controller keeps a list of IPs of Pods that match the selector and are Ready. A Pod failing its readiness probe is removed from endpoints (no traffic) without being restarted. During termination, a Pod is removed from endpoints at the same time SIGTERM is sent — and because removal propagates asynchronously to every node, apps should keep serving for a few seconds after SIGTERM (a preStop sleep is the common trick):

lifecycle:
  preStop:
    exec: { command: ["sh", "-c", "sleep 5"] }

4Session Affinity, Traffic Policies, Topology

• sessionAffinity: ClientIP pins clients to a Pod (prefer stateless apps instead).
• externalTrafficPolicy: Local preserves client source IP and avoids extra hops (only nodes with Pods receive traffic).
• internalTrafficPolicy: Local and topology-aware routing keep traffic within a node/zone to cut latency and cross-AZ costs.
• Long-lived connections (gRPC/HTTP2) are balanced per connection, not per request — use client-side balancing over a headless Service or a service mesh.

5Debugging Services

kubectl get svc,endpointslices -n petpal -l app=api      # are there endpoints?
kubectl get pods -n petpal -l app=api --show-labels       # do labels match the selector?
kubectl run tmp --rm -it --image=busybox:1.36 -- wget -qO- http://api.petpal/health
kubectl port-forward svc/api 8080:80 -n petpal            # reach it from your laptop

90% of "Service doesn't work": selector doesn't match Pod labels, targetPort wrong, or Pods not Ready.

Under the HoodFrom Service IP to Pod▾

A ClusterIP is virtual — no interface owns it. kube-proxy on every node watches Services and EndpointSlices and programs the kernel: in iptables mode, a KUBE-SERVICES chain matches the Service IP:port and jumps to a per-service chain that picks one endpoint chain at random (probabilistic rules), which DNATs the packet to a Pod IP:port. IPVS mode uses kernel load-balancing tables (better at thousands of Services), and eBPF data planes (Cilium) do the lookup in BPF maps attached to sockets/interfaces, skipping iptables entirely. The CNI then routes the packet to the Pod's IP, possibly on another node (via VPC routing, overlay encapsulation like VXLAN, or BGP). Because the decision happens per connection at the client's node, there's no central load balancer hop for in-cluster traffic. CoreDNS answers api.petpal.svc.cluster.local with the ClusterIP (or Pod IPs for headless Services).

⚠️ Common Pitfalls

1. Selector/label mismatch — Service with no endpoints.
2. LoadBalancer per service — dozens of cloud LBs and bills; use Ingress/Gateway.
3. No preStop delay — 502s during rollouts.
4. gRPC behind a ClusterIP — all requests stick to one Pod.
5. Hard-coding Pod IPs.

📋 Quick Reference

Service = stable VIP + DNS (svc.ns.svc.cluster.local) → ready Pods by selector
types: ClusterIP · NodePort · LoadBalancer · ExternalName · headless (clusterIP: None)
EndpointSlices = ready Pod IPs · kube-proxy iptables/IPVS/eBPF DNAT · preStop sleep for graceful drain
debug: endpoints? labels? targetPort? readiness? · port-forward · busybox wget

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: k8s_pods_deployments_and_services_theory_1.md, k8s_pods_deployments_and_services_theory_2.md · generated by tools/build_lessons.py