Production Practices and GitOps

1GitOps

GitOps makes Git the single source of truth for cluster state: a controller inside the cluster continuously pulls desired state from Git and reconciles the cluster to match it.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: api-prod, namespace: argocd }
spec:
  project: petpal
  source:
    repoURL: https://github.com/petpal/deploy.git
    targetRevision: main
    path: apps/api/overlays/prod          # Kustomize overlay or Helm chart + values
  destination: { server: "https://kubernetes.default.svc", namespace: petpal }
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [CreateNamespace=true]

Typical flow: app CI builds and pushes api:sha-abc123 → a bot (or Argo CD Image Updater / Flux image automation) opens a PR in the deploy repo bumping the tag in the staging overlay → merge → Argo CD syncs staging → promotion PR for prod (devops-cloud/cicd/). Progressive delivery with Argo Rollouts/Flagger adds automated canaries with metric analysis.

2Production Readiness Checklist for Workloads

3Policy as Code in the Cluster

Admission policy engines enforce the checklist automatically: Kyverno (policies as YAML), OPA Gatekeeper (Rego), or built-in ValidatingAdmissionPolicy (CEL):

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: require-requests-and-probes }
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-containers
      match: { any: [{ resources: { kinds: [Deployment, StatefulSet] } }] }
      validate:
        message: "containers need resource requests and a readiness probe"
        pattern:
          spec:
            template:
              spec:
                containers:
                  - resources: { requests: { cpu: "?*", memory: "?*" } }
                    readinessProbe: {}

Run the same policies in CI (kyverno apply, conftest, kubeconform schema validation) so violations fail before merge.

4Cluster Operations

• Upgrades: Kubernetes ships ~3 minor versions a year; managed services support each for ~14 months. Upgrade control plane, then add-ons, then nodes (surge/blue-green node groups); check deprecated APIs first (kubent, pluto).
• Multi-tenancy: namespaces per team with quotas, RBAC, network policies; separate clusters for hard isolation (prod vs non-prod, compliance).
• Backups and DR: Git holds manifests; Velero for volumes/cluster resources; rehearse rebuilding a cluster from Git.
• Cost: right-size requests, Karpenter consolidation, Spot, per-namespace cost visibility (OpenCost/Kubecost).

Under the HoodHow Argo CD Reconciles▾

Argo CD's repo server clones the Git repository at the target revision and renders manifests (plain YAML, Kustomize build or Helm template — Argo uses Helm only as a renderer, no release Secrets). The application controller watches live cluster objects (via informers) and, every few minutes or on a Git webhook, compares the rendered desired objects with the live ones using a normalized diff (ignoring defaulted fields and configured ignoreDifferences such as replicas managed by an HPA). The app status is Synced/OutOfSync and Healthy/Progressing/Degraded (health from resource-specific checks — e.g. a Deployment is Healthy when its rollout completed). A sync applies resources in waves (argocd.argoproj.io/sync-wave annotations, with PreSync hooks such as migrations), prunes objects removed from Git when prune is on, and selfHeal re-applies desired state when someone changes the cluster manually — making Git the effective control point.

⚠️ Common Pitfalls

1. CI with cluster-admin credentials — a CI compromise owns the cluster.
2. Manual hotfixes with kubectl under GitOps — reverted by self-heal (or worse, silently kept without it).
3. Secrets committed to the deploy repo.
4. Auto-sync to prod without promotion gates.
5. Policies only documented, not enforced.
6. Skipping Kubernetes upgrades until forced by end of support.

📋 Quick Reference

GitOps: Git = truth · Argo CD Application/ApplicationSet · Flux Kustomization/HelmRelease · prune + selfHeal · promotion by PR
readiness: probes · requests/limits · replicas + spread + PDB · HPA · PSA restricted · NetworkPolicy · RBAC · signed images · SLO alerts
policy: Kyverno / Gatekeeper / ValidatingAdmissionPolicy (+ same checks in CI: kubeconform, conftest)
ops: upgrades (deprecated APIs) · quotas per namespace · Velero · OpenCost

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: k8s_production_practices_and_gitops_theory.md · generated by tools/build_lessons.py