# Exercise 2: State Incidents — Diagnose and Recover

For each incident, explain what happened using `tf_state_and_remote_backends_theory_1.md` and `_theory_2.md`, list the exact recovery steps (commands), and propose a prevention.

## Incident A — "Terraform wants to recreate production"

An engineer renamed the backend `key` from `prod/terraform.tfstate` to `prod/network/terraform.tfstate` to "tidy up", ran `terraform init` (answering "no" to a migration prompt they didn't read), then `terraform plan`:

```
Plan: 87 to add, 0 to change, 0 to destroy.
```

## Incident B — The stale lock

CI was cancelled mid-apply. Every run since fails with:

```
Error acquiring the state lock
Lock Info:
  ID:        3f2a6c1e-…
  Path:      acme-tfstate/prod/data/terraform.tfstate
  Operation: OperationTypeApply
  Who:       runner@ci-7f9c
  Created:   2025-06-12 10:41:07 UTC
```

## Incident C — Drift that would undo a hotfix

During an outage, on-call raised an RDS instance's `max_connections` parameter and resized the instance in the console. A week later a routine PR shows:

```
~ resource "aws_db_parameter_group" "main" { ~ parameter { value = "800" -> "400" } }
~ resource "aws_db_instance" "main"        { ~ instance_class = "db.r6g.2xlarge" -> "db.r6g.large" }
```

## Incident D — Corrupted state after a crash

A laptop lost power during `terraform apply` with **local** state. The `terraform.tfstate` file is now truncated JSON; some resources were created in AWS before the crash.

## Incident E — The leaked state

A developer pasted `terraform state pull` output into a public gist while asking for help. The state includes an RDS master password, an IAM access key (`aws_iam_access_key`) and a TLS private key.

## Deliverables

For each incident: **cause**, **recovery steps** (commands), **prevention**.

## Requirements Checklist

- [ ] A: init without migration pointed Terraform at an empty state at the new key; nothing was destroyed. Recovery: restore the backend key (or copy the old object to the new key / `init -migrate-state`), verify `plan` shows no changes. Prevention: `-migrate-state`, plan review in CI, prevent_destroy on critical resources.
- [ ] B: confirm no run is active (CI history, runner gone), then `terraform force-unlock 3f2a6c1e-…`; afterwards run `plan` to check for a partial apply. Prevention: CI concurrency groups, no cancelling during apply, lock timeouts.
- [ ] C: drift from a console hotfix; resolution: codify the new values in a PR (or `apply -refresh-only` then update config), never blindly apply the revert. Prevention: emergency changes followed by a codifying PR; scheduled drift detection.
- [ ] D: restore from `terraform.tfstate.backup` (or versioned remote state), then `import` resources created after the backup, and verify with plan. Prevention: remote backend with versioning.
- [ ] E: treat every secret as compromised — rotate the DB password, delete/rotate the access key, revoke/replace certificates/keys; delete the gist; audit CloudTrail for use. Prevention: remote encrypted state with restricted access, keep secrets out of state (managed passwords, Secrets Manager), avoid `aws_iam_access_key` resources.