Secrets Manager, SSM Parameter Store and KMS

1Three Services, Three Jobs

2KMS: Keys You Never See

aws kms create-key --description "petpal app data" --key-spec SYMMETRIC_DEFAULT
aws kms create-alias --alias-name alias/petpal-app --target-key-id <key-id>
aws kms encrypt --key-id alias/petpal-app --plaintext fileb://card-token.txt   # ≤ 4 KB

• Keys never leave KMS's HSMs in plaintext; you call Encrypt/Decrypt/GenerateDataKey via API, every call is logged in CloudTrail.
• AWS managed keys (aws/s3, aws/rds…) vs customer managed keys (your key policy, rotation, cross-account grants, disable/schedule deletion) — use CMKs when you need control over who can decrypt.
• Key policies are the primary access control (IAM policies only work if the key policy allows the account); grants give temporary, scoped permissions (used by services like EBS).
• Automatic rotation (yearly or custom period): new key material, old material kept to decrypt old data — transparent to callers.
• Envelope encryption: GenerateDataKey returns a plaintext data key (encrypt locally with AES-GCM, then discard) and the same key encrypted under the KMS key (store it next to the ciphertext). Decrypt = Decrypt(encryptedDataKey) → plaintext data key → decrypt data locally. That's how S3/EBS/RDS encryption work, and how the AWS Encryption SDK works (security/security/07-hashing_vs_encryption).

3Secrets Manager

aws secretsmanager create-secret --name prod/petpal/stripe --secret-string '{"apiKey":"sk_live_..."}' \
  --kms-key-id alias/petpal-secrets
aws secretsmanager get-secret-value --secret-id prod/petpal/stripe

• Rotation: a Lambda (AWS-provided templates for RDS/Aurora/Redshift/DocumentDB, or your own) runs on a schedule through four steps — createSecret (new version AWSPENDING), setSecret (change the password in the database), testSecret, finishSecret (move AWSCURRENT label). Apps always read AWSCURRENT.
• Managed rotation: RDS manage_master_user_password lets RDS own the secret and rotate it (devops-cloud/terraform/08-…).
• Native integrations: ECS/EKS (secrets valueFrom / external-secrets), Lambda (Parameters and Secrets extension), CloudFormation dynamic references.
• Resource policies for cross-account access; replication to other regions for DR.
• Price: ~$0.40 per secret per month + API calls — cache reads.

4SSM Parameter Store

aws ssm put-parameter --name /prod/petpal/api/base_url --type String --value https://api.petpal.io
aws ssm put-parameter --name /prod/petpal/api/legacy_token --type SecureString --value "..." --key-id alias/petpal-app
aws ssm get-parameters-by-path --path /prod/petpal/api/ --recursive --with-decryption

• Hierarchical names → IAM policies by path (arn:aws:ssm:…:parameter/prod/petpal/*).
• Standard tier free (4 KB values, 10,000 parameters), Advanced for 8 KB values and parameter policies (expiration).
• SecureString = KMS-encrypted; no built-in rotation.
• Great for config shared between IaC units (devops-cloud/terraform/03-… Part 2) and apps.

Choose: Secrets Manager for credentials that should rotate or be replicated; Parameter Store for configuration and secrets that don't need rotation (cost-sensitive).

5Using Secrets in Applications

• Fetch at startup and cache with a TTL (e.g. 5 minutes) so rotations propagate without restarts, and so you don't call the API per request (cost, throttling).
• On authentication failure, refresh the secret once and retry (the secret may have just rotated).
• Never log secrets; never put them in environment variables baked into images or task definitions as plain values (valueFrom is fine — injected at start).
• Least privilege: each workload can read only its own secrets (secretsmanager:GetSecretValue on specific ARNs, kms:Decrypt on the specific key via kms:ViaService condition).

Under the HoodWhat Happens When ECS Injects a Secret▾

When a Fargate task starts, the agent — using the task execution role — calls secretsmanager:GetSecretValue (or ssm:GetParameters) for each secrets entry. Secrets Manager stores the secret encrypted with a data key that's itself encrypted under the secret's KMS key; to return the value, Secrets Manager calls kms:Decrypt on behalf of the caller (so the execution role also needs decrypt permission on a customer-managed key, typically constrained with kms:ViaService = secretsmanager.<region>.amazonaws.com). The plaintext is passed into the container as an environment variable at process start. Every step — the GetSecretValue call and the KMS decrypt — appears in CloudTrail with the role and task identity, giving an audit trail of who read which secret when. Note the consequence: a rotated secret reaches a running task only when the task restarts, unless the application reads the secret itself with a cache TTL.

⚠️ Common Pitfalls

1. Secrets in code, tfvars, task definition env values or images.
2. Fetching secrets on every request — cost, latency, throttling.
3. Caching forever — rotations break the app until restart.
4. Over-broad permissions (secretsmanager:* on *, kms:Decrypt on *).
5. Using AWS managed keys when cross-account access is needed (their key policies can't be edited).
6. Scheduling KMS key deletion without checking what's encrypted with it (data becomes unrecoverable after the waiting period).

📋 Quick Reference

KMS: CMKs + key policies + rotation · Encrypt ≤ 4 KB · GenerateDataKey for envelope encryption · CloudTrail on every use
Secrets Manager: rotation Lambda (4 steps, AWSCURRENT/AWSPENDING) · managed RDS passwords · replication · ~$0.40/secret
Parameter Store: hierarchical config · SecureString via KMS · free standard tier · IAM by path
apps: cache with TTL, refresh on auth failure, least privilege (ARN-scoped + kms:ViaService)

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: aws_secrets_manager_ssm_and_kms_theory.md · generated by tools/build_lessons.py