# CloudWatch

CloudWatch is AWS's built-in observability service: **metrics**, **logs**, **alarms**, **dashboards**, plus events/scheduling (now EventBridge) and some tracing integration (X-Ray / Application Signals). Concepts map directly onto `devops-cloud/observability/`.

## 1. Metrics

A metric is identified by **namespace + name + dimensions**:

```
namespace: AWS/ApplicationELB
name:      TargetResponseTime
dimensions: LoadBalancer=app/shop-alb/50dc6c495c0c9188, TargetGroup=targetgroup/api/943f017f100becff
```

- AWS services publish metrics automatically (EC2 `CPUUtilization`, ALB `HTTPCode_Target_5XX_Count`, SQS `ApproximateAgeOfOldestMessage`, RDS `ReplicaLag`, Lambda `Errors`/`Duration`/`Throttles`…). Default EC2 resolution is 5 minutes; **detailed monitoring** gives 1 minute.
- **Custom metrics** via `PutMetricData`, the CloudWatch agent (memory, disk — not visible to the hypervisor!), or **Embedded Metric Format (EMF)**: write a structured JSON log line and CloudWatch extracts metrics from it — no API calls in the hot path.
- **Statistics**: `Sum`, `Average`, `Minimum`, `Maximum`, `SampleCount`, and percentiles `p50`, `p99`… Every unique dimension combination is a separate (billed) metric — keep dimensions low-cardinality.

```bash
aws cloudwatch put-metric-data --namespace Shop --metric-name OrdersPlaced \
  --dimensions Service=checkout,Env=prod --value 1 --unit Count
```

## 2. Logs

- **Log groups** (one per app/environment, with a **retention** setting — default is *never expire*, which silently accumulates cost) contain **log streams** (one per instance/container/Lambda execution environment).
- Shipping: Lambda and ECS (`awslogs` driver) send stdout automatically; EC2 uses the CloudWatch agent.
- **Logs Insights** queries structured/JSON logs:

```
fields @timestamp, requestId, route, durationMs
| filter level = "error" and route like /checkout/
| stats count(*) as errors, pct(durationMs, 99) as p99 by bin(5m)
| sort errors desc
```

- **Metric filters** turn log patterns into metrics (e.g. count of `"level":"error"` lines); **subscription filters** stream logs to Lambda/Kinesis/OpenSearch.

## 3. Alarms

An alarm watches one metric (or a **metric math** expression) and moves between `OK`, `ALARM` and `INSUFFICIENT_DATA`:

```bash
aws cloudwatch put-metric-alarm --alarm-name api-5xx-rate \
  --metrics '[
    {"Id":"e","MetricStat":{"Metric":{"Namespace":"AWS/ApplicationELB","MetricName":"HTTPCode_Target_5XX_Count","Dimensions":[{"Name":"LoadBalancer","Value":"app/shop-alb/50dc"}]},"Period":60,"Stat":"Sum"},"ReturnData":false},
    {"Id":"r","MetricStat":{"Metric":{"Namespace":"AWS/ApplicationELB","MetricName":"RequestCount","Dimensions":[{"Name":"LoadBalancer","Value":"app/shop-alb/50dc"}]},"Period":60,"Stat":"Sum"},"ReturnData":false},
    {"Id":"ratio","Expression":"100 * e / MAX([r, 1])","Label":"5xx %","ReturnData":true}]' \
  --evaluation-periods 5 --datapoints-to-alarm 3 --threshold 1 \
  --comparison-operator GreaterThanThreshold --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:eu-west-1:123456789012:oncall
```

- **`M out of N` datapoints** (3 of 5) avoids flapping on one bad minute.
- **`treat-missing-data`** matters: missing data for an error count usually means "no errors" (`notBreaching`); for a heartbeat metric it means "dead" (`breaching`).
- **Composite alarms** combine alarms with boolean logic (page only if 5xx AND latency are both bad) to cut noise.
- **Anomaly detection** alarms learn a band from history (daily/weekly seasonality).
- Actions: SNS (→ PagerDuty/Slack/email), Auto Scaling policies, EC2 actions (recover/reboot), Systems Manager incidents.

## 4. Dashboards and Synthetics

Dashboards show metrics and Logs Insights widgets, cross-account and cross-region. **Synthetics canaries** run scripted checks (Puppeteer/Selenium) against your endpoints every few minutes — catching outages even when there's no traffic. **RUM** measures real browser performance.

## Under the Hood: Resolution, Retention and Aggregation

CloudWatch stores metric data points with **automatic roll-up**: 1-second (high-resolution custom metrics) data is kept for 3 hours, 1-minute data for 15 days, 5-minute for 63 days, 1-hour for 455 days. Older data is only available at the coarser resolution — a 1-minute spike from two months ago is gone. Percentile statistics are computed from the raw values (or, for high-volume sources, from server-side sketches) **per period** and can't be re-aggregated from other percentiles, which is why you must publish raw values (or EMF) rather than pre-computed p99s from each instance. Alarms evaluate on a fixed schedule over the metric's period; a metric published late (e.g. delayed log extraction) may be evaluated as missing first — tune `datapoints-to-alarm` and missing-data handling accordingly.

![Applications and AWS services send metrics and logs to CloudWatch; log groups with retention feed Logs Insights and metric filters, alarms evaluate metrics or metric math with M of N datapoints and notify SNS or trigger auto scaling, and dashboards and synthetics complete the picture](diagrams/01-cloudwatch-flow.svg)

## Common Pitfalls

1. **Log groups with no retention** — costs grow forever.
2. **Alarming on single datapoints** — flapping pages.
3. **Wrong missing-data treatment** — silent dead services or false alarms.
4. **High-cardinality dimensions** (user ID, request ID) — metric explosion and cost.
5. **Assuming EC2 memory/disk metrics exist** — they need the agent.
6. **Averages for latency** — alarm on p95/p99.

## Quick Reference

| Need | CloudWatch feature |
|---|---|
| service health numbers | metrics (namespace/name/dimensions) |
| app metrics cheaply | EMF structured logs |
| search logs | Logs Insights |
| log → metric | metric filters |
| alert | alarm (M of N, missing-data policy) → SNS |
| reduce noise | composite alarms, anomaly detection |
| outside-in checks | Synthetics canaries |