# Metrics

## 1. What a Metric Is

A **metric** is a numeric measurement sampled over time and stored as a **time series**: `name{labels} → [(t1, v1), (t2, v2), …]`. Metrics are cheap to store and fast to query because they're aggregated at the source — you count requests, you don't store each one.

```
http_requests_total{service="checkout", route="/v1/orders", method="POST", status="201"}  10342
http_requests_total{service="checkout", route="/v1/orders", method="POST", status="500"}     17
```

## 2. The Four Metric Types (Prometheus / OpenMetrics)

| Type | Behaviour | Example | Query with |
|---|---|---|---|
| **Counter** | only goes up (resets to 0 on restart) | `http_requests_total`, `bytes_sent_total` | `rate()`, `increase()` |
| **Gauge** | goes up and down | `queue_depth`, `memory_bytes`, `in_flight_requests` | raw value, `avg_over_time()` |
| **Histogram** | counts observations into cumulative **buckets** + `_sum` + `_count` | `http_request_duration_seconds` | `histogram_quantile()` |
| **Summary** | client-side quantiles (p50/p99) per instance | legacy | can't be aggregated across instances |

Prefer **histograms** for latency and sizes: quantiles computed at query time over **all** instances. Summaries' pre-computed p99s can't be averaged meaningfully.

## 3. Instrumenting a Service

```ts
import client from "prom-client";
const httpDuration = new client.Histogram({
  name: "http_request_duration_seconds",
  help: "HTTP request latency",
  labelNames: ["route", "method", "status"],
  buckets: [0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
});
app.use((req, res, next) => {
  const end = httpDuration.startTimer();
  res.on("finish", () => end({ route: req.route?.path ?? "unmatched", method: req.method, status: String(res.statusCode) }));
  next();
});
app.get("/metrics", async (_req, res) => { res.type(client.register.contentType).send(await client.register.metrics()); });
```

Exposition format scraped by Prometheus:

```
# HELP http_request_duration_seconds HTTP request latency
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{route="/v1/orders",method="POST",status="201",le="0.1"} 812
http_request_duration_seconds_bucket{route="/v1/orders",method="POST",status="201",le="0.25"} 990
http_request_duration_seconds_bucket{route="/v1/orders",method="POST",status="201",le="+Inf"} 1000
http_request_duration_seconds_sum{route="/v1/orders",method="POST",status="201"} 84.2
http_request_duration_seconds_count{route="/v1/orders",method="POST",status="201"} 1000
```

## 4. Pull vs Push

- **Pull (Prometheus)**: the server **scrapes** `/metrics` on every target every N seconds, discovered via Kubernetes/Consul/EC2 service discovery. A failed scrape is itself a signal (`up == 0`).
- **Push (OTLP, StatsD, CloudWatch PutMetricData, Pushgateway)**: the app sends metrics. Needed for short-lived jobs and across network boundaries.
- OpenTelemetry metrics can be exported either way; the OTel Collector bridges.

## 5. PromQL Essentials

```promql
# request rate per route (per second, over the last 5 minutes)
sum by (route) (rate(http_requests_total[5m]))

# error ratio
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

# p99 latency across all instances
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

# saturation: memory used vs limit per pod
container_memory_working_set_bytes / on(pod) kube_pod_container_resource_limits{resource="memory"}
```

`rate()` handles counter **resets** (a restart drops the counter to 0; rate detects the decrease and compensates).

## 6. Labels and Cardinality

Every unique label combination is a separate series. Series count = product of label cardinalities:

```
route (40) × method (4) × status (10) × instance (30) × le buckets (10) = 480,000 series
```

That's already a lot for one metric. **Never** label with user IDs, request IDs, email addresses, full URLs, or error messages — unbounded values belong in logs/traces. Use route **templates** (`/users/:id`), status **classes** when possible, and drop unused labels at scrape time (relabeling).

## Under the Hood: How Prometheus Stores Series

Prometheus's TSDB keeps the most recent ~2 hours of samples in an in-memory **head block**, backed by a write-ahead log for crash recovery. Each series has its own chunks; samples are compressed with the Gorilla scheme: timestamps as **delta-of-delta** (scrapes are regular, so most deltas-of-deltas are 0 → 1 bit), values as **XOR with the previous value** (slowly changing floats share most bits) — about **1.3 bytes per sample**. Every 2 hours the head is cut into an immutable block on disk with an **inverted index** (label=value → posting list of series IDs); queries intersect posting lists to find series, then decode only the needed chunks. Old blocks are compacted into larger ones. Memory use is dominated by the number of **active series** in the head (each costs a few KB), which is why cardinality, not sample count, is what takes Prometheus down. Long-term storage (Thanos, Mimir, Cortex) ships these blocks to object storage.

![A service increments counters and observes histogram buckets, exposes them on /metrics, Prometheus scrapes them into a time series database with a head block and inverted index, and PromQL computes rates, error ratios and histogram quantiles for dashboards and alerts](diagrams/01-metrics-pipeline.svg)

## Common Pitfalls

1. **High-cardinality labels** — memory explosion.
2. **Averages for latency** — hide the tail; use histogram quantiles.
3. **Summaries aggregated across instances** — mathematically meaningless.
4. **Using `rate()` on gauges or raw counters without `rate()`**.
5. **Badly chosen histogram buckets** — p99 estimates are only as precise as the bucket boundaries around them.
6. **Metric names without units/suffixes** (`_seconds`, `_bytes`, `_total`).

## Quick Reference

| Need | Use |
|---|---|
| how many / how fast | counter + `rate()` |
| current level | gauge |
| latency distribution | histogram + `histogram_quantile()` |
| error ratio | `rate(errors) / rate(total)` |
| naming | `snake_case`, unit suffix, `_total` for counters |
| labels | bounded values only |