Metrics

1What a Metric Is

A metric is a numeric measurement sampled over time and stored as a time series: name{labels} → [(t1, v1), (t2, v2), …]. Metrics are cheap to store and fast to query because they're aggregated at the source — you count requests, you don't store each one.

http_requests_total{service="checkout", route="/v1/orders", method="POST", status="201"}  10342
http_requests_total{service="checkout", route="/v1/orders", method="POST", status="500"}     17

2The Four Metric Types (Prometheus / OpenMetrics)

Prefer histograms for latency and sizes: quantiles computed at query time over all instances. Summaries' pre-computed p99s can't be averaged meaningfully.

3Instrumenting a Service

import client from "prom-client";
const httpDuration = new client.Histogram({
  name: "http_request_duration_seconds",
  help: "HTTP request latency",
  labelNames: ["route", "method", "status"],
  buckets: [0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
});
app.use((req, res, next) => {
  const end = httpDuration.startTimer();
  res.on("finish", () => end({ route: req.route?.path ?? "unmatched", method: req.method, status: String(res.statusCode) }));
  next();
});
app.get("/metrics", async (_req, res) => { res.type(client.register.contentType).send(await client.register.metrics()); });

Exposition format scraped by Prometheus:

# HELP http_request_duration_seconds HTTP request latency
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{route="/v1/orders",method="POST",status="201",le="0.1"} 812
http_request_duration_seconds_bucket{route="/v1/orders",method="POST",status="201",le="0.25"} 990
http_request_duration_seconds_bucket{route="/v1/orders",method="POST",status="201",le="+Inf"} 1000
http_request_duration_seconds_sum{route="/v1/orders",method="POST",status="201"} 84.2
http_request_duration_seconds_count{route="/v1/orders",method="POST",status="201"} 1000

4Pull vs Push

• Pull (Prometheus): the server scrapes /metrics on every target every N seconds, discovered via Kubernetes/Consul/EC2 service discovery. A failed scrape is itself a signal (up == 0).
• Push (OTLP, StatsD, CloudWatch PutMetricData, Pushgateway): the app sends metrics. Needed for short-lived jobs and across network boundaries.
• OpenTelemetry metrics can be exported either way; the OTel Collector bridges.

5PromQL Essentials

# request rate per route (per second, over the last 5 minutes)
sum by (route) (rate(http_requests_total[5m]))

# error ratio
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

# p99 latency across all instances
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

# saturation: memory used vs limit per pod
container_memory_working_set_bytes / on(pod) kube_pod_container_resource_limits{resource="memory"}

rate() handles counter resets (a restart drops the counter to 0; rate detects the decrease and compensates).

6Labels and Cardinality

Every unique label combination is a separate series. Series count = product of label cardinalities:

route (40) × method (4) × status (10) × instance (30) × le buckets (10) = 480,000 series

That's already a lot for one metric. Never label with user IDs, request IDs, email addresses, full URLs, or error messages — unbounded values belong in logs/traces. Use route templates (/users/:id), status classes when possible, and drop unused labels at scrape time (relabeling).

Under the HoodHow Prometheus Stores Series▾

Prometheus's TSDB keeps the most recent ~2 hours of samples in an in-memory head block, backed by a write-ahead log for crash recovery. Each series has its own chunks; samples are compressed with the Gorilla scheme: timestamps as delta-of-delta (scrapes are regular, so most deltas-of-deltas are 0 → 1 bit), values as XOR with the previous value (slowly changing floats share most bits) — about 1.3 bytes per sample. Every 2 hours the head is cut into an immutable block on disk with an inverted index (label=value → posting list of series IDs); queries intersect posting lists to find series, then decode only the needed chunks. Old blocks are compacted into larger ones. Memory use is dominated by the number of active series in the head (each costs a few KB), which is why cardinality, not sample count, is what takes Prometheus down. Long-term storage (Thanos, Mimir, Cortex) ships these blocks to object storage.

⚠️ Common Pitfalls

1. High-cardinality labels — memory explosion.
2. Averages for latency — hide the tail; use histogram quantiles.
3. Summaries aggregated across instances — mathematically meaningless.
4. Using rate() on gauges or raw counters without rate().
5. Badly chosen histogram buckets — p99 estimates are only as precise as the bucket boundaries around them.
6. Metric names without units/suffixes (_seconds, _bytes, _total).

📋 Quick Reference

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: obs_metrics_theory.md · generated by tools/build_lessons.py