CS Fundamentals Trainer · Guides
Design a metrics and monitoring system: time series at scale
A case study in write-heavy time-series data: what is collected and how (push or pull), ingestion through a queue, a time-series store with compression, downsampling and retention, an alert evaluator that avoids flapping, cardinality as the main cost driver, and dashboards that query the store efficiently.
5 min read
The problem. Thousands of servers and services emit numbers over time: CPU, request counts, latency, queue depth. Engineers want dashboards that load quickly, and alerts that wake someone when something is wrong and not before. The system has to stay up when everything else is failing, because that is when it is needed.
Requirements and numbers
Collect metrics (counters, gauges, histograms) with labels such as service and region, store them for a retention period, query them quickly for dashboards, and alert on rules. Assume 1 million active time series (one metric with one set of labels), each sampled every 15 seconds: 1,000,000 / 15 ≈ 67,000 data points per second. With 100 million series it is 6.7 million per second. A point is a timestamp and a value, 16 bytes raw; but consecutive timestamps and values are very similar, and time-series compression (delta-of-delta for timestamps, XOR for floats, as in Facebook's Gorilla) stores a point in about 1 to 2 bytes. At 1.5 bytes, 67,000 points per second is about 67,000 x 1.5 x 86,400 ≈ 8.7 GB a day. The system is write-heavy and append-only, reads are far fewer, and recent data is queried most.
Collecting: pull or push
- Pull (Prometheus): the monitoring system scrapes an HTTP endpoint on each target on a schedule. Simple, it knows which targets are down (a failed scrape is itself a signal), and applications do not need to know where the monitor is. It needs service discovery and cannot easily reach short-lived jobs.
- Push (StatsD, OpenTelemetry collectors): agents send data to the system. It suits short-lived jobs, serverless and firewalled networks, and an unreachable target is harder to tell from a silent one.
Many systems combine them: an agent on each host collects locally and either is scraped or pushes batches.
Architecture
Targets --> collectors/agents --> ingestion gateway --> Kafka (partitioned by series)
|
+-----------------------------+-----------------+
ingesters (write to TSDB) stream processors
| (rollups, alerts)
time-series store (hot) --> long-term store (cold, object storage)
|
query service <--- dashboards, API
|
alert evaluator --> notification system (see the notification case study)
- Queue in the middle. A durable log (Kafka) decouples producers from the store, absorbs bursts and lets several consumers (storage, alerting, rollups) read the same data. If the store is slow or down, data waits instead of being lost.
- Storage engine. A time-series database keeps recent data in memory with a write-ahead log, flushes it in time-partitioned blocks and compresses them. The index maps labels to series (an inverted index:
service=api-> list of series ids). Examples: Prometheus TSDB, VictoriaMetrics, InfluxDB, Mimir, or a column store. - Sharding and replication. Shard series by a hash of their identity so that all points of a series are in one place, and replicate for availability. A query fans out to the shards and merges.
Retention and downsampling
Raw 15-second data is useful for days, not for years. Keep it for, say, 15 days, then downsample to 5-minute averages (and min, max, count, so that percentiles and spikes remain visible) kept for a year, then hourly. Rollups are computed by a background job. Old blocks move to cheap object storage. Choosing which aggregations to keep matters: an average hides spikes, so keep the max, and for latency store histogram buckets, because percentiles cannot be recomputed from averages.
Cardinality: the cost driver
The number of series is the product of the distinct values of the labels: 100 services x 50 endpoints x 10 status codes x 1,000 hosts is 50 million series. Adding a label whose values are unbounded (a user id, a request id, a full URL) creates a new series per value and can overwhelm the index and memory. Rules: labels must have a small, known set of values; per-request data belongs in logs and traces, not metrics; enforce a limit per metric at ingestion and alert when it is approached.
Alerting
An evaluator runs rules periodically over recent data: error rate above 5% for 5 minutes. Design points:
- Use a
forduration so a single bad sample does not page anyone, and avoid flapping with hysteresis (different thresholds for firing and clearing). - Alert on symptoms users feel (errors, latency, saturation) rather than every cause (a CPU spike on one host).
- Group, deduplicate and silence related alerts in an alert manager; route by severity and ownership; include a runbook link.
- Alert when data stops arriving (absent checks, a heartbeat), because silence is a failure mode, and monitor the monitoring system from outside it.
Querying
Dashboards ask for the same ranges repeatedly. Make that cheap with: recording rules that precompute expensive queries, caching of recent results, a query limit on points per panel (a 1-year range need not return 2 million points: use the downsampled data and let the step grow), and per-user query limits.
Trade-offs
Pull versus push, accuracy versus cost (sampling, coarse resolution), local simplicity versus a global view (federation, or remote write into a central store), and availability over consistency: dropping a few points is better than blocking a service from running.
Sources
- Alex Xu, System Design Interview Vol. 2, chapter "Metrics Monitoring and Alerting System".
- Pelkonen et al., Gorilla: A Fast, Scalable, In-Memory Time Series Database (VLDB 2015).
- Prometheus: Overview and storage and Instrumentation best practices: labels.
- Google, Site Reliability Engineering, chapter "Monitoring Distributed Systems".
Practise this topic: System Design System Design › Case studies