CS Fundamentals Trainer · Guides

Design a metrics and monitoring system: time series at scale

A case study in write-heavy time-series data: what is collected and how (push or pull), ingestion through a queue, a time-series store with compression, downsampling and retention, an alert evaluator that avoids flapping, cardinality as the main cost driver, and dashboards that query the store efficiently.

5 min read

The problem. Thousands of servers and services emit numbers over time: CPU, request counts, latency, queue depth. Engineers want dashboards that load quickly, and alerts that wake someone when something is wrong and not before. The system has to stay up when everything else is failing, because that is when it is needed.

Requirements and numbers

Collect metrics (counters, gauges, histograms) with labels such as service and region, store them for a retention period, query them quickly for dashboards, and alert on rules. Assume 1 million active time series (one metric with one set of labels), each sampled every 15 seconds: 1,000,000 / 15 ≈ 67,000 data points per second. With 100 million series it is 6.7 million per second. A point is a timestamp and a value, 16 bytes raw; but consecutive timestamps and values are very similar, and time-series compression (delta-of-delta for timestamps, XOR for floats, as in Facebook's Gorilla) stores a point in about 1 to 2 bytes. At 1.5 bytes, 67,000 points per second is about 67,000 x 1.5 x 86,400 ≈ 8.7 GB a day. The system is write-heavy and append-only, reads are far fewer, and recent data is queried most.

Collecting: pull or push

Many systems combine them: an agent on each host collects locally and either is scraped or pushes batches.

Architecture

Targets --> collectors/agents --> ingestion gateway --> Kafka (partitioned by series)
                                                            |
                              +-----------------------------+-----------------+
                       ingesters (write to TSDB)                      stream processors
                              |                                         (rollups, alerts)
                      time-series store (hot) --> long-term store (cold, object storage)
                              |
                        query service  <---  dashboards, API
                              |
                        alert evaluator --> notification system (see the notification case study)

Retention and downsampling

Raw 15-second data is useful for days, not for years. Keep it for, say, 15 days, then downsample to 5-minute averages (and min, max, count, so that percentiles and spikes remain visible) kept for a year, then hourly. Rollups are computed by a background job. Old blocks move to cheap object storage. Choosing which aggregations to keep matters: an average hides spikes, so keep the max, and for latency store histogram buckets, because percentiles cannot be recomputed from averages.

Cardinality: the cost driver

The number of series is the product of the distinct values of the labels: 100 services x 50 endpoints x 10 status codes x 1,000 hosts is 50 million series. Adding a label whose values are unbounded (a user id, a request id, a full URL) creates a new series per value and can overwhelm the index and memory. Rules: labels must have a small, known set of values; per-request data belongs in logs and traces, not metrics; enforce a limit per metric at ingestion and alert when it is approached.

Alerting

An evaluator runs rules periodically over recent data: error rate above 5% for 5 minutes. Design points:

Querying

Dashboards ask for the same ranges repeatedly. Make that cheap with: recording rules that precompute expensive queries, caching of recent results, a query limit on points per panel (a 1-year range need not return 2 million points: use the downsampled data and let the step grow), and per-user query limits.

Trade-offs

Pull versus push, accuracy versus cost (sampling, coarse resolution), local simplicity versus a global view (federation, or remote write into a central store), and availability over consistency: dropping a few points is better than blocking a service from running.

Sources

Practise this topic: System Design System Design › Case studies

More guides