CS Fundamentals Trainer · Guides
Observability: logs, metrics and traces
What each of the three signals answers, why averages hide slow requests and percentiles cannot be averaged, label cardinality, log cost, trace propagation and sampling, and alerting on SLO burn rates, with a two-service example in Python.
26 min read
When a system misbehaves in production you cannot attach a debugger, and the code that ran is on a machine you did not touch. Observability is how well you can work out what a system is doing, and why, from the data it emits. In practice that data comes in three kinds: metrics (numbers over time), logs (records of events) and traces (the path of one request through many services). They answer different questions, they cost different amounts, and each has a classic way of going wrong. This guide shows all three on a small two-service example, then covers the traps that matter most: averages hiding slow requests, percentiles that cannot be added, labels that explode the number of metric series, logs that cost more than they are worth, traces sampled in a way that loses exactly the ones you need, and alerts built from error budgets. The code is in the appendix; each number is one run on a small machine, so the behaviour matters and the exact figures do not.
1. Three signals, three questions
| Metrics | Logs | Traces | |
|---|---|---|---|
| What it is | numbers aggregated over time (counts, rates, durations) | one record per event, with details | the tree of timed steps one request took across services |
| Answers | Is something wrong, and how much? | Why did this happen? | Where did the time go (or the error come from)? |
| Cost grows with | the number of distinct series (labels), not with traffic | traffic and the amount you write per event | traffic (unless you sample) |
| Good for | dashboards and alerts | debugging one incident in detail | finding the slow or failing hop |
| Weak at | no detail about individual requests | hard to see patterns, expensive to keep forever | needs every service to pass the context along |
They work best joined together: an alert fires from a metric, a trace shows which service and call is slow, and the logs of that trace say why. The join key is an identifier, the trace id, written into every log line and every span.
A worked example
I wrote two small services and instrumented them by hand to show the ideas: a web service whose /checkout calls a catalog service for a price. (The hand-made code is a teaching tool. It is not an implementation of the OpenTelemetry data model: it has no resource or instrumentation-scope information, span kinds, status codes, events, links, attributes with standard names, tracestate, baggage or exporters, and it does not act on the sampled flag. In real services use the OpenTelemetry libraries and their automatic instrumentation.) Every fifth new product is missing from the catalog's cache, so that request reads the database (a 300 ms sleep in the test). Forty checkouts later:
Metrics said that something was off: the mean was 46 ms, but the 95th percentile estimated from the histogram was about 417 ms. Six of the 40 requests landed in the 250 to 500 ms bucket and the other 34 in the first bucket.
The trace of the slowest request showed where the time went:
web GET /checkout 302 ms
web call catalog 302 ms
catalog GET /price 300 ms
catalog db.query 300 ms
catalog cache.lookup 0 ms
The logs with that trace id said why:
{"level": "INFO", "service": "web", "msg": "checkout started", "trace_id": "021e8cdc...", "sku": "sku-15"}
{"level": "WARN", "service": "catalog", "msg": "cache miss, reading the database", "trace_id": "021e8cdc...", "sku": "sku-15"}
{"level": "INFO", "service": "web", "msg": "checkout finished", "trace_id": "021e8cdc...", "sku": "sku-15"}
The metric could not say which request, the trace could not say that it was a cache miss, and the log alone would have been one line among thousands. Each signal supplied what the others lacked.
Diagram: The trace context travels with the request
2. Metrics
A metric is a number with a name and labels (http_requests_total{route="/checkout", status="200"}). Each distinct set of label values is a separate time series. Four kinds cover almost everything:
- Counter: only goes up (requests served, errors). You look at its rate over a window, not its value.
- Gauge: goes up and down (queue length, memory in use, connections open).
- Histogram: counts observations into buckets (how many requests took up to 5 ms, 10 ms, 25 ms and so on), plus a sum and a count. This is how you record latency.
- Summary: computes quantiles inside the application. Cheap to read, but see below: it cannot be combined across instances.
What to measure first: for a request-handling service, the RED signals, Rate, Errors and Duration, per route. For a resource (CPU, disk, a connection pool), the USE signals: Utilisation, Saturation and Errors. Google's SRE book names four golden signals: latency, traffic, errors and saturation. They are the same idea in different words.
Averages hide the requests that hurt
In a test of 100,000 requests where 98% took about 20 ms and 2% took about 2 s (a slow path):
| Statistic | Value |
|---|---|
| mean | 60 ms |
| median (p50) | 20 ms |
| p90 | 26 ms |
| p99 | 1,995 ms |
| p99.9 | 2,506 ms |
The mean of 60 ms describes no real request: it is neither the fast ones nor the slow ones. The median and p90 look perfect while one in fifty requests takes two seconds. Always chart percentiles (p50, p95, p99) and not the average of latency.
Tail latency also multiplies with fan-out. If a page makes several backend calls and each has a 2% chance of being slow, the chance that a page load hits at least one slow call is 1 - 0.98^n: 2% for one call, 9.6% for five, 33% for twenty and 64% for fifty. Calls that look rare in a single service dominate the experience of a page.
Percentiles cannot be averaged
Two servers: A is busy (10,000 requests, 0.5% slow) and B is quiet (100 requests, 10% slow). Server A's p99 was 30 ms and B's p99 was 1,087 ms. The average of the two, 559 ms, is meaningless: the p99 of all 10,100 requests together was 31 ms. A dashboard that averages the per-instance p99 values reports a problem that most users do not have (or, with other traffic, hides one that they do).
The fix is to record histograms and add their buckets across instances, then compute the quantile from the combined buckets. That is why Prometheus advises histograms over summaries when you need to aggregate. The price is precision: the quantile is estimated by interpolating inside a bucket, so the buckets must suit your latency:
| Buckets (ms) | p99 estimate | exact p99 | p50 estimate | exact p50 |
|---|---|---|---|---|
| 10, 100, 1000, 10000 | 5,500 | 1,995 | 56 | 20 |
| 5, 10, 25, 50, 100, 250, 500, 1000, 2500 | 1,791 | 1,995 | 19 | 20 |
Coarse buckets turned a 2 s p99 into 5.5 s. Put bucket boundaries around the values you care about, such as your latency target.
Labels are a budget
Every combination of label values is a new series that the monitoring system stores, indexes and charts. Labels like method, route and status give a few hundred series. A label with unbounded values, such as user_id, request_id, an email or a full URL with ids in it, creates one series per value. In a toy registry, 100,000 user_id values took 37 MB for one metric (about 370 bytes per series), before multiplying by every route and status code, and real systems store far more per series. This is called a cardinality explosion, and it is one of the most common ways to bring down a monitoring system. Rule of thumb: put identifiers in logs and traces, put only small, bounded sets of values (route templates such as /orders/{id}, status classes) in metric labels.
3. Logs
A log line is cheap to write and expensive to read at scale. Make it useful:
- Structure it. Write one JSON object per line with named fields (
level,service,msg,trace_id,user_id,order_id) rather than prose that needs regular expressions to read. Structured logs can be filtered and counted. - Always include a correlation id (the trace id where you have tracing) so one request's lines can be found across services. See "Trace ids and request ids" in section 4.
- Use levels with meaning:
ERRORfor something that needs attention,WARNfor something odd that was handled,INFOfor key business events,DEBUGfor development detail that is off in production. - Never log secrets or needless personal data: passwords, tokens, card numbers, full request bodies. Logs are copied to many places and kept for months (see the logging advice in authentication and sessions).
- Log events, not noise. A line per loop iteration or per cache hit fills the disk and hides the one line you need.
- Decide retention and cost: logs are usually the largest bill in a monitoring system.
What a log call costs
Measured on Python 3.13, writing to memory (so no disk or network), over 200,000 calls:
| Call | Time per call |
|---|---|
DEBUG disabled, message built first with an f-string | 1,225 ns |
DEBUG disabled, lazy %s arguments | 154 ns |
INFO enabled, plain text | 7,727 ns |
INFO enabled, JSON | 10,633 ns |
Two lessons. A disabled log call is not free if you build the message yourself: the f-string formatted the list before the logger could decide not to use it, which made it about eight times slower than passing the arguments lazily. And an enabled call costs microseconds before any disk or network time, so logging inside a hot loop is a performance bug. Production loggers also write asynchronously through a buffer, so a slow log destination does not stall requests (and you must decide what to drop if the buffer fills).
4. Traces
A trace is the record of one request across the system. It is made of spans: each span has a name, a start time, a duration and a parent, so the spans form a tree like the one above. The trace context (the trace id and the id of the current span) is passed between services in a header, most commonly the W3C traceparent:
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
version trace id (16 bytes) parent span id flags (01 = sampled)
The receiving service creates its spans as children of the span in the header. This only works if every hop propagates the context: an HTTP client that forgets the header, a thread pool that drops the context variable, or a message queue whose messages carry no header all cut the trace in two. For a queue (message queues and delivery guarantees), put the trace context in the message's headers, and link the consumer's span to it. OpenTelemetry is the standard set of APIs, SDKs and conventions for producing all three signals, so that the instrumentation does not depend on the vendor that stores them.
Trace ids and request ids
A request id (often an X-Request-Id header) identifies one incoming HTTP request, usually at the edge or at one service. A trace id identifies a whole causal chain of work, and that chain can be larger than one request: it can pass through many services, fan out into parallel calls, continue through a queue minutes later, or start from a scheduled job that has no incoming request at all. They are not interchangeable:
- Several requests can belong to one trace (a page load that triggers a dozen API calls), and a single request can be linked to several traces (a batch consumer that handles messages from many producers uses links instead of one parent).
- Retries may reuse a request id but should show up as separate spans or attempts in one trace.
- A request id from the client is untrusted input. A trace id generated inside your system is yours; accept an incoming one only from places you trust.
Choose the strategy on purpose: use the trace id as the correlation key wherever tracing exists, keep a request id (or an order id, or a job id) as an additional business identifier in logs, and write down which one support staff and customers are given.
Privacy in traces
Spans and their attributes are data you store, search and share, and they capture more than people expect: URLs with query strings (tokens, emails), database statements with their values, user and account ids, headers, error messages that echo input, and baggage items that are passed along to every service the request touches, including third parties. The rule from logging applies here too: no secrets, and no personal data that you do not need. Redact or drop sensitive attributes in the SDK or in the collector before they are stored, prefer parameterised database statements in spans, use route templates and not raw URLs, limit who can query traces, and set a retention period that matches your privacy obligations (a deletion request may have to reach traces and logs as well as the database).
Sampling: you cannot keep everything, so what do you keep?
Tracing every request costs storage and some CPU. Head sampling decides at the start of the request, before anyone knows how it will turn out (for example "keep 1%"). Tail sampling waits until the trace is complete and then decides (keep every trace that was slow or had an error, plus a small share of the rest).
A simulation of 1,000,000 requests, of which 0.1% were slow or failed (949 requests):
| Strategy | Traces stored | Interesting traces kept |
|---|---|---|
| Head sampling at 1% | about 10,000 | 6 of 949 |
| Tail sampling (all slow or failed, plus 1% of the rest) | about 10,860 | 949 of 949 |
Head sampling stores almost the same amount and throws away 99% of the traces you will actually want. Read the second row as a result for an ideal collector, not as a guarantee: in the simulation the decision-maker received every span of every request, had unlimited memory and waited as long as it needed. A real tail sampler keeps all slow or failed traces only if it receives all the spans it needs (a span that arrives after the decision, or from a service that is not instrumented, is lost), has enough memory and CPU for the traces that are waiting for a decision (a decision wait of 30 seconds at 10,000 requests a second means hundreds of thousands of open traces), applies a rule that really matches what you care about, and keeps the result long enough to be looked at. Under overload the collector drops data, and it is usually the unusual traces that suffer. Tail sampling needs a collector that holds all the spans of a trace in memory until it is complete (and every span of the trace must reach the same collector), which is why it costs more to run. A common compromise: head-sample cheaply for the bulk, and force-keep traces from requests that are marked as errors or exceed a latency threshold at the edge.
5. From signals to alerts
An alert should mean a person must act now. Alerting on causes (CPU is at 85%) pages people for things users never notice; alert on symptoms (users are seeing errors or slowness) and use the other signals to diagnose.
A service level indicator (SLI) is a measured ratio, such as "the share of requests that succeeded and took under 300 ms". A service level objective (SLO) is the target for it (99.9% over 30 days). The gap between the target and 100% is the error budget, the amount of failure you can afford:
| SLO over 30 days | Failure allowed |
|---|---|
| 99.0% | 432 minutes (7.2 hours) |
| 99.9% | 43.2 minutes |
| 99.95% | 21.6 minutes |
| 99.99% | 4.3 minutes |
A burn rate is how fast you are spending the budget: 1 means you would use exactly all of it in 30 days, and 14.4 means you would spend 2% of it in one hour. The method in Google's SRE workbook pages a person on a high burn rate over a short window (for example 14.4 over one hour) and opens a ticket for a slow burn over a long window (1 over three days), each confirmed by a short window so that an alert clears quickly once the problem is fixed. This avoids both alarms for brief blips and silence during a slow leak. Many teams also add a latency SLI from the histogram, since an endpoint that answers every request after 10 seconds is "up" and useless.
6. Common mistakes
- Charting the average latency instead of percentiles, or averaging percentiles from several instances.
- Histogram buckets that do not surround the latency target, so the quantile is a guess.
- Unbounded values (user ids, request ids, raw URLs) as metric labels.
- Unstructured logs without a correlation id.
- Logging secrets, tokens or whole request bodies.
- Logging inside hot loops, or building the message before checking the level.
- A trace that breaks at the first queue, thread pool or HTTP client that does not forward the context.
- Head sampling only, so rare slow or failed requests are discarded.
- Alerts on causes (CPU, memory) instead of symptoms, so people learn to ignore the pager.
- Dashboards no one looks at and alerts no one acts on, with no owner and no runbook.
7. Things to mention in an interview
- Name the three signals and the question each answers: metrics say something is wrong, traces say where, logs say why, joined by a trace id.
- Explain percentiles and why not averages, with fan-out amplifying the tail.
- Explain histograms and aggregation: buckets can be summed across instances, percentiles cannot be averaged, and bucket choice limits accuracy.
- Explain cardinality and what belongs in a label versus a log.
- Describe trace context propagation (
traceparent) and how it crosses queues. - Compare head and tail sampling with the cost of each.
- Define SLI, SLO and error budget, and alert on symptoms and burn rate.
- Mention the signals to start with: RED for services, USE for resources.
Questions interviewers like to add:
- "Latency looks fine on average, but users complain. Why?" The tail: look at p95 and p99, and at fan-out.
- "Why did our metrics bill explode after a release?" A new label with unbounded values (user or request id) created one series per value.
- "How do you find why one request was slow?" Find its trace by id, read the slowest span, then read the logs for that trace id.
- "What do you do when the trace shows a gap?" Some hop did not propagate the context, or the time was spent in uninstrumented code or in a queue.
- "How would you alert on a payment service?" On the SLO burn rate of successful, fast payments, and not on the CPU of its servers.
Quick recap
- Metrics say that something is wrong, traces say where, logs say why, and a trace id in all three joins them.
- Latency has a tail: with 2% slow requests the mean was 60 ms, the median 20 ms and the p99 1,995 ms; a page with twenty calls hit a slow one a third of the time.
- Percentiles cannot be averaged (an average of p99s said 559 ms; the real p99 was 31 ms); record histograms, sum the buckets, and choose buckets near your target.
- Metric labels must be bounded; identifiers belong in logs and traces. A
user_idlabel cost 37 MB per 100,000 users in a toy registry. - Log as structured events with a correlation id and never secrets; a disabled call with lazy arguments cost 154 ns against 1,225 ns when the message was built first.
- Propagate
traceparentacross every hop, including queues. In a simulation, head sampling at 1% kept 6 of 949 slow or failed traces, and a tail sampler that saw every span kept all of them; a real collector keeps them only within its capacity and its decision window. - Alert on symptoms using SLOs and burn rates: a 99.9% target allows 43.2 minutes of failure a month.
Appendix: the experiment code
Python 3.13, standard library only. These are teaching sketches, not production instrumentation (see the note in section 1). Run each file with python3. Results vary a little with the random seed and the machine.
Latency statistics: means, percentiles, merging and bucket error
import random, statistics, bisect
random.seed(7)
# 1. mean vs percentiles: 98% of requests take about 20 ms, 2% hit a slow path of about 2 s
lat = [random.gauss(20, 4) for _ in range(98_000)] + [random.gauss(2000, 300) for _ in range(2_000)]
lat = sorted(max(1, x) for x in lat)
q = lambda p: lat[int(len(lat) * p) - 1]
print("1. 100,000 requests")
print(f" mean {statistics.mean(lat):.0f} ms p50 {q(.5):.0f} ms p90 {q(.9):.0f} ms p99 {q(.99):.0f} ms p99.9 {q(.999):.0f} ms max {lat[-1]:.0f} ms")
# 2. a page makes 20 backend calls: how often does a user hit at least one slow one?
for n in (1, 5, 20, 50):
print(f" page with {n:>2} calls: {100 * (1 - 0.98 ** n):4.1f}% of page loads include a call from the slow 2%")
# 3. percentiles cannot be averaged: two servers, very different traffic
a = [random.gauss(20, 4) for _ in range(9_950)] + [random.gauss(900, 100) for _ in range(50)] # busy, 0.5% slow
b = [random.gauss(20, 4) for _ in range(90)] + [random.gauss(900, 100) for _ in range(10)] # quiet, 10% slow
p99 = lambda xs: sorted(xs)[int(len(xs) * .99) - 1]
print("3. two servers")
print(f" p99 A (10,000 requests) {p99(a):.0f} ms, p99 B (100 requests) {p99(b):.0f} ms, average of the two {(p99(a) + p99(b)) / 2:.0f} ms, p99 of all the requests together {p99(a + b):.0f} ms")
# 4. percentiles estimated from histogram buckets
def estimate(xs, bounds, p):
counts = [0] * (len(bounds) + 1)
for x in xs: counts[bisect.bisect_left(bounds, x)] += 1
target, seen = p * len(xs), 0
for i, c in enumerate(counts):
if seen + c >= target:
lo = bounds[i - 1] if i > 0 else 0
hi = bounds[i] if i < len(bounds) else bounds[-1]
return lo + (hi - lo) * (target - seen) / c # linear interpolation inside the bucket
seen += c
exact = p99(lat)
for name, bounds in (("buckets 10, 100, 1000, 10000", [10, 100, 1000, 10000]),
("buckets 5, 10, 25, 50, 100, 250, 500, 1000, 2500", [5, 10, 25, 50, 100, 250, 500, 1000, 2500])):
print(f"4. p99 from {name}: estimate {estimate(lat, bounds, .99):.0f} ms, exact {exact:.0f} ms")
for name, bounds in (("buckets 10, 100, 1000, 10000", [10, 100, 1000, 10000]),
("buckets 5, 10, 25, 50, 100, 250, 500, 1000, 2500", [5, 10, 25, 50, 100, 250, 500, 1000, 2500])):
print(f" p50 from {name}: estimate {estimate(lat, bounds, .5):.0f} ms, exact {q(.5):.0f} ms")
A two-service example with metrics, traces and logs
import bisect, contextvars, json, secrets, threading, time, urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
# ---------------- the three signals, hand-made ----------------
ctx = contextvars.ContextVar("ctx", default=None) # (trace_id, current span id)
LOGS, SPANS, LOCK = [], [], threading.Lock()
def log(service, level, msg, **fields): # structured log line: one JSON object
c = ctx.get() or (None, None)
rec = {"ts": round(time.time(), 3), "level": level, "service": service, "msg": msg,
"trace_id": c[0], **fields}
with LOCK: LOGS.append(rec)
class span: # a timed unit of work with a parent
def __init__(self, service, name, trace_id=None, parent=None):
c = ctx.get()
self.service, self.name = service, name
self.trace_id = trace_id or (c[0] if c else secrets.token_hex(16))
self.parent = parent or (c[1] if c else None)
self.id = secrets.token_hex(8)
def __enter__(self):
self.t0 = time.perf_counter(); self.token = ctx.set((self.trace_id, self.id)); return self
def __exit__(self, *e):
ctx.reset(self.token)
with LOCK: SPANS.append(dict(trace_id=self.trace_id, id=self.id, parent=self.parent, service=self.service,
name=self.name, ms=(time.perf_counter() - self.t0) * 1000))
def header(self): return f"00-{self.trace_id}-{self.id}-01" # W3C traceparent
BUCKETS = [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5]
class Histogram:
def __init__(self): self.counts = [0] * (len(BUCKETS) + 1); self.sum = 0; self.n = 0
def observe(self, s): self.counts[bisect.bisect_left(BUCKETS, s)] += 1; self.sum += s; self.n += 1
def render(self, name):
out, run = [], 0
for b, c in zip(BUCKETS, self.counts): run += c; out.append(f'{name}_bucket{{le="{b}"}} {run}')
out.append(f'{name}_bucket{{le="+Inf"}} {self.n}'); out.append(f"{name}_sum {self.sum:.3f}"); out.append(f"{name}_count {self.n}")
return "\n".join(out)
def quantile(self, p):
target, seen = p * self.n, 0
for i, c in enumerate(self.counts):
if seen + c >= target and c:
lo = BUCKETS[i - 1] if i else 0; hi = BUCKETS[i] if i < len(BUCKETS) else BUCKETS[-1]
return lo + (hi - lo) * (target - seen) / c
seen += c
LAT, REQS = Histogram(), {}
def parse_parent(h):
if not h: return None, None
_, tid, sid, _ = h.split("-"); return tid, sid
# ---------------- two services ----------------
cache = {"sku-1": 12, "sku-2": 15}
class Catalog(BaseHTTPRequestHandler):
def log_message(self, *a): pass
def do_GET(self):
tid, parent = parse_parent(self.headers.get("traceparent"))
sku = self.path.split("=")[1]
with span("catalog", "GET /price", tid, parent):
with span("catalog", "cache.lookup"):
hit = sku in cache
if hit: log("catalog", "INFO", "cache hit", sku=sku)
else:
log("catalog", "WARN", "cache miss, reading the database", sku=sku)
with span("catalog", "db.query"): time.sleep(0.3)
cache[sku] = 20
body = json.dumps({"sku": sku, "price": cache[sku]}).encode()
self.send_response(200); self.send_header("Content-Length", str(len(body))); self.end_headers(); self.wfile.write(body)
class Web(BaseHTTPRequestHandler):
def log_message(self, *a): pass
def do_GET(self):
t0 = time.perf_counter()
with span("web", "GET /checkout") as root:
sku = self.path.split("=")[1]
log("web", "INFO", "checkout started", sku=sku)
with span("web", "call catalog") as s:
req = urllib.request.Request(f"http://127.0.0.1:{CATALOG_PORT}/price?sku={sku}", headers={"traceparent": s.header()})
urllib.request.urlopen(req).read()
log("web", "INFO", "checkout finished", sku=sku)
dt = time.perf_counter() - t0
LAT.observe(dt); REQS[200] = REQS.get(200, 0) + 1
self.send_response(200); self.send_header("Content-Length", "2"); self.end_headers(); self.wfile.write(b"ok")
cat = ThreadingHTTPServer(("127.0.0.1", 0), Catalog); CATALOG_PORT = cat.server_address[1]
web = ThreadingHTTPServer(("127.0.0.1", 0), Web); WEB_PORT = web.server_address[1]
for s in (cat, web): threading.Thread(target=s.serve_forever, daemon=True).start()
# 40 checkouts; every fifth one asks for a SKU that is not cached yet
for i in range(40):
sku = f"sku-{i}" if i % 5 == 0 and i >= 10 else ("sku-1" if i % 2 else "sku-2")
urllib.request.urlopen(f"http://127.0.0.1:{WEB_PORT}/checkout?sku={sku}").read()
print("== METRICS: the question 'is something wrong?'")
print(f"requests={REQS[200]} mean={LAT.sum / LAT.n * 1000:.0f} ms p50~{LAT.quantile(.5) * 1000:.0f} ms p95~{LAT.quantile(.95) * 1000:.0f} ms p99~{LAT.quantile(.99) * 1000:.0f} ms")
print(LAT.render("checkout_duration_seconds"))
roots = [s for s in SPANS if s["name"] == "GET /checkout"]
slow = max(roots, key=lambda s: s["ms"])
print(f"\n== TRACES: the question 'where did the time go?' (slowest trace {slow['trace_id'][:8]}..., {slow['ms']:.0f} ms)")
mine = sorted((s for s in SPANS if s["trace_id"] == slow["trace_id"]), key=lambda s: -s["ms"])
depth = {None: -1}
byid = {s["id"]: s for s in mine}
def d(s): return 0 if s["parent"] not in byid else 1 + d(byid[s["parent"]])
for s in sorted(mine, key=lambda s: (d(s), -s["ms"])): print(" " * d(s) + f"{s['service']:8} {s['name']:16} {s['ms']:6.0f} ms")
print(f"\n== LOGS: the question 'why?' (only this trace)")
for r in LOGS:
if r["trace_id"] == slow["trace_id"]: print(json.dumps(r))
Cardinality, log cost, sampling and error budgets
import io, json, logging, random, time, tracemalloc
# ---- 1. metric cardinality: every distinct label combination is a separate time series
class Series: # roughly what a histogram series holds: 12 counters + a sum
__slots__ = ("counts", "sum")
def __init__(self): self.counts = [0] * 12; self.sum = 0.0
def memory_for(n_series):
tracemalloc.start(); reg = {}
for i in range(n_series): reg[("GET", "/checkout", "200", f"user-{i}")] = Series()
cur, _ = tracemalloc.get_traced_memory(); tracemalloc.stop(); return cur
print("1. cost of a label")
print(" method x route x status = 4 x 25 x 6 = 600 series")
for n in (10_000, 100_000):
m = memory_for(n); print(f" a user_id label with {n:>7,} users: {m / 1e6:6.1f} MB in memory for one metric ({m / n:.0f} bytes per series), before multiplying by routes and statuses")
# ---- 2. what a log call costs
N = 200_000
def timed(fn):
t = time.perf_counter(); fn(); return (time.perf_counter() - t) / N * 1e9
buf = io.StringIO()
lg = logging.getLogger("t"); lg.propagate = False
h = logging.StreamHandler(buf); lg.addHandler(h)
user, items = "u-42", list(range(20))
lg.setLevel(logging.WARNING) # DEBUG is off
r = {}
r["disabled, f-string built first"] = timed(lambda: [lg.debug(f"cart for {user}: {items!r}") for _ in range(N)])
r["disabled, lazy % arguments"] = timed(lambda: [lg.debug("cart for %s: %r", user, items) for _ in range(N)])
lg.setLevel(logging.INFO)
r["enabled, plain text"] = timed(lambda: [lg.info("cart for %s: %r", user, items) for _ in range(N)])
class J(logging.Formatter):
def format(self, rec): return json.dumps({"level": rec.levelname, "msg": rec.getMessage(), "user": user, "n_items": len(items)})
h.setFormatter(J())
r["enabled, JSON"] = timed(lambda: [lg.info("cart for %s: %r", user, items) for _ in range(N)])
print("2. cost of one log call (nanoseconds, Python 3.13, output to memory)")
for k, v in r.items(): print(f" {k:32} {v:8.0f} ns")
# ---- 3. sampling traces
random.seed(3)
TOTAL, P_BAD = 1_000_000, 0.001 # 0.1% of requests are slow or failed
bad = sum(1 for _ in range(TOTAL) if random.random() < P_BAD)
print("3. sampling 1,000,000 requests of which about 0.1% are slow or failed")
head = sum(1 for _ in range(bad) if random.random() < 0.01)
print(f" head sampling at 1%: stores about {TOTAL * 0.01:,.0f} traces; of the {bad} interesting ones it kept {head}")
tail_kept = bad + sum(1 for _ in range(TOTAL - bad) if random.random() < 0.01)
print(f" tail sampling (keep every slow or failed trace, plus 1% of the rest): stores about {tail_kept:,} traces; kept {bad} of {bad}")
# ---- 4. SLO arithmetic
print("4. error budget over 30 days (43,200 minutes)")
for slo in (99.0, 99.9, 99.95, 99.99):
mins = 43200 * (1 - slo / 100); print(f" {slo}% -> {mins:7.1f} minutes of failure allowed ({mins / 60:5.2f} hours)")
print(" burn rate 14.4 spends 2% of a 30-day budget in 1 hour; burn rate 1 spends it exactly in 30 days")
Sources
- Betsy Beyer et al., Site Reliability Engineering, chapter "Monitoring Distributed Systems". The four golden signals, symptoms versus causes.
- Google, The Site Reliability Workbook, chapter "Alerting on SLOs". Error budgets and multi-window burn-rate alerts.
- W3C, Trace Context. The
traceparentheader format. - Prometheus, Histograms and summaries. Why quantiles from histograms can be aggregated and summaries cannot.
- OpenTelemetry documentation. Concepts for traces, metrics and logs, and sampling.
Practise this topic: DevOps & Cloud