CS Fundamentals Trainer · Guides

Observability: logs, metrics and traces

What each of the three signals answers, why averages hide slow requests and percentiles cannot be averaged, label cardinality, log cost, trace propagation and sampling, and alerting on SLO burn rates, with a two-service example in Python.

26 min read

When a system misbehaves in production you cannot attach a debugger, and the code that ran is on a machine you did not touch. Observability is how well you can work out what a system is doing, and why, from the data it emits. In practice that data comes in three kinds: metrics (numbers over time), logs (records of events) and traces (the path of one request through many services). They answer different questions, they cost different amounts, and each has a classic way of going wrong. This guide shows all three on a small two-service example, then covers the traps that matter most: averages hiding slow requests, percentiles that cannot be added, labels that explode the number of metric series, logs that cost more than they are worth, traces sampled in a way that loses exactly the ones you need, and alerts built from error budgets. The code is in the appendix; each number is one run on a small machine, so the behaviour matters and the exact figures do not.

1. Three signals, three questions

MetricsLogsTraces
What it isnumbers aggregated over time (counts, rates, durations)one record per event, with detailsthe tree of timed steps one request took across services
AnswersIs something wrong, and how much?Why did this happen?Where did the time go (or the error come from)?
Cost grows withthe number of distinct series (labels), not with traffictraffic and the amount you write per eventtraffic (unless you sample)
Good fordashboards and alertsdebugging one incident in detailfinding the slow or failing hop
Weak atno detail about individual requestshard to see patterns, expensive to keep foreverneeds every service to pass the context along

They work best joined together: an alert fires from a metric, a trace shows which service and call is slow, and the logs of that trace say why. The join key is an identifier, the trace id, written into every log line and every span.

A worked example

I wrote two small services and instrumented them by hand to show the ideas: a web service whose /checkout calls a catalog service for a price. (The hand-made code is a teaching tool. It is not an implementation of the OpenTelemetry data model: it has no resource or instrumentation-scope information, span kinds, status codes, events, links, attributes with standard names, tracestate, baggage or exporters, and it does not act on the sampled flag. In real services use the OpenTelemetry libraries and their automatic instrumentation.) Every fifth new product is missing from the catalog's cache, so that request reads the database (a 300 ms sleep in the test). Forty checkouts later:

Metrics said that something was off: the mean was 46 ms, but the 95th percentile estimated from the histogram was about 417 ms. Six of the 40 requests landed in the 250 to 500 ms bucket and the other 34 in the first bucket.

The trace of the slowest request showed where the time went:

web      GET /checkout       302 ms
  web      call catalog        302 ms
    catalog  GET /price          300 ms
      catalog  db.query            300 ms
      catalog  cache.lookup          0 ms

The logs with that trace id said why:

{"level": "INFO", "service": "web",     "msg": "checkout started",                  "trace_id": "021e8cdc...", "sku": "sku-15"}
{"level": "WARN", "service": "catalog", "msg": "cache miss, reading the database",  "trace_id": "021e8cdc...", "sku": "sku-15"}
{"level": "INFO", "service": "web",     "msg": "checkout finished",                 "trace_id": "021e8cdc...", "sku": "sku-15"}

The metric could not say which request, the trace could not say that it was a cache miss, and the log alone would have been one line among thousands. Each signal supplied what the others lacked.

Diagram: The trace context travels with the request

2. Metrics

A metric is a number with a name and labels (http_requests_total{route="/checkout", status="200"}). Each distinct set of label values is a separate time series. Four kinds cover almost everything:

What to measure first: for a request-handling service, the RED signals, Rate, Errors and Duration, per route. For a resource (CPU, disk, a connection pool), the USE signals: Utilisation, Saturation and Errors. Google's SRE book names four golden signals: latency, traffic, errors and saturation. They are the same idea in different words.

Averages hide the requests that hurt

In a test of 100,000 requests where 98% took about 20 ms and 2% took about 2 s (a slow path):

StatisticValue
mean60 ms
median (p50)20 ms
p9026 ms
p991,995 ms
p99.92,506 ms

The mean of 60 ms describes no real request: it is neither the fast ones nor the slow ones. The median and p90 look perfect while one in fifty requests takes two seconds. Always chart percentiles (p50, p95, p99) and not the average of latency.

Tail latency also multiplies with fan-out. If a page makes several backend calls and each has a 2% chance of being slow, the chance that a page load hits at least one slow call is 1 - 0.98^n: 2% for one call, 9.6% for five, 33% for twenty and 64% for fifty. Calls that look rare in a single service dominate the experience of a page.

Percentiles cannot be averaged

Two servers: A is busy (10,000 requests, 0.5% slow) and B is quiet (100 requests, 10% slow). Server A's p99 was 30 ms and B's p99 was 1,087 ms. The average of the two, 559 ms, is meaningless: the p99 of all 10,100 requests together was 31 ms. A dashboard that averages the per-instance p99 values reports a problem that most users do not have (or, with other traffic, hides one that they do).

The fix is to record histograms and add their buckets across instances, then compute the quantile from the combined buckets. That is why Prometheus advises histograms over summaries when you need to aggregate. The price is precision: the quantile is estimated by interpolating inside a bucket, so the buckets must suit your latency:

Buckets (ms)p99 estimateexact p99p50 estimateexact p50
10, 100, 1000, 100005,5001,9955620
5, 10, 25, 50, 100, 250, 500, 1000, 25001,7911,9951920

Coarse buckets turned a 2 s p99 into 5.5 s. Put bucket boundaries around the values you care about, such as your latency target.

Labels are a budget

Every combination of label values is a new series that the monitoring system stores, indexes and charts. Labels like method, route and status give a few hundred series. A label with unbounded values, such as user_id, request_id, an email or a full URL with ids in it, creates one series per value. In a toy registry, 100,000 user_id values took 37 MB for one metric (about 370 bytes per series), before multiplying by every route and status code, and real systems store far more per series. This is called a cardinality explosion, and it is one of the most common ways to bring down a monitoring system. Rule of thumb: put identifiers in logs and traces, put only small, bounded sets of values (route templates such as /orders/{id}, status classes) in metric labels.

3. Logs

A log line is cheap to write and expensive to read at scale. Make it useful:

What a log call costs

Measured on Python 3.13, writing to memory (so no disk or network), over 200,000 calls:

CallTime per call
DEBUG disabled, message built first with an f-string1,225 ns
DEBUG disabled, lazy %s arguments154 ns
INFO enabled, plain text7,727 ns
INFO enabled, JSON10,633 ns

Two lessons. A disabled log call is not free if you build the message yourself: the f-string formatted the list before the logger could decide not to use it, which made it about eight times slower than passing the arguments lazily. And an enabled call costs microseconds before any disk or network time, so logging inside a hot loop is a performance bug. Production loggers also write asynchronously through a buffer, so a slow log destination does not stall requests (and you must decide what to drop if the buffer fills).

4. Traces

A trace is the record of one request across the system. It is made of spans: each span has a name, a start time, a duration and a parent, so the spans form a tree like the one above. The trace context (the trace id and the id of the current span) is passed between services in a header, most commonly the W3C traceparent:

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
             version  trace id (16 bytes)          parent span id   flags (01 = sampled)

The receiving service creates its spans as children of the span in the header. This only works if every hop propagates the context: an HTTP client that forgets the header, a thread pool that drops the context variable, or a message queue whose messages carry no header all cut the trace in two. For a queue (message queues and delivery guarantees), put the trace context in the message's headers, and link the consumer's span to it. OpenTelemetry is the standard set of APIs, SDKs and conventions for producing all three signals, so that the instrumentation does not depend on the vendor that stores them.

Trace ids and request ids

A request id (often an X-Request-Id header) identifies one incoming HTTP request, usually at the edge or at one service. A trace id identifies a whole causal chain of work, and that chain can be larger than one request: it can pass through many services, fan out into parallel calls, continue through a queue minutes later, or start from a scheduled job that has no incoming request at all. They are not interchangeable:

Choose the strategy on purpose: use the trace id as the correlation key wherever tracing exists, keep a request id (or an order id, or a job id) as an additional business identifier in logs, and write down which one support staff and customers are given.

Privacy in traces

Spans and their attributes are data you store, search and share, and they capture more than people expect: URLs with query strings (tokens, emails), database statements with their values, user and account ids, headers, error messages that echo input, and baggage items that are passed along to every service the request touches, including third parties. The rule from logging applies here too: no secrets, and no personal data that you do not need. Redact or drop sensitive attributes in the SDK or in the collector before they are stored, prefer parameterised database statements in spans, use route templates and not raw URLs, limit who can query traces, and set a retention period that matches your privacy obligations (a deletion request may have to reach traces and logs as well as the database).

Sampling: you cannot keep everything, so what do you keep?

Tracing every request costs storage and some CPU. Head sampling decides at the start of the request, before anyone knows how it will turn out (for example "keep 1%"). Tail sampling waits until the trace is complete and then decides (keep every trace that was slow or had an error, plus a small share of the rest).

A simulation of 1,000,000 requests, of which 0.1% were slow or failed (949 requests):

StrategyTraces storedInteresting traces kept
Head sampling at 1%about 10,0006 of 949
Tail sampling (all slow or failed, plus 1% of the rest)about 10,860949 of 949

Head sampling stores almost the same amount and throws away 99% of the traces you will actually want. Read the second row as a result for an ideal collector, not as a guarantee: in the simulation the decision-maker received every span of every request, had unlimited memory and waited as long as it needed. A real tail sampler keeps all slow or failed traces only if it receives all the spans it needs (a span that arrives after the decision, or from a service that is not instrumented, is lost), has enough memory and CPU for the traces that are waiting for a decision (a decision wait of 30 seconds at 10,000 requests a second means hundreds of thousands of open traces), applies a rule that really matches what you care about, and keeps the result long enough to be looked at. Under overload the collector drops data, and it is usually the unusual traces that suffer. Tail sampling needs a collector that holds all the spans of a trace in memory until it is complete (and every span of the trace must reach the same collector), which is why it costs more to run. A common compromise: head-sample cheaply for the bulk, and force-keep traces from requests that are marked as errors or exceed a latency threshold at the edge.

5. From signals to alerts

An alert should mean a person must act now. Alerting on causes (CPU is at 85%) pages people for things users never notice; alert on symptoms (users are seeing errors or slowness) and use the other signals to diagnose.

A service level indicator (SLI) is a measured ratio, such as "the share of requests that succeeded and took under 300 ms". A service level objective (SLO) is the target for it (99.9% over 30 days). The gap between the target and 100% is the error budget, the amount of failure you can afford:

SLO over 30 daysFailure allowed
99.0%432 minutes (7.2 hours)
99.9%43.2 minutes
99.95%21.6 minutes
99.99%4.3 minutes

A burn rate is how fast you are spending the budget: 1 means you would use exactly all of it in 30 days, and 14.4 means you would spend 2% of it in one hour. The method in Google's SRE workbook pages a person on a high burn rate over a short window (for example 14.4 over one hour) and opens a ticket for a slow burn over a long window (1 over three days), each confirmed by a short window so that an alert clears quickly once the problem is fixed. This avoids both alarms for brief blips and silence during a slow leak. Many teams also add a latency SLI from the histogram, since an endpoint that answers every request after 10 seconds is "up" and useless.

6. Common mistakes

  1. Charting the average latency instead of percentiles, or averaging percentiles from several instances.
  2. Histogram buckets that do not surround the latency target, so the quantile is a guess.
  3. Unbounded values (user ids, request ids, raw URLs) as metric labels.
  4. Unstructured logs without a correlation id.
  5. Logging secrets, tokens or whole request bodies.
  6. Logging inside hot loops, or building the message before checking the level.
  7. A trace that breaks at the first queue, thread pool or HTTP client that does not forward the context.
  8. Head sampling only, so rare slow or failed requests are discarded.
  9. Alerts on causes (CPU, memory) instead of symptoms, so people learn to ignore the pager.
  10. Dashboards no one looks at and alerts no one acts on, with no owner and no runbook.

7. Things to mention in an interview

Questions interviewers like to add:

Quick recap

  1. Metrics say that something is wrong, traces say where, logs say why, and a trace id in all three joins them.
  2. Latency has a tail: with 2% slow requests the mean was 60 ms, the median 20 ms and the p99 1,995 ms; a page with twenty calls hit a slow one a third of the time.
  3. Percentiles cannot be averaged (an average of p99s said 559 ms; the real p99 was 31 ms); record histograms, sum the buckets, and choose buckets near your target.
  4. Metric labels must be bounded; identifiers belong in logs and traces. A user_id label cost 37 MB per 100,000 users in a toy registry.
  5. Log as structured events with a correlation id and never secrets; a disabled call with lazy arguments cost 154 ns against 1,225 ns when the message was built first.
  6. Propagate traceparent across every hop, including queues. In a simulation, head sampling at 1% kept 6 of 949 slow or failed traces, and a tail sampler that saw every span kept all of them; a real collector keeps them only within its capacity and its decision window.
  7. Alert on symptoms using SLOs and burn rates: a 99.9% target allows 43.2 minutes of failure a month.

Appendix: the experiment code

Python 3.13, standard library only. These are teaching sketches, not production instrumentation (see the note in section 1). Run each file with python3. Results vary a little with the random seed and the machine.

Latency statistics: means, percentiles, merging and bucket error

import random, statistics, bisect
random.seed(7)

# 1. mean vs percentiles: 98% of requests take about 20 ms, 2% hit a slow path of about 2 s
lat = [random.gauss(20, 4) for _ in range(98_000)] + [random.gauss(2000, 300) for _ in range(2_000)]
lat = sorted(max(1, x) for x in lat)
q = lambda p: lat[int(len(lat) * p) - 1]
print("1. 100,000 requests")
print(f"   mean {statistics.mean(lat):.0f} ms   p50 {q(.5):.0f} ms   p90 {q(.9):.0f} ms   p99 {q(.99):.0f} ms   p99.9 {q(.999):.0f} ms   max {lat[-1]:.0f} ms")

# 2. a page makes 20 backend calls: how often does a user hit at least one slow one?
for n in (1, 5, 20, 50):
    print(f"   page with {n:>2} calls: {100 * (1 - 0.98 ** n):4.1f}% of page loads include a call from the slow 2%")

# 3. percentiles cannot be averaged: two servers, very different traffic
a = [random.gauss(20, 4) for _ in range(9_950)] + [random.gauss(900, 100) for _ in range(50)]      # busy, 0.5% slow
b = [random.gauss(20, 4) for _ in range(90)] + [random.gauss(900, 100) for _ in range(10)]        # quiet, 10% slow
p99 = lambda xs: sorted(xs)[int(len(xs) * .99) - 1]
print("3. two servers")
print(f"   p99 A (10,000 requests) {p99(a):.0f} ms, p99 B (100 requests) {p99(b):.0f} ms, average of the two {(p99(a) + p99(b)) / 2:.0f} ms, p99 of all the requests together {p99(a + b):.0f} ms")

# 4. percentiles estimated from histogram buckets
def estimate(xs, bounds, p):
    counts = [0] * (len(bounds) + 1)
    for x in xs: counts[bisect.bisect_left(bounds, x)] += 1
    target, seen = p * len(xs), 0
    for i, c in enumerate(counts):
        if seen + c >= target:
            lo = bounds[i - 1] if i > 0 else 0
            hi = bounds[i] if i < len(bounds) else bounds[-1]
            return lo + (hi - lo) * (target - seen) / c      # linear interpolation inside the bucket
        seen += c
exact = p99(lat)
for name, bounds in (("buckets 10, 100, 1000, 10000", [10, 100, 1000, 10000]),
                     ("buckets 5, 10, 25, 50, 100, 250, 500, 1000, 2500", [5, 10, 25, 50, 100, 250, 500, 1000, 2500])):
    print(f"4. p99 from {name}: estimate {estimate(lat, bounds, .99):.0f} ms, exact {exact:.0f} ms")
for name, bounds in (("buckets 10, 100, 1000, 10000", [10, 100, 1000, 10000]),
                     ("buckets 5, 10, 25, 50, 100, 250, 500, 1000, 2500", [5, 10, 25, 50, 100, 250, 500, 1000, 2500])):
    print(f"   p50 from {name}: estimate {estimate(lat, bounds, .5):.0f} ms, exact {q(.5):.0f} ms")

A two-service example with metrics, traces and logs

import bisect, contextvars, json, secrets, threading, time, urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

# ---------------- the three signals, hand-made ----------------
ctx = contextvars.ContextVar("ctx", default=None)          # (trace_id, current span id)
LOGS, SPANS, LOCK = [], [], threading.Lock()

def log(service, level, msg, **fields):                     # structured log line: one JSON object
    c = ctx.get() or (None, None)
    rec = {"ts": round(time.time(), 3), "level": level, "service": service, "msg": msg,
           "trace_id": c[0], **fields}
    with LOCK: LOGS.append(rec)

class span:                                                  # a timed unit of work with a parent
    def __init__(self, service, name, trace_id=None, parent=None):
        c = ctx.get()
        self.service, self.name = service, name
        self.trace_id = trace_id or (c[0] if c else secrets.token_hex(16))
        self.parent = parent or (c[1] if c else None)
        self.id = secrets.token_hex(8)
    def __enter__(self):
        self.t0 = time.perf_counter(); self.token = ctx.set((self.trace_id, self.id)); return self
    def __exit__(self, *e):
        ctx.reset(self.token)
        with LOCK: SPANS.append(dict(trace_id=self.trace_id, id=self.id, parent=self.parent, service=self.service,
                                     name=self.name, ms=(time.perf_counter() - self.t0) * 1000))
    def header(self): return f"00-{self.trace_id}-{self.id}-01"   # W3C traceparent

BUCKETS = [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5]
class Histogram:
    def __init__(self): self.counts = [0] * (len(BUCKETS) + 1); self.sum = 0; self.n = 0
    def observe(self, s): self.counts[bisect.bisect_left(BUCKETS, s)] += 1; self.sum += s; self.n += 1
    def render(self, name):
        out, run = [], 0
        for b, c in zip(BUCKETS, self.counts): run += c; out.append(f'{name}_bucket{{le="{b}"}} {run}')
        out.append(f'{name}_bucket{{le="+Inf"}} {self.n}'); out.append(f"{name}_sum {self.sum:.3f}"); out.append(f"{name}_count {self.n}")
        return "\n".join(out)
    def quantile(self, p):
        target, seen = p * self.n, 0
        for i, c in enumerate(self.counts):
            if seen + c >= target and c:
                lo = BUCKETS[i - 1] if i else 0; hi = BUCKETS[i] if i < len(BUCKETS) else BUCKETS[-1]
                return lo + (hi - lo) * (target - seen) / c
            seen += c
LAT, REQS = Histogram(), {}

def parse_parent(h):
    if not h: return None, None
    _, tid, sid, _ = h.split("-"); return tid, sid

# ---------------- two services ----------------
cache = {"sku-1": 12, "sku-2": 15}
class Catalog(BaseHTTPRequestHandler):
    def log_message(self, *a): pass
    def do_GET(self):
        tid, parent = parse_parent(self.headers.get("traceparent"))
        sku = self.path.split("=")[1]
        with span("catalog", "GET /price", tid, parent):
            with span("catalog", "cache.lookup"):
                hit = sku in cache
            if hit: log("catalog", "INFO", "cache hit", sku=sku)
            else:
                log("catalog", "WARN", "cache miss, reading the database", sku=sku)
                with span("catalog", "db.query"): time.sleep(0.3)
                cache[sku] = 20
            body = json.dumps({"sku": sku, "price": cache[sku]}).encode()
        self.send_response(200); self.send_header("Content-Length", str(len(body))); self.end_headers(); self.wfile.write(body)

class Web(BaseHTTPRequestHandler):
    def log_message(self, *a): pass
    def do_GET(self):
        t0 = time.perf_counter()
        with span("web", "GET /checkout") as root:
            sku = self.path.split("=")[1]
            log("web", "INFO", "checkout started", sku=sku)
            with span("web", "call catalog") as s:
                req = urllib.request.Request(f"http://127.0.0.1:{CATALOG_PORT}/price?sku={sku}", headers={"traceparent": s.header()})
                urllib.request.urlopen(req).read()
            log("web", "INFO", "checkout finished", sku=sku)
        dt = time.perf_counter() - t0
        LAT.observe(dt); REQS[200] = REQS.get(200, 0) + 1
        self.send_response(200); self.send_header("Content-Length", "2"); self.end_headers(); self.wfile.write(b"ok")

cat = ThreadingHTTPServer(("127.0.0.1", 0), Catalog); CATALOG_PORT = cat.server_address[1]
web = ThreadingHTTPServer(("127.0.0.1", 0), Web); WEB_PORT = web.server_address[1]
for s in (cat, web): threading.Thread(target=s.serve_forever, daemon=True).start()

# 40 checkouts; every fifth one asks for a SKU that is not cached yet
for i in range(40):
    sku = f"sku-{i}" if i % 5 == 0 and i >= 10 else ("sku-1" if i % 2 else "sku-2")
    urllib.request.urlopen(f"http://127.0.0.1:{WEB_PORT}/checkout?sku={sku}").read()

print("== METRICS: the question 'is something wrong?'")
print(f"requests={REQS[200]}  mean={LAT.sum / LAT.n * 1000:.0f} ms  p50~{LAT.quantile(.5) * 1000:.0f} ms  p95~{LAT.quantile(.95) * 1000:.0f} ms  p99~{LAT.quantile(.99) * 1000:.0f} ms")
print(LAT.render("checkout_duration_seconds"))
roots = [s for s in SPANS if s["name"] == "GET /checkout"]
slow = max(roots, key=lambda s: s["ms"])
print(f"\n== TRACES: the question 'where did the time go?' (slowest trace {slow['trace_id'][:8]}..., {slow['ms']:.0f} ms)")
mine = sorted((s for s in SPANS if s["trace_id"] == slow["trace_id"]), key=lambda s: -s["ms"])
depth = {None: -1}
byid = {s["id"]: s for s in mine}
def d(s): return 0 if s["parent"] not in byid else 1 + d(byid[s["parent"]])
for s in sorted(mine, key=lambda s: (d(s), -s["ms"])): print("  " * d(s) + f"{s['service']:8} {s['name']:16} {s['ms']:6.0f} ms")
print(f"\n== LOGS: the question 'why?' (only this trace)")
for r in LOGS:
    if r["trace_id"] == slow["trace_id"]: print(json.dumps(r))

Cardinality, log cost, sampling and error budgets

import io, json, logging, random, time, tracemalloc

# ---- 1. metric cardinality: every distinct label combination is a separate time series
class Series:                                    # roughly what a histogram series holds: 12 counters + a sum
    __slots__ = ("counts", "sum")
    def __init__(self): self.counts = [0] * 12; self.sum = 0.0
def memory_for(n_series):
    tracemalloc.start(); reg = {}
    for i in range(n_series): reg[("GET", "/checkout", "200", f"user-{i}")] = Series()
    cur, _ = tracemalloc.get_traced_memory(); tracemalloc.stop(); return cur
print("1. cost of a label")
print("   method x route x status = 4 x 25 x 6 = 600 series")
for n in (10_000, 100_000):
    m = memory_for(n); print(f"   a user_id label with {n:>7,} users: {m / 1e6:6.1f} MB in memory for one metric ({m / n:.0f} bytes per series), before multiplying by routes and statuses")

# ---- 2. what a log call costs
N = 200_000
def timed(fn):
    t = time.perf_counter(); fn(); return (time.perf_counter() - t) / N * 1e9
buf = io.StringIO()
lg = logging.getLogger("t"); lg.propagate = False
h = logging.StreamHandler(buf); lg.addHandler(h)
user, items = "u-42", list(range(20))
lg.setLevel(logging.WARNING)                   # DEBUG is off
r = {}
r["disabled, f-string built first"] = timed(lambda: [lg.debug(f"cart for {user}: {items!r}") for _ in range(N)])
r["disabled, lazy % arguments"] = timed(lambda: [lg.debug("cart for %s: %r", user, items) for _ in range(N)])
lg.setLevel(logging.INFO)
r["enabled, plain text"] = timed(lambda: [lg.info("cart for %s: %r", user, items) for _ in range(N)])
class J(logging.Formatter):
    def format(self, rec): return json.dumps({"level": rec.levelname, "msg": rec.getMessage(), "user": user, "n_items": len(items)})
h.setFormatter(J())
r["enabled, JSON"] = timed(lambda: [lg.info("cart for %s: %r", user, items) for _ in range(N)])
print("2. cost of one log call (nanoseconds, Python 3.13, output to memory)")
for k, v in r.items(): print(f"   {k:32} {v:8.0f} ns")

# ---- 3. sampling traces
random.seed(3)
TOTAL, P_BAD = 1_000_000, 0.001                  # 0.1% of requests are slow or failed
bad = sum(1 for _ in range(TOTAL) if random.random() < P_BAD)
print("3. sampling 1,000,000 requests of which about 0.1% are slow or failed")
head = sum(1 for _ in range(bad) if random.random() < 0.01)
print(f"   head sampling at 1%: stores about {TOTAL * 0.01:,.0f} traces; of the {bad} interesting ones it kept {head}")
tail_kept = bad + sum(1 for _ in range(TOTAL - bad) if random.random() < 0.01)
print(f"   tail sampling (keep every slow or failed trace, plus 1% of the rest): stores about {tail_kept:,} traces; kept {bad} of {bad}")

# ---- 4. SLO arithmetic
print("4. error budget over 30 days (43,200 minutes)")
for slo in (99.0, 99.9, 99.95, 99.99):
    mins = 43200 * (1 - slo / 100); print(f"   {slo}% -> {mins:7.1f} minutes of failure allowed ({mins / 60:5.2f} hours)")
print("   burn rate 14.4 spends 2% of a 30-day budget in 1 hour; burn rate 1 spends it exactly in 30 days")

Sources

  1. Betsy Beyer et al., Site Reliability Engineering, chapter "Monitoring Distributed Systems". The four golden signals, symptoms versus causes.
  2. Google, The Site Reliability Workbook, chapter "Alerting on SLOs". Error budgets and multi-window burn-rate alerts.
  3. W3C, Trace Context. The traceparent header format.
  4. Prometheus, Histograms and summaries. Why quantiles from histograms can be aggregated and summaries cannot.
  5. OpenTelemetry documentation. Concepts for traces, metrics and logs, and sampling.

Practise this topic: DevOps & Cloud

More guides