CS Fundamentals Trainer · Guides

Microservices, API gateways and service meshes

What splitting into services really costs in latency and availability, how to draw boundaries, synchronous against asynchronous communication, the API gateway and BFF, service discovery and the service mesh, and the strangler fig migration.

40 min read

"Should we use microservices?" is among the most common architecture questions, and the honest answer starts with what the question really asks: should we turn function calls into network calls so that teams can build, deploy and scale parts of the system independently? That trade can be right, and it has a price that is easy to underestimate: every hop adds latency and a new way to fail, and consistency across services becomes your problem. This guide gives you the numbers for that price, then the patterns that make it manageable: service boundaries, communication styles, the API gateway, service discovery, and the service mesh.

I measured the call overhead in Node.js over loopback and simulated the availability and latency arithmetic in Python (code in the appendix). It builds on resilience patterns, observability and distributed transactions, which supply the tools you need once you have a network between your components.

1. What problem do microservices solve?

Mainly an organisational one. A single deployable (a monolith) is simple to build, test, run and debug, and fast: calls are function calls and one database transaction can cover everything. It starts to hurt when many teams must change the same code base: builds get slow, a release means coordinating everyone, one team's bug takes down everything, and one hot component forces you to scale the whole thing.

Splitting the system into independently deployable services buys:

It costs:

Conway's law says that systems mirror the communication structure of the organisation that builds them. Treat it as a design tool: choose service boundaries to match team boundaries, not the other way round. A team of eight with one product rarely needs thirty services. A common, sound path is a modular monolith first (one deployable with strict module boundaries and each module owning its tables) and extraction of a service only when a concrete need appears (a different scaling profile, a different release cadence, a team boundary).

2. The price of a network hop

I measured the same small operation, totalling an order of ten items, three ways in Node.js on a two-core virtual machine:

How the call is madeTime per callCompared with a function call
In-process function call19 ns1x
The same, with a JSON encode and decode of the order (no network)4.7 µs251x
A real HTTP request over loopback with keep-alive, one at a time89 µs median, 562 µs 99th percentileabout 4,700x

Even on loopback, with no wire, no TLS and no routing, a service call is thousands of times slower than a function call, and most of the cost is serialisation, the protocol and the context switches, not the distance. A real network adds more: roughly 0.1 to 1 ms inside a data centre, 10 to 100 ms across a continent or an ocean (typical figures, not measured here), plus TLS, load balancers and queues. And a single pair of cores served about 13,000 such requests per second in total, client and server together, which shows that calls also consume CPU on both sides.

Availability multiplies

If a request needs n services in series and each is 99.9% available, the request is available only 99.9% to the power n:

Services in seriesAvailabilityDowntime per year
199.90%about 526 minutes
399.70%1,575 minutes
599.50%2,623 minutes
1099.00%5,232 minutes (3.6 days)
2098.02%10,413 minutes
5095.12%25,646 minutes (18 days)

This assumes failures are independent, so it is an optimistic estimate. The fixes are the ones from the resilience guide: make dependencies optional (the page works without recommendations: with 3 of 10 optional the availability goes from 99.004% to 99.302%, with 6 optional to 99.601%), use redundancy (two independent replicas of a 99% service give 99.99%, three give 99.9999%) and avoid long synchronous chains (below).

Latency adds, and the tail multiplies

I modelled each call with a median of 20 ms and a long tail (99th percentile 72 ms, a lognormal distribution), and measured the end-to-end time:

PatternMedian99th percentile
1 call20 ms72 ms
a chain of 3 synchronous calls66 ms145 ms
a chain of 5112 ms206 ms
a chain of 8181 ms296 ms
waiting for the slowest of 5 parallel calls37 ms96 ms
the slowest of 10 parallel calls45 ms108 ms
the slowest of 50 parallel calls67 ms139 ms

Chains add: eight hops cost 9 times a single call. Fan-out is gentler on the median but hurts the tail: when a request waits for 50 parallel calls, 39.6% of requests hit at least one call that is slower than the one-in-a-hundred worst case of a single call, and the median is more than three times a single call. This is the "tail at scale" effect (Dean and Barroso): the more services a request touches, the more often it meets somebody's bad moment. Defences: fewer hops, caching, timeouts with fallbacks, and hedged requests (sending a duplicate to another replica when the first is slow) for idempotent reads.

3. Drawing the boundaries

Good boundaries make most changes local to one service. Bad ones create a distributed monolith: the cost of microservices with none of the benefits.

Guidance for finding boundaries:

Warning signs of a distributed monolith:

Who owns a service?

Service boundaries are also people boundaries. Conway's law says that a system's structure tends to copy the communication structure of the organisation that builds it; if four teams must agree before one change ships, the architecture will have four interlocking parts whatever the diagram says. Microservices work when the boundaries are drawn deliberately along both lines (the "inverse Conway manoeuvre": shape the teams to get the architecture you want). In practice:

4. How services talk

Diagram: Synchronous calls and asynchronous events

Synchronous (REST, gRPC): simple to reason about, immediate answer, but the caller depends on the callee being up and fast right now, so availability and latency multiply as shown. Use for queries that need an answer to proceed. gRPC (HTTP/2 with Protocol Buffers) is more compact and fast than JSON over HTTP/1.1, supports deadlines and streaming, and gives a typed contract; REST is more universal for external APIs (see designing a REST API).

Asynchronous (events or commands through a broker): the producer does not wait, services are decoupled in time, spikes are absorbed by the queue, and new consumers can be added without touching the producer. The costs are eventual consistency, harder tracing, duplicate and out-of-order delivery (see message queues and delivery guarantees), and a broker to operate.

A pragmatic split: synchronous for queries and for the one call that the user is waiting on, asynchronous for everything that can happen after the response. An order service can answer "order placed" after saving it and publishing an event; the email, analytics, loyalty points and shipping reactions follow on their own.

For a business process that spans services, use a saga: either choreography (each service reacts to events and emits the next one: decoupled, but the overall flow is hard to see) or orchestration (a coordinator tells each service what to do and handles compensation: easier to follow and change, with the coordinator as a new component). The previous guide on distributed transactions covers both and the limits of idempotency keys.

Contracts and versions

Services evolve separately, so the interface is a contract. Change it compatibly: add optional fields, do not remove or reinterpret existing ones, and use the expand and contract pattern for breaking changes (add the new form, migrate every consumer, remove the old). Consumer-driven contract tests let each consumer publish the requests and responses it relies on, and the provider's pipeline fails when a change would break them. Use a schema (OpenAPI, protobuf, Avro) and a compatibility check in CI.

5. The API gateway

Clients (a browser, a phone app, a partner) should not know your internal services. An API gateway is a single entry point that routes external requests to the services behind it and handles the cross-cutting concerns in one place:

Diagram: An API gateway in front of the services

Typical jobs:

A variation is the Backend for Frontend (BFF): a separate gateway per client type (web, mobile, partner) owned by that client's team, shaped to its needs, instead of one general-purpose gateway that serves everybody badly.

Pitfalls: the gateway is on the path of every request, so it must be highly available and scaled horizontally; it becomes a bottleneck and a deploy dependency if business logic accumulates in it (keep it thin: smart endpoints, dumb pipes); and its timeouts, retries and limits need the same care as any caller's (resilience patterns). Do not mistake it for a load balancer, which spreads traffic across instances of one service (load balancing); a gateway usually sits on top of load balancers.

6. Finding services: discovery and load balancing

Instances come and go (deploys, autoscaling, crashes), so their addresses change. Callers need a way to find healthy ones:

Whichever you use, add health checks (readiness and liveness are different questions), connection reuse, and timeouts and retry budgets; a stale list sends traffic to dead instances.

7. The service mesh

When there are many services, every one of them needs the same plumbing: mutual TLS, retries, timeouts, circuit breaking, load balancing, metrics and traces. Writing that into a library for each language is expensive and hard to keep consistent. A service mesh moves it out of the application into the infrastructure: a small proxy (a sidecar, often Envoy) runs next to every service instance and handles all of its inbound and outbound traffic, and a control plane configures them.

Diagram: Sidecar proxies and a control plane

What it gives you, without changing application code:

What it costs: extra latency and CPU on every hop (two proxies per call, typically a millisecond or less but not free), memory for each sidecar, a complex control plane to run and upgrade, and a new class of failures and misconfigurations that are hard to debug. Newer designs run the proxy per node or without sidecars to reduce this overhead. A mesh also does not fix design problems: retries configured in the mesh and in the application multiply (see the resilience guide), and it cannot make a chatty design fast.

Do you need one? Often not at first. With a handful of services, a good client library, a gateway and Kubernetes network policies are enough. Consider a mesh when you have dozens of services in several languages, a hard requirement for mutual TLS everywhere, and a platform team to operate it.

8. Data, queries and consistency

With a database per service you lose joins and cross-service transactions:

Moving to a database per service, step by step

Splitting the data is usually harder than splitting the code, because the shared database is where the coupling hides: joins, foreign keys, shared tables and transactions that span what will become several services. A migration that works on a live system goes in small, reversible steps:

  1. Map ownership. For each table decide which capability owns it (who may write it). Tables written by several parts of the monolith are the first problem to resolve, not the last.
  2. Separate inside the database first. Give each future service its own schema and its own database user, and revoke access to other schemas. Violations now fail loudly, while everything is still one database and one deployment.
  3. Replace each cross-boundary join and foreign key. Options: an API call, a batched API call, a local read model filled from events, or accepting that the data is duplicated and the foreign key becomes a reconciliation job.
  4. Synchronise during the move. Use change data capture or the outbox pattern (event sourcing and CQRS); avoid dual writes from application code, which drift apart when one write fails.
  5. Cut reads over, then writes, behind a flag, comparing old and new results (shadow reads) before the switch. Keep the old data for a rollback period.
  6. Move the data physically (a separate server or managed instance) last. Many services can stay in one database server with separate schemas for a long time.

What a join becomes. I measured a page of 50 orders with the customer's name, with 100,000 orders and 10,000 customers. In one database it is a single SQL join. After the split, the customer data sits behind an HTTP service (real HTTP on loopback, so there is no network latency in these numbers):

How the page is builtMedian99th percentile
one SQL join in a shared database0.040 ms0.077 ms
50 HTTP calls, one per order11.5 ms16.7 ms
one batched HTTP call (ids=1,2,3,...)0.45 ms1.2 ms
a local read model (the order service keeps a copy of the names)0.036 ms0.062 ms

The naive version is about 290 times slower and, with a real network between the services, it would add 50 round trips on top; batching brings it back to roughly 11 times the join; a local read model costs the same as the join, in exchange for eventual consistency (a renamed customer shows the old name until the event arrives) and a copy of the data to keep correct. The same trade appears everywhere:

What you give upWhat replaces itThe price
joins across the boundarybatched calls, read models, a reporting storelatency, stale data, duplicated data
foreign keys across the boundaryvalidation at write time, events, reconciliation jobsorphans are possible and must be detected
one ACID transaction across servicessagas and the outboxcompensations, intermediate states visible to users
one place to query for reportsevents into a warehouse or search indexa pipeline to build and watch
one database to back up and patchN databasesoperational work for each

What you gain: each service can choose and scale its storage, a schema change in one service cannot break another, and an outage or a slow query in one database is contained. Do not pay the price without a reason: if two parts of the system must always change together and be consistent, they probably belong in the same service.

9. Operating many services

Migrating from a monolith: the strangler fig

Diagram: The strangler fig: route by route, behind a facade

Do not rewrite everything at once. Put a facade in front, extract one capability at a time (starting with one that is loosely coupled and valuable), route its traffic to the new service, migrate its data, and retire that part of the monolith. At every step the system works and you can stop.

9.1 A complete failure, end to end: one slow dependency takes down the rest

The earlier sections measured single effects (availability, latency, retries). Real outages combine them. I simulated a small system in simulated time (script in the appendix; a model, not a real deployment): clients call Orders; Orders has 40 workers, calls Inventory (fast) and Payments (it can finish 20 charges at a time, 50 ms each, so about 400 per second); browsing is served by Orders too and never calls Payments. Load: 100 checkouts and 200 browses per second. Clients give up after 3 seconds. For 30 seconds Payments becomes slow (800 ms per charge, as with a degraded database), so it can now finish only about 25 charges per second while 100 are asked for.

Diagram: The simulated system: browsing and checkout share Orders' workers

What happens with the naive settings (1 s timeout, 3 retries with no delay, one shared pool, no circuit breaker, no idempotency key):

  1. Payments' queue grows and calls start to exceed the 1 s timeout. The caller gives up, but Payments still does the work: 97% of the charges it finished during the fault had nobody waiting.
  2. Each checkout now holds an Orders worker for up to 4 seconds (4 attempts of 1 s). The 40 workers are all busy within a moment.
  3. Browsing, which never touches Payments, fails too: it queues behind the blocked checkouts and only 2% of browses succeeded during the fault. This is the cascade: the failure of one dependency became the failure of everything that shares a pool with it.
  4. Partial success: charges finish after the caller timed out, so customers are charged and told it failed (about 3,700 of them), and retries of an unfinished charge charge again (about 320 duplicates).
  5. After Payments recovers, the backlog and the queued retries keep checkout broken for about 8 more seconds.

Fixing one thing at a time (average of three runs; counts are for the whole 70-second run):

ConfigurationCheckout success during the faultBrowse success during the faultPayments offered (can do 25/s)Duplicate chargesCharged but client saw a failureCheckout healthy again after
A naive0.6%2.0%40/s3203,7338 s
B only separate pools (30 for checkout, 10 for browsing)1.8%100%30/s2372401 s
C only a 250 ms timeout and a retry budget (at most one retry per ten requests)0.0%100%110/s4033,9339 s
D all of that, plus a circuit breaker, deadline propagation and an idempotency key0.0%100%1/s0352 s
E D, and when Payments fails, accept the order as pending and charge it later100% accepted100%1/s000 s

What each row teaches:

The general lessons are the ones of the resilience patterns guide, now seen together: every call needs a timeout shorter than its caller's, retries need a budget and backoff, give each dependency its own pool, stop calling a dependency that is failing, make operations idempotent so a retry cannot repeat them, and decide in advance what the system does instead. Model limits: the real Payments service would also shed load, clients retry on their own, and the pool sizes, timeouts and fault length here are my choices; change them in the script and the table changes, but the order of the effects (shared pool first, retries second, duplicates third) is what to look for.

10. Common mistakes

11. Things to mention in an interview

Quick recap

Appendix: the experiment code

The first script (Python 3, standard library only, fixed seeds) computes the availability table and simulates the latency of chains and fan-outs from a lognormal distribution with a median of 20 ms; the distribution is an assumption chosen to have a realistic long tail, not a measurement of any real service, and it treats calls as independent, which real calls often are not. The second script (Node.js 22) measures a function call, a JSON round trip and an HTTP request over loopback on a small two-core virtual machine, so the nanoseconds will differ on your hardware; the ratio is what matters. It starts its own HTTP server and does not need the internet.

Availability and tail latency arithmetic

import random, math

# 1. availability of a request that needs n services in series, each 99.9% available
print("n services in series, each 99.9% available:")
for n in (1, 3, 5, 10, 20, 50):
    a = 0.999 ** n
    print(f"  n={n:>2}: {a*100:.2f}% available = {(1-a)*365*24*60:,.0f} minutes of downtime per year")
# with a fallback (degraded answer) on a share of the dependencies
print("10 services, 99.9% each, when k of them are optional (the page still works without them):")
for k in (0, 3, 6, 9):
    a = 0.999 ** (10 - k); print(f"  {k} optional: {a*100:.3f}%")
# replicas: an instance 99% available, r replicas behind a load balancer
for r in (1, 2, 3): print(f"  one service at 99%: {r} independent replicas -> {(1-(0.01)**r)*100:.4f}%")

# 2. tail latency: a request that calls n services in parallel and waits for all of them
rnd = random.Random(3)
def call():  # lognormal: median 20 ms, a long tail (1% of calls above ~ 85 ms)
    return math.exp(rnd.gauss(math.log(20), 0.55))
def pct(v, p): v = sorted(v); return v[int(len(v) * p) - 1]
single = [call() for _ in range(100000)]
P99 = pct(single, .99)
print(f"one call: median {pct(single, .5):.0f} ms, 99th percentile {pct(single, .99):.0f} ms")
for n in (1, 5, 10, 50):
    fan = [max(call() for _ in range(n)) for _ in range(20000)]
    slow = sum(x > P99 for x in fan) / len(fan)
    print(f"  wait for the slowest of {n:>2} parallel calls: median {pct(fan, .5):.0f} ms, 99th {pct(fan, .99):.0f} ms; {slow*100:.1f}% of requests hit at least one 'p99-slow' call")

# 3. a synchronous chain against parallel and asynchronous designs
chain = lambda n: [sum(call() for _ in range(n)) for _ in range(20000)]
for n in (1, 3, 5, 8):
    c = chain(n); print(f"  chain of {n} synchronous calls: median {pct(c, .5):.0f} ms, 99th {pct(c, .99):.0f} ms")

The cost of a call: function, JSON and HTTP (Node.js)

import http from 'node:http';
const handler = (req, res) => { let b = ''; req.on('data', (c) => (b += c)); req.on('end', () => { const o = JSON.parse(b); res.setHeader('content-type', 'application/json'); res.end(JSON.stringify({ total: o.items.reduce((s, i) => s + i.price * i.qty, 0) })); }); };
const server = http.createServer(handler).listen(0);
await new Promise((r) => setTimeout(r, 100));
const order = { id: 7, items: Array.from({ length: 10 }, (_, i) => ({ sku: 'sku-' + i, price: 10 + i, qty: 2 })) };
const total = (o) => o.items.reduce((s, i) => s + i.price * i.qty, 0);

// 1. a function call
let s = 0; const N1 = 2_000_000; let t = performance.now();
for (let i = 0; i < N1; i++) s += total(order);
const fn = ((performance.now() - t) / N1) * 1e6;
// 2. the same call with the JSON encode and decode a network call needs, but no network
t = performance.now(); const N2 = 200_000;
for (let i = 0; i < N2; i++) s += total(JSON.parse(JSON.stringify(order)));
const ser = ((performance.now() - t) / N2) * 1e6;
// 3. a real HTTP request over loopback, keep-alive, one at a time
const agent = new http.Agent({ keepAlive: true, maxSockets: 1 });
const call = () => new Promise((ok) => { const r = http.request({ port: server.address().port, method: 'POST', agent, headers: { 'content-type': 'application/json' } }, (res) => { let b = ''; res.on('data', (c) => (b += c)); res.on('end', () => ok(JSON.parse(b))); }); r.end(JSON.stringify(order)); });
for (let i = 0; i < 2000; i++) await call();
const lat = []; for (let i = 0; i < 10000; i++) { const t0 = performance.now(); await call(); lat.push(performance.now() - t0); }
lat.sort((a, b) => a - b);
console.log(`in-process function call: ${fn.toFixed(0)} ns`);
console.log(`function call plus JSON encode and decode of the order: ${ser.toFixed(0)} ns (${(ser / fn).toFixed(0)}x)`);
console.log(`HTTP call over loopback, keep-alive, one at a time: median ${(lat[5000] * 1000).toFixed(0)} us, 99th percentile ${(lat[9900] * 1000).toFixed(0)} us (${((lat[5000] * 1e6) / fn).toFixed(0)}x the function call)`);
// 4. concurrency: requests per second with a pool of connections
const t2 = performance.now(); const agent2 = new http.Agent({ keepAlive: true, maxSockets: 32 });
const call2 = () => new Promise((ok) => { const r = http.request({ port: server.address().port, method: 'POST', agent: agent2, headers: { 'content-type': 'application/json' } }, (res) => { res.resume(); res.on('end', ok); }); r.end(JSON.stringify(order)); });
let done = 0; await Promise.all(Array.from({ length: 32 }, async () => { for (let i = 0; i < 1000; i++) { await call2(); done++; } }));
console.log(`32 concurrent connections: ${(done / ((performance.now() - t2) / 1000)).toFixed(0)} requests per second, client and server on the same two cores`);
if (s === 42) console.log(); server.close(); agent.destroy(); agent2.destroy(); process.exit(0);

A cascading failure: timeouts, retries, pools, a circuit breaker and idempotency

Python 3, standard library only; it simulates time, so a run takes a few seconds and is repeatable from its seeds.

"""A cascading failure across three services, in simulated time (milliseconds).
Clients -> Orders (a worker pool) -> Inventory (fast) and Payments (a service with limited capacity).
Payments becomes slow for 30 s. Which settings of timeouts, retries, pools, a circuit breaker and idempotency keys decide what happens?"""
import heapq, itertools, random, collections, statistics

CLIENT_TIMEOUT = 3000          # the browser or the gateway gives up after 3 s
FAULT = (15_000, 45_000)       # Payments is slow in this window; the run lasts 70 s
UNTIL = 70_000
CHECKOUT_RPS, BROWSE_RPS = 100, 200
PAY_CAPACITY, PAY_MS, PAY_SLOW_FACTOR = 20, 50, 16   # 20 concurrent charges of 50 ms, or 800 ms when slow

class Sim:
    def __init__(s, seed): s.t = 0.0; s.q = []; s.n = itertools.count(); s.rnd = random.Random(seed)
    def at(s, dt, fn): heapq.heappush(s.q, (s.t + dt, next(s.n), fn))
    def run(s, until):
        while s.q and s.q[0][0] <= until:
            s.t, _, fn = heapq.heappop(s.q); fn()

class Pool:
    """A fixed number of workers and a FIFO queue (optionally bounded: extra requests are rejected at once)."""
    def __init__(s, size, max_queue=None): s.size, s.busy, s.wait, s.max_queue = size, 0, collections.deque(), max_queue
    def submit(s, job, reject):
        if s.busy < s.size: s.busy += 1; job(s.release)
        elif s.max_queue is not None and len(s.wait) >= s.max_queue: reject()
        else: s.wait.append(job)
    def release(s):
        s.busy -= 1
        if s.wait and s.busy < s.size: s.busy += 1; s.wait.popleft()(s.release)

class Breaker:
    """Opens when at least half of the last 20 outcomes failed; after 5 s lets one probe through."""
    def __init__(s): s.hist = collections.deque(maxlen=20); s.open_until = 0; s.probing = False
    def allow(s, t):
        if t < s.open_until: return False
        if s.open_until and not s.probing: s.probing = True; return True    # half-open: a single probe
        return not s.open_until
    def record(s, ok, t):
        if s.probing: s.probing = False; s.hist.clear(); s.open_until = 0 if ok else t + 5000; return
        s.hist.append(ok)
        if len(s.hist) >= 10 and s.hist.count(False) >= len(s.hist) / 2: s.open_until = t + 5000; s.hist.clear()

CONFIGS = {
    "A naive: 1 s timeout, 3 retries, one shared pool": dict(timeout=1000, attempts=4, backoff=0, budget=False, breaker=False, bulkhead=False, idem=False, deadline=False, fallback=False),
    "B only separate pools (bulkhead)": dict(timeout=1000, attempts=4, backoff=0, budget=False, breaker=False, bulkhead=True, idem=False, deadline=False, fallback=False),
    "C only short timeout and a retry budget": dict(timeout=250, attempts=2, backoff=100, budget=True, breaker=False, bulkhead=False, idem=False, deadline=False, fallback=False),
    "D all together, plus breaker and idempotency": dict(timeout=250, attempts=2, backoff=100, budget=True, breaker=True, bulkhead=True, idem=True, deadline=True, fallback=False),
    "E all of D, and when Payments fails accept the order as pending": dict(timeout=250, attempts=2, backoff=100, budget=True, breaker=True, bulkhead=True, idem=True, deadline=True, fallback=True),
}

def run(cfg, seed):
    sim = Sim(seed); rnd = sim.rnd
    pay = Pool(PAY_CAPACITY)
    if cfg["bulkhead"]: pools = {"checkout": Pool(30, 30), "browse": Pool(10, 30)}   # the same 40 workers, split
    else: pools = {"checkout": (shared := Pool(40)), "browse": shared}
    breaker = Breaker(); tokens = [10.0]
    charges = collections.Counter(); attempts = []; reqs = []; waste = [0, 0]      # waste: [charges nobody waited for, charges done], during the fault

    def pay_call(req_id, reply):
        def job(release):
            slow = FAULT[0] <= sim.t < FAULT[1]
            svc = PAY_MS * (PAY_SLOW_FACTOR if slow else 1) * (0.8 + 0.4 * rnd.random())
            def done():
                if not (cfg["idem"] and charges[req_id]): charges[req_id] += 1     # an idempotency key makes the repeat a no-op
                release(); used = reply()
                if FAULT[0] <= sim.t < FAULT[1]: waste[1] += 1; waste[0] += (not used)
            sim.at(svc, done)
        pay.submit(job, lambda: None)

    def finish(r, ok, why=""):
        if "done" in r: return
        r["done"] = True; r["ok"] = ok and sim.t - r["t0"] <= CLIENT_TIMEOUT; r["why"] = why; r["lat"] = sim.t - r["t0"]

    def fail(r, why):
        if cfg["fallback"]: finish(r, True, "pending")           # accept the order now, charge it later from a queue
        else: finish(r, False, why)

    def checkout(r):
        def job(release):
            if cfg["deadline"] and sim.t - r["t0"] > CLIENT_TIMEOUT: release(); finish(r, False, "expired"); return
            def attempt(n):
                if cfg["breaker"] and not breaker.allow(sim.t): release(); fail(r, "breaker"); return
                st = {"over": False}; attempts.append(sim.t)
                def reply():
                    if st["over"]: return False                  # the caller gave up: this work was wasted
                    st["over"] = True; breaker.record(True, sim.t); release(); finish(r, True); return True
                def timeout():
                    if st["over"]: return
                    st["over"] = True; breaker.record(False, sim.t)
                    if n < cfg["attempts"] and (not cfg["budget"] or tokens[0] >= 1):
                        if cfg["budget"]: tokens[0] -= 1
                        sim.at(cfg["backoff"] * 2 ** (n - 1) * (0.5 + rnd.random()), lambda: attempt(n + 1))
                    else: release(); fail(r, "timeout")
                if n == 1 and cfg["budget"]: tokens[0] = min(10.0, tokens[0] + 0.1)
                pay_call(r["id"], reply); sim.at(cfg["timeout"], timeout)
            sim.at(10, lambda: attempt(1))                      # the inventory check
        pools["checkout"].submit(job, lambda: finish(r, False, "shed"))

    def browse(r):
        def job(release):
            if cfg["deadline"] and sim.t - r["t0"] > CLIENT_TIMEOUT: release(); finish(r, False, "expired"); return
            def done(): release(); finish(r, True)
            sim.at(5, done)
        pools["browse"].submit(job, lambda: finish(r, False, "shed"))

    def arrivals(kind, rps, fn):
        t = 0.0
        while True:
            t += rnd.expovariate(rps / 1000)
            if t > UNTIL - CLIENT_TIMEOUT: break                # do not count requests whose outcome would be cut off by the end of the run
            r = {"id": len(reqs), "kind": kind, "t0": t}; reqs.append(r); sim.at(t, lambda r=r: fn(r))
    arrivals("checkout", CHECKOUT_RPS, checkout); arrivals("browse", BROWSE_RPS, browse)
    sim.run(UNTIL)
    return reqs, charges, attempts, waste

def phase(t0):
    return "before" if t0 < FAULT[0] else "during" if t0 < FAULT[1] else "after"

def summarise(cfg, seeds=(1, 2, 3)):
    acc = collections.defaultdict(list)
    for seed in seeds:
        reqs, charges, attempts, waste = run(cfg, seed)
        for ph in ("before", "during", "after"):
            for kind in ("checkout", "browse"):
                rs = [r for r in reqs if r["kind"] == kind and phase(r["t0"]) == ph]
                acc[(ph, kind, "ok")].append(100 * sum(bool(r.get("ok")) for r in rs) / len(rs))
                lat = sorted(r["lat"] for r in rs if r.get("ok"))
                acc[(ph, kind, "p99")].append(lat[int(len(lat) * .99)] if lat else float("nan"))
        co = [r for r in reqs if r["kind"] == "checkout" and FAULT[0] <= r["t0"] < FAULT[1]]
        att = [a for a in attempts if FAULT[0] <= a < FAULT[1]]
        acc["amp"].append(len(att) / len(co)); acc["rate"].append(len(att) / ((FAULT[1] - FAULT[0]) / 1000)); acc["waste"].append(100 * waste[0] / max(1, waste[1]))
        acc["pend"].append(100 * sum(r.get("why") == "pending" for r in co) / len(co))
        acc["dup"].append(sum(1 for r in reqs if r["kind"] == "checkout" and charges[r["id"]] > 1))
        acc["lost"].append(sum(1 for r in reqs if r["kind"] == "checkout" and charges[r["id"]] >= 1 and not r.get("ok")))
        # recovery: seconds after the fault ends until a 5-second window of checkouts is at least 95% successful
        rec = None
        for s in range(0, 25):
            w = [r for r in reqs if r["kind"] == "checkout" and FAULT[1] + s * 1000 <= r["t0"] < FAULT[1] + s * 1000 + 5000]
            if w and sum(bool(r.get("ok")) for r in w) / len(w) >= .95: rec = s; break
        acc["rec"].append(rec if rec is not None else 25)
    m = lambda k: statistics.mean(acc[k])
    return m, acc

if __name__ == "__main__":
    print(f"Payments is slow ({PAY_MS} ms -> {PAY_MS*PAY_SLOW_FACTOR} ms per charge, capacity {PAY_CAPACITY} at a time) from {FAULT[0]//1000} s to {FAULT[1]//1000} s; {CHECKOUT_RPS} checkouts/s and {BROWSE_RPS} browses/s (browsing never calls Payments).")
    print("Success = the client got a good answer within 3 s. Average of 3 seeds.\n")
    for name, cfg in CONFIGS.items():
        m, _ = summarise(cfg)
        print(name)
        print(f"   checkout success %  before {m(('before','checkout','ok')):5.1f}  during {m(('during','checkout','ok')):5.1f}  after {m(('after','checkout','ok')):5.1f}")
        print(f"   browse   success %  before {m(('before','browse','ok')):5.1f}  during {m(('during','browse','ok')):5.1f}  after {m(('after','browse','ok')):5.1f}   (unrelated to Payments)")
        print(f"   Payments during the fault: {m('rate'):.0f} attempts/s offered (it can finish 25/s), {m('waste'):.0f}% of its finished charges had no caller waiting; {m('pend'):.0f}% of checkouts answered 'accepted, pending'")
        print(f"   duplicate charges: {m('dup'):.0f} | charged but the client saw a failure: {m('lost'):.0f} | checkout back to >=95% after the fault ends: {m('rec'):.0f} s")

What a join becomes after the split: SQL join, per-row HTTP calls, a batched call and a local read model

Python 3 with sqlite3 and http.server, on loopback. Absolute timings depend on the machine; the ratios are the point.

"""What a join becomes after 'database per service': listing 50 orders with the customer's name.
Orders and customers live in separate SQLite databases; the customer service answers over real HTTP on loopback (stdlib only)."""
import http.client, http.server, json, random, socket, sqlite3, statistics, threading, time

N_CUSTOMERS, N_ORDERS, PAGE = 10_000, 100_000, 50
rnd = random.Random(4)

# one shared database (before the split)
shared = sqlite3.connect(":memory:", check_same_thread=False)
shared.executescript("create table customers(id integer primary key, name text); create table orders(id integer primary key, customer_id int, total int);")
shared.executemany("insert into customers values (?,?)", ((i, f"customer {i}") for i in range(N_CUSTOMERS)))
shared.executemany("insert into orders values (?,?,?)", ((i, rnd.randrange(N_CUSTOMERS), rnd.randrange(1, 500)) for i in range(N_ORDERS)))
shared.execute("create index o_c on orders(customer_id)")

# after the split: the customer service owns `customers`, the order service owns `orders`
cust_db = sqlite3.connect(":memory:", check_same_thread=False)
cust_db.execute("create table customers(id integer primary key, name text)")
cust_db.executemany("insert into customers values (?,?)", shared.execute("select id, name from customers"))
ord_db = sqlite3.connect(":memory:", check_same_thread=False)
ord_db.execute("create table orders(id integer primary key, customer_id int, total int)")
ord_db.executemany("insert into orders values (?,?,?)", shared.execute("select id, customer_id, total from orders"))

class H(http.server.BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"
    def setup(self): super().setup(); self.connection.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)   # avoid the 40 ms delayed-ACK stall
    def log_message(self, *a): pass
    def do_GET(self):
        if self.path.startswith("/customers/"):
            row = cust_db.execute("select id, name from customers where id=?", (int(self.path.split("/")[2]),)).fetchone()
            out = {"id": row[0], "name": row[1]}
        else:                                                    # /customers?ids=1,2,3
            ids = [int(x) for x in self.path.split("=")[1].split(",")]
            out = [{"id": i, "name": n} for i, n in cust_db.execute(f"select id, name from customers where id in ({','.join('?'*len(ids))})", ids)]
        b = json.dumps(out).encode()
        self.send_response(200); self.send_header("Content-Length", str(len(b))); self.end_headers(); self.wfile.write(b)
srv = http.server.ThreadingHTTPServer(("127.0.0.1", 8120), H); threading.Thread(target=srv.serve_forever, daemon=True).start()
conn = http.client.HTTPConnection("127.0.0.1", 8120); conn.connect(); conn.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
def get(path): conn.request("GET", path); return json.loads(conn.getresponse().read())

def a_join():
    return shared.execute("select o.id, o.total, c.name from orders o join customers c on c.id=o.customer_id order by o.id desc limit ?", (PAGE,)).fetchall()
def b_one_call_per_order():
    rows = ord_db.execute("select id, customer_id, total from orders order by id desc limit ?", (PAGE,)).fetchall()
    return [(i, t, get(f"/customers/{c}")["name"]) for i, c, t in rows]
def c_one_batched_call():
    rows = ord_db.execute("select id, customer_id, total from orders order by id desc limit ?", (PAGE,)).fetchall()
    names = {c["id"]: c["name"] for c in get("/customers?ids=" + ",".join(str(c) for _, c, _ in rows))}
    return [(i, t, names[c]) for i, c, t in rows]
replica = {}
def d_local_read_model():
    """The order service keeps its own copy of the customer names, filled from customer events."""
    rows = ord_db.execute("select id, customer_id, total from orders order by id desc limit ?", (PAGE,)).fetchall()
    return [(i, t, replica[c]) for i, c, t in rows]
replica.update(dict(cust_db.execute("select id, name from customers")))

def bench(fn, reps=200):
    fn(); xs = []
    for _ in range(reps):
        t = time.perf_counter(); fn(); xs.append((time.perf_counter() - t) * 1000)
    xs.sort(); return statistics.median(xs), xs[int(len(xs) * .99)]
assert sorted(a_join()) == sorted(b_one_call_per_order()) == sorted(c_one_batched_call()) == sorted(d_local_read_model())
print(f"A page of {PAGE} orders with the customer name; {N_ORDERS:,} orders, {N_CUSTOMERS:,} customers. Milliseconds per page (median / 99th percentile, 200 runs):")
for label, fn in (("one SQL join in a shared database", a_join), (f"separate databases: {PAGE} HTTP calls, one per order", b_one_call_per_order),
                  ("separate databases: one batched HTTP call", c_one_batched_call), ("separate databases: local read model (copy of the names)", d_local_read_model)):
    m, p = bench(fn); print(f"  {label:<58} {m:7.3f} / {p:7.3f} ms")
srv.shutdown()

Sources

Practise this topic: System Design System Design › Reliability and scale

More guides