CS Fundamentals Trainer · Guides
Microservices, API gateways and service meshes
What splitting into services really costs in latency and availability, how to draw boundaries, synchronous against asynchronous communication, the API gateway and BFF, service discovery and the service mesh, and the strangler fig migration.
40 min read
"Should we use microservices?" is among the most common architecture questions, and the honest answer starts with what the question really asks: should we turn function calls into network calls so that teams can build, deploy and scale parts of the system independently? That trade can be right, and it has a price that is easy to underestimate: every hop adds latency and a new way to fail, and consistency across services becomes your problem. This guide gives you the numbers for that price, then the patterns that make it manageable: service boundaries, communication styles, the API gateway, service discovery, and the service mesh.
I measured the call overhead in Node.js over loopback and simulated the availability and latency arithmetic in Python (code in the appendix). It builds on resilience patterns, observability and distributed transactions, which supply the tools you need once you have a network between your components.
1. What problem do microservices solve?
Mainly an organisational one. A single deployable (a monolith) is simple to build, test, run and debug, and fast: calls are function calls and one database transaction can cover everything. It starts to hurt when many teams must change the same code base: builds get slow, a release means coordinating everyone, one team's bug takes down everything, and one hot component forces you to scale the whole thing.
Splitting the system into independently deployable services buys:
- Independent deployment and ownership: a team ships its service without asking the others (if the boundaries are right).
- Independent scaling: scale the image-resizing service, not the whole shop.
- Fault and technology isolation: a crash or a memory leak stays in one service; a service can use a different language or database where it matters.
It costs:
- Network latency and failure on every call, as measured below.
- No shared transaction: a business operation spanning services needs sagas or accepted inconsistency.
- Operational weight: many pipelines, deployments, dashboards, alerts, versions and on-call rotations.
- Harder debugging and testing: a request is a trace across processes, and many bugs appear only in the interaction.
Conway's law says that systems mirror the communication structure of the organisation that builds them. Treat it as a design tool: choose service boundaries to match team boundaries, not the other way round. A team of eight with one product rarely needs thirty services. A common, sound path is a modular monolith first (one deployable with strict module boundaries and each module owning its tables) and extraction of a service only when a concrete need appears (a different scaling profile, a different release cadence, a team boundary).
2. The price of a network hop
I measured the same small operation, totalling an order of ten items, three ways in Node.js on a two-core virtual machine:
| How the call is made | Time per call | Compared with a function call |
|---|---|---|
| In-process function call | 19 ns | 1x |
| The same, with a JSON encode and decode of the order (no network) | 4.7 µs | 251x |
| A real HTTP request over loopback with keep-alive, one at a time | 89 µs median, 562 µs 99th percentile | about 4,700x |
Even on loopback, with no wire, no TLS and no routing, a service call is thousands of times slower than a function call, and most of the cost is serialisation, the protocol and the context switches, not the distance. A real network adds more: roughly 0.1 to 1 ms inside a data centre, 10 to 100 ms across a continent or an ocean (typical figures, not measured here), plus TLS, load balancers and queues. And a single pair of cores served about 13,000 such requests per second in total, client and server together, which shows that calls also consume CPU on both sides.
Availability multiplies
If a request needs n services in series and each is 99.9% available, the request is available only 99.9% to the power n:
| Services in series | Availability | Downtime per year |
|---|---|---|
| 1 | 99.90% | about 526 minutes |
| 3 | 99.70% | 1,575 minutes |
| 5 | 99.50% | 2,623 minutes |
| 10 | 99.00% | 5,232 minutes (3.6 days) |
| 20 | 98.02% | 10,413 minutes |
| 50 | 95.12% | 25,646 minutes (18 days) |
This assumes failures are independent, so it is an optimistic estimate. The fixes are the ones from the resilience guide: make dependencies optional (the page works without recommendations: with 3 of 10 optional the availability goes from 99.004% to 99.302%, with 6 optional to 99.601%), use redundancy (two independent replicas of a 99% service give 99.99%, three give 99.9999%) and avoid long synchronous chains (below).
Latency adds, and the tail multiplies
I modelled each call with a median of 20 ms and a long tail (99th percentile 72 ms, a lognormal distribution), and measured the end-to-end time:
| Pattern | Median | 99th percentile |
|---|---|---|
| 1 call | 20 ms | 72 ms |
| a chain of 3 synchronous calls | 66 ms | 145 ms |
| a chain of 5 | 112 ms | 206 ms |
| a chain of 8 | 181 ms | 296 ms |
| waiting for the slowest of 5 parallel calls | 37 ms | 96 ms |
| the slowest of 10 parallel calls | 45 ms | 108 ms |
| the slowest of 50 parallel calls | 67 ms | 139 ms |
Chains add: eight hops cost 9 times a single call. Fan-out is gentler on the median but hurts the tail: when a request waits for 50 parallel calls, 39.6% of requests hit at least one call that is slower than the one-in-a-hundred worst case of a single call, and the median is more than three times a single call. This is the "tail at scale" effect (Dean and Barroso): the more services a request touches, the more often it meets somebody's bad moment. Defences: fewer hops, caching, timeouts with fallbacks, and hedged requests (sending a duplicate to another replica when the first is slow) for idempotent reads.
3. Drawing the boundaries
Good boundaries make most changes local to one service. Bad ones create a distributed monolith: the cost of microservices with none of the benefits.
Guidance for finding boundaries:
- Start from the business capabilities and bounded contexts of domain-driven design: "ordering", "catalogue", "payments", "shipping", each with its own language and rules, and not from technical layers ("the database service", "the validation service").
- A service owns its data. No other service reads or writes its tables; they use its API or its events. A shared database is a hidden coupling: any schema change becomes a cross-team negotiation.
- Optimise for high cohesion and low coupling: things that change together live together.
- Size by team ownership and cognitive load, not by lines of code. "Micro" is not a target.
Warning signs of a distributed monolith:
- Releases must be coordinated across services (they must go out in a certain order or together).
- Services share a database or a library with business logic.
- A single user action needs many synchronous calls between services, so that one slow service slows all.
- Changing a field requires edits in half a dozen repositories.
- Everything is down when one "small" service is down.
Who owns a service?
Service boundaries are also people boundaries. Conway's law says that a system's structure tends to copy the communication structure of the organisation that builds it; if four teams must agree before one change ships, the architecture will have four interlocking parts whatever the diagram says. Microservices work when the boundaries are drawn deliberately along both lines (the "inverse Conway manoeuvre": shape the teams to get the architecture you want). In practice:
- One team owns each service end to end: code, data, pipeline, on-call and the API contract. "You build it, you run it" matters because the team that is woken at night is the one that fixes the cause. A service with no clear owner decays; keep a service catalogue with the owner, the on-call route, the runbook, the SLOs and the dependencies of each one.
- A team owns a few services, not dozens. There is no measured threshold, but a platform with far more services than people is a warning sign: every service carries a fixed cost (pipeline, dashboards, alerts, patching, upgrades), and the shortage shows up as services that nobody understands.
- Align teams with business capabilities, not layers. A "front-end team", a "back-end team" and a "database team" force every feature through three queues. A stream-aligned team that owns a capability (checkout, search) can ship alone. Specialist platform teams then provide the paved road (deployment, observability, a service template) and enabling teams spread skills; this is the vocabulary of the Team Topologies book.
- Own the contract, not just the code. Consumers depend on your API, events and error behaviour; changes go through versioning and a deprecation period, and the owner knows who calls them (a gateway and tracing make that visible).
- Be careful with shared code. A shared library with business logic couples release schedules and recreates the monolith's coordination. Share contracts and infrastructure helpers, not domain logic.
- Small teams and few services? Stay modular. If one team of eight owns everything, a well-structured monolith gives the independence that matters (clear module boundaries) without the network costs measured above. Split along a boundary when a team boundary appears.
4. How services talk
Diagram: Synchronous calls and asynchronous events
Synchronous (REST, gRPC): simple to reason about, immediate answer, but the caller depends on the callee being up and fast right now, so availability and latency multiply as shown. Use for queries that need an answer to proceed. gRPC (HTTP/2 with Protocol Buffers) is more compact and fast than JSON over HTTP/1.1, supports deadlines and streaming, and gives a typed contract; REST is more universal for external APIs (see designing a REST API).
Asynchronous (events or commands through a broker): the producer does not wait, services are decoupled in time, spikes are absorbed by the queue, and new consumers can be added without touching the producer. The costs are eventual consistency, harder tracing, duplicate and out-of-order delivery (see message queues and delivery guarantees), and a broker to operate.
A pragmatic split: synchronous for queries and for the one call that the user is waiting on, asynchronous for everything that can happen after the response. An order service can answer "order placed" after saving it and publishing an event; the email, analytics, loyalty points and shipping reactions follow on their own.
For a business process that spans services, use a saga: either choreography (each service reacts to events and emits the next one: decoupled, but the overall flow is hard to see) or orchestration (a coordinator tells each service what to do and handles compensation: easier to follow and change, with the coordinator as a new component). The previous guide on distributed transactions covers both and the limits of idempotency keys.
Contracts and versions
Services evolve separately, so the interface is a contract. Change it compatibly: add optional fields, do not remove or reinterpret existing ones, and use the expand and contract pattern for breaking changes (add the new form, migrate every consumer, remove the old). Consumer-driven contract tests let each consumer publish the requests and responses it relies on, and the provider's pipeline fails when a change would break them. Use a schema (OpenAPI, protobuf, Avro) and a compatibility check in CI.
5. The API gateway
Clients (a browser, a phone app, a partner) should not know your internal services. An API gateway is a single entry point that routes external requests to the services behind it and handles the cross-cutting concerns in one place:
Diagram: An API gateway in front of the services
Typical jobs:
- Routing by path or host to the right service, and versioning of public APIs.
- TLS termination and authentication: verify the token once at the edge and pass the identity on (see authentication and sessions). Internal services must still check authorisation: do not trust "it came from the gateway" alone.
- Rate limiting and quotas per client (designing a rate limiter), and protection against abusive traffic.
- Response caching and compression, request and response transformation.
- Aggregation: one call from a mobile app that the gateway fans out to several services, saving round trips on slow networks.
- Observability: a trace id is created here and propagated, and the access log lives in one place.
A variation is the Backend for Frontend (BFF): a separate gateway per client type (web, mobile, partner) owned by that client's team, shaped to its needs, instead of one general-purpose gateway that serves everybody badly.
Pitfalls: the gateway is on the path of every request, so it must be highly available and scaled horizontally; it becomes a bottleneck and a deploy dependency if business logic accumulates in it (keep it thin: smart endpoints, dumb pipes); and its timeouts, retries and limits need the same care as any caller's (resilience patterns). Do not mistake it for a load balancer, which spreads traffic across instances of one service (load balancing); a gateway usually sits on top of load balancers.
6. Finding services: discovery and load balancing
Instances come and go (deploys, autoscaling, crashes), so their addresses change. Callers need a way to find healthy ones:
- DNS-based: a name resolves to the instances (or to a virtual IP). Simple, but DNS caching makes changes slow to appear (DNS and CDNs).
- A registry: instances register and send heartbeats; callers query it (Consul, etcd, ZooKeeper, which use consensus to stay consistent). Kubernetes does this with Services and endpoints.
- Client-side load balancing: the caller gets the list and picks an instance itself (round robin, least loaded, or power of two choices), avoiding an extra hop.
- Server-side load balancing: a proxy in the middle does the picking.
Whichever you use, add health checks (readiness and liveness are different questions), connection reuse, and timeouts and retry budgets; a stale list sends traffic to dead instances.
7. The service mesh
When there are many services, every one of them needs the same plumbing: mutual TLS, retries, timeouts, circuit breaking, load balancing, metrics and traces. Writing that into a library for each language is expensive and hard to keep consistent. A service mesh moves it out of the application into the infrastructure: a small proxy (a sidecar, often Envoy) runs next to every service instance and handles all of its inbound and outbound traffic, and a control plane configures them.
Diagram: Sidecar proxies and a control plane
What it gives you, without changing application code:
- Mutual TLS between services with automatic certificate rotation, and identity-based authorisation policy ("only the order service may call payments").
- Traffic management: retries, timeouts, circuit breaking, canary releases and traffic shifting (send 5% of requests to the new version), fault injection for testing.
- Uniform telemetry: request rate, error rate and latency for every service pair, and trace spans (the application must still forward the trace headers).
What it costs: extra latency and CPU on every hop (two proxies per call, typically a millisecond or less but not free), memory for each sidecar, a complex control plane to run and upgrade, and a new class of failures and misconfigurations that are hard to debug. Newer designs run the proxy per node or without sidecars to reduce this overhead. A mesh also does not fix design problems: retries configured in the mesh and in the application multiply (see the resilience guide), and it cannot make a chatty design fast.
Do you need one? Often not at first. With a handful of services, a good client library, a gateway and Kubernetes network policies are enough. Consider a mesh when you have dozens of services in several languages, a hard requirement for mutual TLS everywhere, and a platform team to operate it.
8. Data, queries and consistency
With a database per service you lose joins and cross-service transactions:
- API composition: a caller (or a gateway or BFF) calls several services and combines the results. Simple, but latency and availability multiply, and large joins are impractical.
- A read model (CQRS): a service keeps its own denormalised copy of the data it needs from others, updated from their events; the query is then local and fast, and the data is eventually consistent. The event sourcing and CQRS guide covers this.
- Eventual consistency by design: accept that "the order is placed but the loyalty points arrive a second later", and tell the user the truth in the interface.
- The outbox pattern for publishing an event atomically with a local database change, so no event is lost or invented.
- Multi-tenant concerns (isolation, per-tenant limits, noisy neighbours) apply to the whole platform: see designing a multi-tenant system.
Moving to a database per service, step by step
Splitting the data is usually harder than splitting the code, because the shared database is where the coupling hides: joins, foreign keys, shared tables and transactions that span what will become several services. A migration that works on a live system goes in small, reversible steps:
- Map ownership. For each table decide which capability owns it (who may write it). Tables written by several parts of the monolith are the first problem to resolve, not the last.
- Separate inside the database first. Give each future service its own schema and its own database user, and revoke access to other schemas. Violations now fail loudly, while everything is still one database and one deployment.
- Replace each cross-boundary join and foreign key. Options: an API call, a batched API call, a local read model filled from events, or accepting that the data is duplicated and the foreign key becomes a reconciliation job.
- Synchronise during the move. Use change data capture or the outbox pattern (event sourcing and CQRS); avoid dual writes from application code, which drift apart when one write fails.
- Cut reads over, then writes, behind a flag, comparing old and new results (shadow reads) before the switch. Keep the old data for a rollback period.
- Move the data physically (a separate server or managed instance) last. Many services can stay in one database server with separate schemas for a long time.
What a join becomes. I measured a page of 50 orders with the customer's name, with 100,000 orders and 10,000 customers. In one database it is a single SQL join. After the split, the customer data sits behind an HTTP service (real HTTP on loopback, so there is no network latency in these numbers):
| How the page is built | Median | 99th percentile |
|---|---|---|
| one SQL join in a shared database | 0.040 ms | 0.077 ms |
| 50 HTTP calls, one per order | 11.5 ms | 16.7 ms |
one batched HTTP call (ids=1,2,3,...) | 0.45 ms | 1.2 ms |
| a local read model (the order service keeps a copy of the names) | 0.036 ms | 0.062 ms |
The naive version is about 290 times slower and, with a real network between the services, it would add 50 round trips on top; batching brings it back to roughly 11 times the join; a local read model costs the same as the join, in exchange for eventual consistency (a renamed customer shows the old name until the event arrives) and a copy of the data to keep correct. The same trade appears everywhere:
| What you give up | What replaces it | The price |
|---|---|---|
| joins across the boundary | batched calls, read models, a reporting store | latency, stale data, duplicated data |
| foreign keys across the boundary | validation at write time, events, reconciliation jobs | orphans are possible and must be detected |
| one ACID transaction across services | sagas and the outbox | compensations, intermediate states visible to users |
| one place to query for reports | events into a warehouse or search index | a pipeline to build and watch |
| one database to back up and patch | N databases | operational work for each |
What you gain: each service can choose and scale its storage, a schema change in one service cannot break another, and an outage or a slow query in one database is contained. Do not pay the price without a reason: if two parts of the system must always change together and be consistent, they probably belong in the same service.
9. Operating many services
- Every service needs a pipeline, a runbook, dashboards and alerts. Standardise them with templates, or the effort grows with the number of services.
- Observability is not optional: propagate a trace id through every hop and log it; without it, a slow request is a search through dozens of logs. See observability.
- Deploy safely: rolling, blue-green or canary releases, feature flags, and automatic rollback on error-rate or latency regressions.
- Test at several levels: unit and component tests per service, contract tests between them, and a few end-to-end tests of critical journeys (not hundreds, they are slow and flaky).
- Limit the blast radius: timeouts, bulkheads and circuit breakers on every call; budgets for retries.
Migrating from a monolith: the strangler fig
Diagram: The strangler fig: route by route, behind a facade
Do not rewrite everything at once. Put a facade in front, extract one capability at a time (starting with one that is loosely coupled and valuable), route its traffic to the new service, migrate its data, and retire that part of the monolith. At every step the system works and you can stop.
9.1 A complete failure, end to end: one slow dependency takes down the rest
The earlier sections measured single effects (availability, latency, retries). Real outages combine them. I simulated a small system in simulated time (script in the appendix; a model, not a real deployment): clients call Orders; Orders has 40 workers, calls Inventory (fast) and Payments (it can finish 20 charges at a time, 50 ms each, so about 400 per second); browsing is served by Orders too and never calls Payments. Load: 100 checkouts and 200 browses per second. Clients give up after 3 seconds. For 30 seconds Payments becomes slow (800 ms per charge, as with a degraded database), so it can now finish only about 25 charges per second while 100 are asked for.
Diagram: The simulated system: browsing and checkout share Orders' workers
What happens with the naive settings (1 s timeout, 3 retries with no delay, one shared pool, no circuit breaker, no idempotency key):
- Payments' queue grows and calls start to exceed the 1 s timeout. The caller gives up, but Payments still does the work: 97% of the charges it finished during the fault had nobody waiting.
- Each checkout now holds an Orders worker for up to 4 seconds (4 attempts of 1 s). The 40 workers are all busy within a moment.
- Browsing, which never touches Payments, fails too: it queues behind the blocked checkouts and only 2% of browses succeeded during the fault. This is the cascade: the failure of one dependency became the failure of everything that shares a pool with it.
- Partial success: charges finish after the caller timed out, so customers are charged and told it failed (about 3,700 of them), and retries of an unfinished charge charge again (about 320 duplicates).
- After Payments recovers, the backlog and the queued retries keep checkout broken for about 8 more seconds.
Fixing one thing at a time (average of three runs; counts are for the whole 70-second run):
| Configuration | Checkout success during the fault | Browse success during the fault | Payments offered (can do 25/s) | Duplicate charges | Charged but client saw a failure | Checkout healthy again after |
|---|---|---|---|---|---|---|
| A naive | 0.6% | 2.0% | 40/s | 320 | 3,733 | 8 s |
| B only separate pools (30 for checkout, 10 for browsing) | 1.8% | 100% | 30/s | 237 | 240 | 1 s |
| C only a 250 ms timeout and a retry budget (at most one retry per ten requests) | 0.0% | 100% | 110/s | 403 | 3,933 | 9 s |
| D all of that, plus a circuit breaker, deadline propagation and an idempotency key | 0.0% | 100% | 1/s | 0 | 35 | 2 s |
| E D, and when Payments fails, accept the order as pending and charge it later | 100% accepted | 100% | 1/s | 0 | 0 | 0 s |
What each row teaches:
- B (bulkhead) protects the unrelated feature completely, and recovers fast because less work piled up, but checkout itself still fails, and the duplicates remain.
- C (short timeout, retry budget) is worse than naive for Payments: the workers are free again quickly, so more attempts reach the struggling service (110 per second against a capacity of 25), nearly all wasted, and checkout still fails and still produces thousands of duplicate or orphaned charges. A retry policy alone does not protect a dependency; a limit on concurrency or a breaker does.
- D (breaker, deadlines, idempotency) cuts the load on Payments to almost nothing, so its queue drains and checkout recovers two seconds after the fault ends; idempotency keys remove the duplicate charges; the few remaining "charged but failed" cases are requests already in flight when the breaker opened.
- E (degrade on purpose) is a business decision, not a technical one: accepting an order whose payment is pending means a queue, a reconciliation process, and a rule for orders that cannot be charged later. It is only possible for operations that can be completed later.
The general lessons are the ones of the resilience patterns guide, now seen together: every call needs a timeout shorter than its caller's, retries need a budget and backoff, give each dependency its own pool, stop calling a dependency that is failing, make operations idempotent so a retry cannot repeat them, and decide in advance what the system does instead. Model limits: the real Payments service would also shed load, clients retry on their own, and the pool sizes, timeouts and fault length here are my choices; change them in the script and the table changes, but the order of the effects (shared pool first, retries second, duplicates third) is what to look for.
10. Common mistakes
- Starting with microservices for a small team or an unproven product, and spending the effort on infrastructure.
- Splitting by technical layer or by entity ("user service", "order table service") rather than by business capability.
- A shared database, or a shared library full of business logic, between services.
- Long synchronous call chains, and retries at every level.
- No timeouts, no circuit breakers: one slow service takes down the others.
- Treating the gateway as the place for business logic.
- Adopting a service mesh before there is a problem for it to solve.
- Ignoring versioning: breaking changes shipped without a migration path.
- Underestimating the operational workload: no standard tooling, no tracing, no ownership of each service.
11. Things to mention in an interview
- That microservices are mostly about teams and deployment independence, and that you would start with a modular monolith unless there is a clear reason.
- The cost of the network: latency, partial failure, the availability arithmetic (99.9% to the power n), and the fan-out tail.
- Service boundaries from bounded contexts, data owned by one service, and the symptoms of a distributed monolith.
- Synchronous against asynchronous communication, sagas for cross-service processes, and the outbox pattern.
- What a gateway does (and does not), the BFF variation, and the difference from a load balancer.
- Discovery and health checks; what a service mesh adds (mTLS, traffic policy, telemetry) and what it costs.
- A migration plan with the strangler fig.
Quick recap
- Microservices trade function calls for network calls to gain independent teams and deployments. A call over loopback was about 4,700 times slower than a function call (19 ns against 89 µs median).
- Availability multiplies (ten 99.9% services in series give 99.0%), latency adds in a chain and the tail grows with fan-out (40% of requests hit a slow call when waiting for 50 parallel ones).
- Draw boundaries by business capability, give each service its own data, and watch for a distributed monolith.
- Prefer asynchronous events for work that can wait, synchronous calls where the user waits, and sagas plus the outbox pattern for cross-service consistency.
- The API gateway is the thin front door (routing, authentication, limits, aggregation, BFF per client); the service mesh moves retries, mTLS and telemetry into sidecars, at a cost in latency and complexity.
- Migrate with the strangler fig, standardise operations, and make observability and resilience part of every service.
Appendix: the experiment code
The first script (Python 3, standard library only, fixed seeds) computes the availability table and simulates the latency of chains and fan-outs from a lognormal distribution with a median of 20 ms; the distribution is an assumption chosen to have a realistic long tail, not a measurement of any real service, and it treats calls as independent, which real calls often are not. The second script (Node.js 22) measures a function call, a JSON round trip and an HTTP request over loopback on a small two-core virtual machine, so the nanoseconds will differ on your hardware; the ratio is what matters. It starts its own HTTP server and does not need the internet.
Availability and tail latency arithmetic
import random, math
# 1. availability of a request that needs n services in series, each 99.9% available
print("n services in series, each 99.9% available:")
for n in (1, 3, 5, 10, 20, 50):
a = 0.999 ** n
print(f" n={n:>2}: {a*100:.2f}% available = {(1-a)*365*24*60:,.0f} minutes of downtime per year")
# with a fallback (degraded answer) on a share of the dependencies
print("10 services, 99.9% each, when k of them are optional (the page still works without them):")
for k in (0, 3, 6, 9):
a = 0.999 ** (10 - k); print(f" {k} optional: {a*100:.3f}%")
# replicas: an instance 99% available, r replicas behind a load balancer
for r in (1, 2, 3): print(f" one service at 99%: {r} independent replicas -> {(1-(0.01)**r)*100:.4f}%")
# 2. tail latency: a request that calls n services in parallel and waits for all of them
rnd = random.Random(3)
def call(): # lognormal: median 20 ms, a long tail (1% of calls above ~ 85 ms)
return math.exp(rnd.gauss(math.log(20), 0.55))
def pct(v, p): v = sorted(v); return v[int(len(v) * p) - 1]
single = [call() for _ in range(100000)]
P99 = pct(single, .99)
print(f"one call: median {pct(single, .5):.0f} ms, 99th percentile {pct(single, .99):.0f} ms")
for n in (1, 5, 10, 50):
fan = [max(call() for _ in range(n)) for _ in range(20000)]
slow = sum(x > P99 for x in fan) / len(fan)
print(f" wait for the slowest of {n:>2} parallel calls: median {pct(fan, .5):.0f} ms, 99th {pct(fan, .99):.0f} ms; {slow*100:.1f}% of requests hit at least one 'p99-slow' call")
# 3. a synchronous chain against parallel and asynchronous designs
chain = lambda n: [sum(call() for _ in range(n)) for _ in range(20000)]
for n in (1, 3, 5, 8):
c = chain(n); print(f" chain of {n} synchronous calls: median {pct(c, .5):.0f} ms, 99th {pct(c, .99):.0f} ms")
The cost of a call: function, JSON and HTTP (Node.js)
import http from 'node:http';
const handler = (req, res) => { let b = ''; req.on('data', (c) => (b += c)); req.on('end', () => { const o = JSON.parse(b); res.setHeader('content-type', 'application/json'); res.end(JSON.stringify({ total: o.items.reduce((s, i) => s + i.price * i.qty, 0) })); }); };
const server = http.createServer(handler).listen(0);
await new Promise((r) => setTimeout(r, 100));
const order = { id: 7, items: Array.from({ length: 10 }, (_, i) => ({ sku: 'sku-' + i, price: 10 + i, qty: 2 })) };
const total = (o) => o.items.reduce((s, i) => s + i.price * i.qty, 0);
// 1. a function call
let s = 0; const N1 = 2_000_000; let t = performance.now();
for (let i = 0; i < N1; i++) s += total(order);
const fn = ((performance.now() - t) / N1) * 1e6;
// 2. the same call with the JSON encode and decode a network call needs, but no network
t = performance.now(); const N2 = 200_000;
for (let i = 0; i < N2; i++) s += total(JSON.parse(JSON.stringify(order)));
const ser = ((performance.now() - t) / N2) * 1e6;
// 3. a real HTTP request over loopback, keep-alive, one at a time
const agent = new http.Agent({ keepAlive: true, maxSockets: 1 });
const call = () => new Promise((ok) => { const r = http.request({ port: server.address().port, method: 'POST', agent, headers: { 'content-type': 'application/json' } }, (res) => { let b = ''; res.on('data', (c) => (b += c)); res.on('end', () => ok(JSON.parse(b))); }); r.end(JSON.stringify(order)); });
for (let i = 0; i < 2000; i++) await call();
const lat = []; for (let i = 0; i < 10000; i++) { const t0 = performance.now(); await call(); lat.push(performance.now() - t0); }
lat.sort((a, b) => a - b);
console.log(`in-process function call: ${fn.toFixed(0)} ns`);
console.log(`function call plus JSON encode and decode of the order: ${ser.toFixed(0)} ns (${(ser / fn).toFixed(0)}x)`);
console.log(`HTTP call over loopback, keep-alive, one at a time: median ${(lat[5000] * 1000).toFixed(0)} us, 99th percentile ${(lat[9900] * 1000).toFixed(0)} us (${((lat[5000] * 1e6) / fn).toFixed(0)}x the function call)`);
// 4. concurrency: requests per second with a pool of connections
const t2 = performance.now(); const agent2 = new http.Agent({ keepAlive: true, maxSockets: 32 });
const call2 = () => new Promise((ok) => { const r = http.request({ port: server.address().port, method: 'POST', agent: agent2, headers: { 'content-type': 'application/json' } }, (res) => { res.resume(); res.on('end', ok); }); r.end(JSON.stringify(order)); });
let done = 0; await Promise.all(Array.from({ length: 32 }, async () => { for (let i = 0; i < 1000; i++) { await call2(); done++; } }));
console.log(`32 concurrent connections: ${(done / ((performance.now() - t2) / 1000)).toFixed(0)} requests per second, client and server on the same two cores`);
if (s === 42) console.log(); server.close(); agent.destroy(); agent2.destroy(); process.exit(0);
A cascading failure: timeouts, retries, pools, a circuit breaker and idempotency
Python 3, standard library only; it simulates time, so a run takes a few seconds and is repeatable from its seeds.
"""A cascading failure across three services, in simulated time (milliseconds).
Clients -> Orders (a worker pool) -> Inventory (fast) and Payments (a service with limited capacity).
Payments becomes slow for 30 s. Which settings of timeouts, retries, pools, a circuit breaker and idempotency keys decide what happens?"""
import heapq, itertools, random, collections, statistics
CLIENT_TIMEOUT = 3000 # the browser or the gateway gives up after 3 s
FAULT = (15_000, 45_000) # Payments is slow in this window; the run lasts 70 s
UNTIL = 70_000
CHECKOUT_RPS, BROWSE_RPS = 100, 200
PAY_CAPACITY, PAY_MS, PAY_SLOW_FACTOR = 20, 50, 16 # 20 concurrent charges of 50 ms, or 800 ms when slow
class Sim:
def __init__(s, seed): s.t = 0.0; s.q = []; s.n = itertools.count(); s.rnd = random.Random(seed)
def at(s, dt, fn): heapq.heappush(s.q, (s.t + dt, next(s.n), fn))
def run(s, until):
while s.q and s.q[0][0] <= until:
s.t, _, fn = heapq.heappop(s.q); fn()
class Pool:
"""A fixed number of workers and a FIFO queue (optionally bounded: extra requests are rejected at once)."""
def __init__(s, size, max_queue=None): s.size, s.busy, s.wait, s.max_queue = size, 0, collections.deque(), max_queue
def submit(s, job, reject):
if s.busy < s.size: s.busy += 1; job(s.release)
elif s.max_queue is not None and len(s.wait) >= s.max_queue: reject()
else: s.wait.append(job)
def release(s):
s.busy -= 1
if s.wait and s.busy < s.size: s.busy += 1; s.wait.popleft()(s.release)
class Breaker:
"""Opens when at least half of the last 20 outcomes failed; after 5 s lets one probe through."""
def __init__(s): s.hist = collections.deque(maxlen=20); s.open_until = 0; s.probing = False
def allow(s, t):
if t < s.open_until: return False
if s.open_until and not s.probing: s.probing = True; return True # half-open: a single probe
return not s.open_until
def record(s, ok, t):
if s.probing: s.probing = False; s.hist.clear(); s.open_until = 0 if ok else t + 5000; return
s.hist.append(ok)
if len(s.hist) >= 10 and s.hist.count(False) >= len(s.hist) / 2: s.open_until = t + 5000; s.hist.clear()
CONFIGS = {
"A naive: 1 s timeout, 3 retries, one shared pool": dict(timeout=1000, attempts=4, backoff=0, budget=False, breaker=False, bulkhead=False, idem=False, deadline=False, fallback=False),
"B only separate pools (bulkhead)": dict(timeout=1000, attempts=4, backoff=0, budget=False, breaker=False, bulkhead=True, idem=False, deadline=False, fallback=False),
"C only short timeout and a retry budget": dict(timeout=250, attempts=2, backoff=100, budget=True, breaker=False, bulkhead=False, idem=False, deadline=False, fallback=False),
"D all together, plus breaker and idempotency": dict(timeout=250, attempts=2, backoff=100, budget=True, breaker=True, bulkhead=True, idem=True, deadline=True, fallback=False),
"E all of D, and when Payments fails accept the order as pending": dict(timeout=250, attempts=2, backoff=100, budget=True, breaker=True, bulkhead=True, idem=True, deadline=True, fallback=True),
}
def run(cfg, seed):
sim = Sim(seed); rnd = sim.rnd
pay = Pool(PAY_CAPACITY)
if cfg["bulkhead"]: pools = {"checkout": Pool(30, 30), "browse": Pool(10, 30)} # the same 40 workers, split
else: pools = {"checkout": (shared := Pool(40)), "browse": shared}
breaker = Breaker(); tokens = [10.0]
charges = collections.Counter(); attempts = []; reqs = []; waste = [0, 0] # waste: [charges nobody waited for, charges done], during the fault
def pay_call(req_id, reply):
def job(release):
slow = FAULT[0] <= sim.t < FAULT[1]
svc = PAY_MS * (PAY_SLOW_FACTOR if slow else 1) * (0.8 + 0.4 * rnd.random())
def done():
if not (cfg["idem"] and charges[req_id]): charges[req_id] += 1 # an idempotency key makes the repeat a no-op
release(); used = reply()
if FAULT[0] <= sim.t < FAULT[1]: waste[1] += 1; waste[0] += (not used)
sim.at(svc, done)
pay.submit(job, lambda: None)
def finish(r, ok, why=""):
if "done" in r: return
r["done"] = True; r["ok"] = ok and sim.t - r["t0"] <= CLIENT_TIMEOUT; r["why"] = why; r["lat"] = sim.t - r["t0"]
def fail(r, why):
if cfg["fallback"]: finish(r, True, "pending") # accept the order now, charge it later from a queue
else: finish(r, False, why)
def checkout(r):
def job(release):
if cfg["deadline"] and sim.t - r["t0"] > CLIENT_TIMEOUT: release(); finish(r, False, "expired"); return
def attempt(n):
if cfg["breaker"] and not breaker.allow(sim.t): release(); fail(r, "breaker"); return
st = {"over": False}; attempts.append(sim.t)
def reply():
if st["over"]: return False # the caller gave up: this work was wasted
st["over"] = True; breaker.record(True, sim.t); release(); finish(r, True); return True
def timeout():
if st["over"]: return
st["over"] = True; breaker.record(False, sim.t)
if n < cfg["attempts"] and (not cfg["budget"] or tokens[0] >= 1):
if cfg["budget"]: tokens[0] -= 1
sim.at(cfg["backoff"] * 2 ** (n - 1) * (0.5 + rnd.random()), lambda: attempt(n + 1))
else: release(); fail(r, "timeout")
if n == 1 and cfg["budget"]: tokens[0] = min(10.0, tokens[0] + 0.1)
pay_call(r["id"], reply); sim.at(cfg["timeout"], timeout)
sim.at(10, lambda: attempt(1)) # the inventory check
pools["checkout"].submit(job, lambda: finish(r, False, "shed"))
def browse(r):
def job(release):
if cfg["deadline"] and sim.t - r["t0"] > CLIENT_TIMEOUT: release(); finish(r, False, "expired"); return
def done(): release(); finish(r, True)
sim.at(5, done)
pools["browse"].submit(job, lambda: finish(r, False, "shed"))
def arrivals(kind, rps, fn):
t = 0.0
while True:
t += rnd.expovariate(rps / 1000)
if t > UNTIL - CLIENT_TIMEOUT: break # do not count requests whose outcome would be cut off by the end of the run
r = {"id": len(reqs), "kind": kind, "t0": t}; reqs.append(r); sim.at(t, lambda r=r: fn(r))
arrivals("checkout", CHECKOUT_RPS, checkout); arrivals("browse", BROWSE_RPS, browse)
sim.run(UNTIL)
return reqs, charges, attempts, waste
def phase(t0):
return "before" if t0 < FAULT[0] else "during" if t0 < FAULT[1] else "after"
def summarise(cfg, seeds=(1, 2, 3)):
acc = collections.defaultdict(list)
for seed in seeds:
reqs, charges, attempts, waste = run(cfg, seed)
for ph in ("before", "during", "after"):
for kind in ("checkout", "browse"):
rs = [r for r in reqs if r["kind"] == kind and phase(r["t0"]) == ph]
acc[(ph, kind, "ok")].append(100 * sum(bool(r.get("ok")) for r in rs) / len(rs))
lat = sorted(r["lat"] for r in rs if r.get("ok"))
acc[(ph, kind, "p99")].append(lat[int(len(lat) * .99)] if lat else float("nan"))
co = [r for r in reqs if r["kind"] == "checkout" and FAULT[0] <= r["t0"] < FAULT[1]]
att = [a for a in attempts if FAULT[0] <= a < FAULT[1]]
acc["amp"].append(len(att) / len(co)); acc["rate"].append(len(att) / ((FAULT[1] - FAULT[0]) / 1000)); acc["waste"].append(100 * waste[0] / max(1, waste[1]))
acc["pend"].append(100 * sum(r.get("why") == "pending" for r in co) / len(co))
acc["dup"].append(sum(1 for r in reqs if r["kind"] == "checkout" and charges[r["id"]] > 1))
acc["lost"].append(sum(1 for r in reqs if r["kind"] == "checkout" and charges[r["id"]] >= 1 and not r.get("ok")))
# recovery: seconds after the fault ends until a 5-second window of checkouts is at least 95% successful
rec = None
for s in range(0, 25):
w = [r for r in reqs if r["kind"] == "checkout" and FAULT[1] + s * 1000 <= r["t0"] < FAULT[1] + s * 1000 + 5000]
if w and sum(bool(r.get("ok")) for r in w) / len(w) >= .95: rec = s; break
acc["rec"].append(rec if rec is not None else 25)
m = lambda k: statistics.mean(acc[k])
return m, acc
if __name__ == "__main__":
print(f"Payments is slow ({PAY_MS} ms -> {PAY_MS*PAY_SLOW_FACTOR} ms per charge, capacity {PAY_CAPACITY} at a time) from {FAULT[0]//1000} s to {FAULT[1]//1000} s; {CHECKOUT_RPS} checkouts/s and {BROWSE_RPS} browses/s (browsing never calls Payments).")
print("Success = the client got a good answer within 3 s. Average of 3 seeds.\n")
for name, cfg in CONFIGS.items():
m, _ = summarise(cfg)
print(name)
print(f" checkout success % before {m(('before','checkout','ok')):5.1f} during {m(('during','checkout','ok')):5.1f} after {m(('after','checkout','ok')):5.1f}")
print(f" browse success % before {m(('before','browse','ok')):5.1f} during {m(('during','browse','ok')):5.1f} after {m(('after','browse','ok')):5.1f} (unrelated to Payments)")
print(f" Payments during the fault: {m('rate'):.0f} attempts/s offered (it can finish 25/s), {m('waste'):.0f}% of its finished charges had no caller waiting; {m('pend'):.0f}% of checkouts answered 'accepted, pending'")
print(f" duplicate charges: {m('dup'):.0f} | charged but the client saw a failure: {m('lost'):.0f} | checkout back to >=95% after the fault ends: {m('rec'):.0f} s")
What a join becomes after the split: SQL join, per-row HTTP calls, a batched call and a local read model
Python 3 with sqlite3 and http.server, on loopback. Absolute timings depend on the machine; the ratios are the point.
"""What a join becomes after 'database per service': listing 50 orders with the customer's name.
Orders and customers live in separate SQLite databases; the customer service answers over real HTTP on loopback (stdlib only)."""
import http.client, http.server, json, random, socket, sqlite3, statistics, threading, time
N_CUSTOMERS, N_ORDERS, PAGE = 10_000, 100_000, 50
rnd = random.Random(4)
# one shared database (before the split)
shared = sqlite3.connect(":memory:", check_same_thread=False)
shared.executescript("create table customers(id integer primary key, name text); create table orders(id integer primary key, customer_id int, total int);")
shared.executemany("insert into customers values (?,?)", ((i, f"customer {i}") for i in range(N_CUSTOMERS)))
shared.executemany("insert into orders values (?,?,?)", ((i, rnd.randrange(N_CUSTOMERS), rnd.randrange(1, 500)) for i in range(N_ORDERS)))
shared.execute("create index o_c on orders(customer_id)")
# after the split: the customer service owns `customers`, the order service owns `orders`
cust_db = sqlite3.connect(":memory:", check_same_thread=False)
cust_db.execute("create table customers(id integer primary key, name text)")
cust_db.executemany("insert into customers values (?,?)", shared.execute("select id, name from customers"))
ord_db = sqlite3.connect(":memory:", check_same_thread=False)
ord_db.execute("create table orders(id integer primary key, customer_id int, total int)")
ord_db.executemany("insert into orders values (?,?,?)", shared.execute("select id, customer_id, total from orders"))
class H(http.server.BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def setup(self): super().setup(); self.connection.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) # avoid the 40 ms delayed-ACK stall
def log_message(self, *a): pass
def do_GET(self):
if self.path.startswith("/customers/"):
row = cust_db.execute("select id, name from customers where id=?", (int(self.path.split("/")[2]),)).fetchone()
out = {"id": row[0], "name": row[1]}
else: # /customers?ids=1,2,3
ids = [int(x) for x in self.path.split("=")[1].split(",")]
out = [{"id": i, "name": n} for i, n in cust_db.execute(f"select id, name from customers where id in ({','.join('?'*len(ids))})", ids)]
b = json.dumps(out).encode()
self.send_response(200); self.send_header("Content-Length", str(len(b))); self.end_headers(); self.wfile.write(b)
srv = http.server.ThreadingHTTPServer(("127.0.0.1", 8120), H); threading.Thread(target=srv.serve_forever, daemon=True).start()
conn = http.client.HTTPConnection("127.0.0.1", 8120); conn.connect(); conn.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
def get(path): conn.request("GET", path); return json.loads(conn.getresponse().read())
def a_join():
return shared.execute("select o.id, o.total, c.name from orders o join customers c on c.id=o.customer_id order by o.id desc limit ?", (PAGE,)).fetchall()
def b_one_call_per_order():
rows = ord_db.execute("select id, customer_id, total from orders order by id desc limit ?", (PAGE,)).fetchall()
return [(i, t, get(f"/customers/{c}")["name"]) for i, c, t in rows]
def c_one_batched_call():
rows = ord_db.execute("select id, customer_id, total from orders order by id desc limit ?", (PAGE,)).fetchall()
names = {c["id"]: c["name"] for c in get("/customers?ids=" + ",".join(str(c) for _, c, _ in rows))}
return [(i, t, names[c]) for i, c, t in rows]
replica = {}
def d_local_read_model():
"""The order service keeps its own copy of the customer names, filled from customer events."""
rows = ord_db.execute("select id, customer_id, total from orders order by id desc limit ?", (PAGE,)).fetchall()
return [(i, t, replica[c]) for i, c, t in rows]
replica.update(dict(cust_db.execute("select id, name from customers")))
def bench(fn, reps=200):
fn(); xs = []
for _ in range(reps):
t = time.perf_counter(); fn(); xs.append((time.perf_counter() - t) * 1000)
xs.sort(); return statistics.median(xs), xs[int(len(xs) * .99)]
assert sorted(a_join()) == sorted(b_one_call_per_order()) == sorted(c_one_batched_call()) == sorted(d_local_read_model())
print(f"A page of {PAGE} orders with the customer name; {N_ORDERS:,} orders, {N_CUSTOMERS:,} customers. Milliseconds per page (median / 99th percentile, 200 runs):")
for label, fn in (("one SQL join in a shared database", a_join), (f"separate databases: {PAGE} HTTP calls, one per order", b_one_call_per_order),
("separate databases: one batched HTTP call", c_one_batched_call), ("separate databases: local read model (copy of the names)", d_local_read_model)):
m, p = bench(fn); print(f" {label:<58} {m:7.3f} / {p:7.3f} ms")
srv.shutdown()
Sources
- Sam Newman, Building Microservices, 2nd edition (O'Reilly), and Monolith to Microservices (O'Reilly) for the strangler fig and decomposition patterns.
- Martin Fowler and James Lewis, Microservices: a definition of this new architectural term, and Fowler's MonolithFirst and StranglerFigApplication.
- Chris Richardson, microservices.io, the pattern language including API gateway, database per service, saga and transactional outbox.
- Jeffrey Dean and Luiz André Barroso, The Tail at Scale (Communications of the ACM, 2013).
- Melvin Conway, How Do Committees Invent? (Datamation, 1968).
- Documentation of Envoy and Istio for the sidecar and control-plane model, and the Kubernetes documentation on Services and EndpointSlices.
- Eric Evans, Domain-Driven Design (Addison-Wesley, 2003), for bounded contexts.
- Melvin Conway, How Do Committees Invent? (Datamation, 1968); Matthew Skelton and Manuel Pais, Team Topologies (IT Revolution, 2019).
- Site Reliability Engineering (Google), chapter 22, Addressing Cascading Failures; Michael Nygard, Release It! (stability patterns: timeouts, circuit breakers, bulkheads).
- Chris Richardson, Database per service; Sam Newman, Monolith to Microservices (O'Reilly, 2019), the chapters on decomposing the database.
Practise this topic: System Design System Design › Reliability and scale