CS Fundamentals Trainer · Guides
Resilience patterns: timeouts, retries and circuit breakers
How a slow dependency spreads, and the tools that contain it: timeouts and deadlines, retries with backoff, jitter and budgets, circuit breakers, bulkheads, backpressure and load shedding, with measured experiments and two step-by-step animations.
30 min read
Every dependency your service calls will eventually be slow, wrong or gone. Resilience is what your service does next: whether one sick dependency stays a local problem or spreads until everything is down. The tools are few and old: timeouts, retries with backoff and jitter, circuit breakers, bulkheads, backpressure and load shedding. Each one is simple. The difficulty is that they interact: a retry policy that is reasonable for one client becomes an attack on the server when a thousand clients share it, and a timeout that is too generous is worse than none for the caller who is waiting.
This guide builds the picture in the order a request meets the tools. I ran small experiments for each number quoted here; the code is in the appendix.
1. How a failure spreads
A slow dependency is more dangerous than a dead one. A dead one refuses connections and the caller learns at once. A slow one holds on to the caller's resources (a thread, a connection, memory) for as long as it takes, and every new request that arrives meanwhile does the same.
Diagram: One slow dependency exhausts the callers' workers
I measured the effect with 10 workers and 50 requests that arrive together, while the dependency takes 1 second instead of a few milliseconds:
| Policy | Time to answer all 50 | Result |
|---|---|---|
| No timeout | 5.0 s (five rounds of 10) | all 50 succeed, but the last user waited five seconds |
| Timeout of 0.2 s | 1.0 s | all 50 fail fast; workers are free after 0.2 s each |
Notice what the timeout did and did not do. It did not make the requests succeed; the dependency really was too slow. It bounded the damage: no request occupies a worker for longer than 0.2 s, so the service keeps its capacity for requests that do not need the sick dependency, and the caller knows quickly and can use a fallback. That is the whole purpose of the rest of this guide: limit how long, how often and how many.
2. Timeouts
A timeout is the maximum time you will wait for an answer. Three rules cover most of the practice.
- Every call that leaves the process gets one. Many HTTP clients, database drivers and queue clients default to no timeout or to minutes. Check the default of every client you use; it is a common root cause of incidents.
- Choose it from the data. Look at the latency distribution of the dependency (see observability on percentiles) and set the timeout a little above the latency you consider normal, for example just above the 99th or 99.9th percentile. Too short and you turn slow successes into failures and, with retries, into load; too long and it protects nothing.
- Separate the kinds. A connect timeout (is anyone there?) can be short, a read timeout depends on the operation, and an overall deadline bounds the whole request.
Deadlines travel with the request
If a user-facing request has 2 seconds in total and service A spends 1.5 s before calling B, then B should not be given its usual 2 s timeout, because nobody will be waiting for the answer. Pass the remaining time budget along (gRPC calls this a deadline; in HTTP it is often a header you define) and have each hop use min(its own timeout, what is left). Without it, work continues for requests whose caller has already given up, and that wasted work is exactly what makes an overloaded system slower.
A timeout does not stop the work on the other side
When a caller's timeout fires, the caller stops waiting. It does not stop the dependency. Unless something tells it, the server keeps going: the query keeps running, the downstream calls are still made, the message is still processed, and a write may still commit after the caller has reported a failure. I measured the cost with a small TCP service whose request takes 1 second (20 steps of 50 ms) and 20 clients that give up after 0.2 s and close the connection:
| Server behaviour | Work steps executed (of 400 if every request ran to the end) |
|---|---|
| ignores the closed connection | 400 (100%), all of it for callers who had left |
| checks before each step whether the client is still connected | 80 (20%) |
The timeouts in the first experiment of this guide did stop the simulated dependency, because it lived in the same process and asyncio cancelled it. Across a network that does not happen by itself. Propagating cancellation takes work at every layer:
- Signal it. Use a mechanism that carries cancellation across the call: a context or cancellation token that is passed down (Go
context, .NETCancellationToken, Python task cancellation), a gRPC deadline and cancellation (the server sees the context cancelled), or closing the HTTP connection or stream (HTTP/2RST_STREAM). Whether the server notices a closed connection depends on the framework. - Check it. The server must look: pass the context to database calls and downstream clients, check it between steps of a long job, and stop. A handler that never looks at the cancellation signal cannot be cancelled.
- Know what cannot be undone. Cancellation is a request to stop, not a rollback. A transaction that has committed stays committed, a message already placed on a queue will be consumed, and an email already sent stays sent. For these, the timeout is an ambiguous outcome (see "What is safe to retry"), and the operation needs an idempotency key or a way to look up its result.
- Bound the work on the server too. A server-side deadline (taken from the request's remaining budget, or a fixed maximum) stops runaway requests even when the client never signals anything, and queues should drop requests whose deadline has already passed instead of starting them.
- Watch for the leftovers. Compare the work the dependency does with the work its callers still wanted (requests that finished after the caller's timeout; the number of in-flight queries compared with the number of live callers). The difference is the waste, and it is highest exactly when the system is overloaded.
3. Retries
A retry fixes transient failures: a dropped packet, a brief failover, a request that landed on a restarting instance. It makes persistent failures worse, because it multiplies the load on something that is already struggling.
The arithmetic is simple. If each attempt fails with probability p and you allow 3 retries, the expected number of attempts per request is 1 + p + p² + p³:
| Failure rate p | Attempts per request (3 retries) |
|---|---|
| 0.1 | 1.11 |
| 0.5 | 1.88 |
| 0.9 | 3.44 |
At a 10% failure rate retries cost almost nothing, which is why they look free in testing. At 90% the dependency receives 3.4 times its normal load exactly when it can least handle it. And this is for one layer. If service A retries 3 times against B, and B retries 3 times against C, a single failure at C can cause 4 × 4 = 16 calls. Retry at one layer, usually the one closest to the user or the one that knows the operation is safe to repeat, and not at every layer.
What is safe to retry
Retrying is only safe when repeating the request cannot do harm. Reads are. Writes are if they are idempotent (an UPDATE ... SET status = 'paid' is; balance = balance - 10 is not) or carry an idempotency key that the server checks; see designing a REST API and the limits of idempotency keys in distributed transactions. A timeout is not a failure: the request may have succeeded and only the answer was lost. That is why the key matters.
Which errors to retry
"Retry on failure" is too coarse. What to do depends on what the failure says about the request, and on whether the request can be repeated at all (the previous section):
| What happened | Retry? | Why and how |
|---|---|---|
| Connection refused, DNS failure, connection reset before the request was sent | Yes | The request never reached the server, so repeating it is safe even for writes. Try another instance if you can. |
| Timeout, or connection lost after the request was sent | Only if idempotent (or with an idempotency key) | The outcome is unknown: the server may have done the work. |
429 Too Many Requests, gRPC RESOURCE_EXHAUSTED | Yes, after the wait the server asks for (Retry-After), with backoff | The server is telling you to slow down. Retrying sooner makes it worse, and a retry budget should count these. |
503 Service Unavailable, gRPC UNAVAILABLE | Yes, with backoff and jitter, within the budget | Usually transient (restart, overload, failover). Respect Retry-After. |
502 Bad Gateway, 504 Gateway Timeout | Like a timeout | A proxy failed or gave up; the upstream may still have processed the request. |
500 Internal Server Error, gRPC INTERNAL, UNKNOWN | Rarely, once or twice, and only for idempotent requests | Often a deterministic bug that will fail again; repeated retries are only load. |
408 Request Timeout | Yes (idempotent) | The server did not receive the whole request in time. |
409 Conflict, 412 Precondition Failed | No, not as-is | The state changed. Re-read, decide again, then send a new request. |
400, 401, 403, 404, 422, gRPC INVALID_ARGUMENT, PERMISSION_DENIED, NOT_FOUND | No | The request itself is wrong or not allowed. Exception: one retry of a 401 after refreshing an expired token. |
gRPC DEADLINE_EXCEEDED, CANCELLED (by you) | No, the deadline is spent | The caller's budget is used up (see "Deadlines travel with the request"). |
Treat the table as a starting point and check your own services' conventions: some APIs return 500 for transient problems and 400 for conditions that would resolve by waiting, and different providers disagree on 429 versus 503. Make the retry decision in one place (a policy object or the client library), keep the list of retryable errors explicit, and log the error class of every retry so you can see which kinds actually succeed on the second attempt: an error that is almost never fixed by retrying should be removed from the list.
Backoff and jitter
Waiting a growing time between attempts (exponential backoff: 0.1 s, 0.2 s, 0.4 s, ...) gives the dependency room to recover. But if a thousand clients failed at the same moment, they will all retry at the same moment, after the same delay, again and again. Backoff alone does not break up the crowd. Jitter (randomness) does.
I simulated 1,000 clients that all fail at time zero and retry six times, and counted the largest number of clients that retry within the same 50 ms window in each round:
| Strategy | Largest crowd in one 50 ms window, rounds 1 to 6 |
|---|---|
| Fixed delay | 1000, 1000, 1000, 1000, 1000, 1000 |
| Exponential, no jitter | 1000, 1000, 1000, 1000, 1000, 1000 |
| Equal jitter (half fixed, half random) | 1000, 478, 267, 133, 81, 42 |
| Full jitter (random from 0 to the backoff ceiling) | 519, 260, 138, 71, 36, 24 |
Exponential backoff without jitter keeps the crowd intact: it is a thousand clients retrying together, only less often. Full jitter spreads the first round by half and keeps going. Add a cap on the delay and a limit on the number of attempts, and respect a Retry-After header when the server sends one.
Interactive animation: retry storm
The animation above shows the same effect in a server with limited capacity. When every client retries at once, the server stays overloaded and, in this model, serves nobody: it spends its capacity on requests that are about to be abandoned. This is a retry storm (or metastable failure): the system stays broken after the original cause has gone, because the retries themselves sustain the overload. With backoff and jitter the load falls away and everyone is served.
A retry budget
A fixed "retry up to 3 times" rule cannot tell a healthy period from a sick one. A retry budget limits retries as a share of normal traffic, for example "retries may be at most 10% of the requests sent in the last minute". During a small blip the budget allows the retries; during an outage it runs out and the client stops adding load. Several client libraries and service meshes offer this.
Budgets in a distributed system
A budget kept inside each client process is a good start, but it has limits in a fleet:
- Each process sees only its own traffic. With 1,000 instances, each limiting retries to 10% of its own requests, the fleet total is about 10% only when the load is evenly spread and the failures are spread with it. If the failing requests are concentrated on a few instances, or a few clients make most of the calls, the aggregate retry load on the dependency can be far higher or lower than 10%. A budget over a short window also lets a burst of retries through at the start of an outage, because every process still has tokens.
- Layers multiply their budgets. If A and B each allow 10% extra retries, the load on C can be 1.1 × 1.1 = 1.21 times normal, and with larger budgets or more layers, much more. The earlier advice stands: retry at one layer, or make the layers aware of each other.
- Tell the next hop that this is a retry. Propagate the attempt number (a header or metadata field such as
retry-attempt, defined by your platform) so that a downstream service knows a request is already a retry, does not retry it again, and can give it a lower priority than first attempts when it has to shed load. gRPC's retry throttling and proxies such as Envoy implement variants of these ideas: a token bucket per client that failures drain and successes refill, and an upper limit on active retries per upstream. - Enforce on the server as well. The server is the only place that sees the total load. It can reject retries first (by the attempt header), use admission control driven by its own queue and CPU, and return
Retry-Aftervalues that spread clients out. A client-side budget is polite; a server-side limit is the one that protects. - A shared budget needs shared state, which is itself a dependency (a central rate limiter) that can fail or add latency. Many systems accept approximate local budgets plus server-side shedding, and use a central limit only for very expensive operations.
- Measure it. Track the ratio of retries to first attempts per client and per dependency, the number of retries that succeeded, and how often the budget refused a retry. A budget that never refuses is too large; one that refuses constantly during normal operation is too small or hides a real problem.
4. Circuit breakers
If a dependency has failed five times in a row, the sixth attempt will probably fail too. A circuit breaker remembers this and fails fast: it does not call the dependency at all for a while.
Diagram: The three states of a circuit breaker
- Closed: normal. Calls go through and the breaker counts failures.
- Open: calls fail immediately, without touching the dependency. This protects the dependency (it gets a break) and the caller (it does not wait).
- Half-open: after a cool-down, let one trial call (or a few) through. If it succeeds, close the breaker; if it fails, open it again.
Step through a breaker below. It opens after 3 failures in a row, fails the next 3 requests at once, tries a trial call, and closes again once the service has recovered. In the run with a recovering service the dependency received 10 of the 16 requests and 6 failed immediately without waiting; if the service does not recover, only 7 of 16 reached it and 9 failed fast.
Interactive animation: circuit breaker
Design questions that decide whether a breaker helps:
- What counts as a failure? Timeouts and 5xx responses, yes. A 404 or a validation error is the caller's problem, not the dependency's.
- Count or rate? "3 in a row" is easy to explain. A failure rate over a window (for example, more than 50% of at least 20 calls in 10 seconds) copes better with busy services and with occasional errors.
- What does the caller do when it is open? The breaker gives you a fast failure, not an answer. Decide the fallback: a cached value, a default, a degraded page, or an honest error. Fail-fast with no plan just fails faster.
- One breaker per dependency, or per instance? A breaker per host stops sending traffic to one bad instance while the others serve; a single breaker for the whole service would open for everyone because of one bad node.
- Remember it is per process. Each instance of your service has its own breaker with its own view, so they open at slightly different moments. That is normally fine.
Tuning a breaker with production traffic
The numbers used above (3 in a row, a cool-down of a few requests) are for teaching. A threshold that works is derived from how the real dependency behaves, and a threshold copied from a blog post or a library default is a guess. A procedure that works well:
- Measure the baseline. From production metrics, take the normal error rate, the latency percentiles and the request rate per instance for each dependency, including their daily pattern (a quiet night and a busy afternoon give different statistics) and their normal incidents (deploys, failovers).
- Account for background errors. A dependency is rarely at 0% errors. I simulated a breaker that opens after 3 failures in a row, calling a dependency that is healthy except for a steady background error rate. By chance alone it opened about 3 times an hour at a 2% error rate and 100 calls per second (about 3 for each hour and each instance, so a fleet of 50 instances opens a few times every minute), and almost never at 0.5%. At 5 calls per second, 2% errors opened it about once in ten hours. "N in a row" is sensitive to the traffic volume, and it makes a healthy dependency look sick whenever volume is high.
- Prefer a rate over a window with a minimum volume for busy services: for example open when more than 50% of at least 20 calls in the last 10 seconds failed. Set the minimum so that a handful of unlucky calls cannot trip it, and set the window long enough to smooth the noise and short enough to react before the callers' workers are exhausted (the 50 requests and 10 workers of the first experiment gave a few seconds). For services with little traffic, consider a longer window or no breaker at all (a timeout may be enough).
- Count slow calls as failures with a threshold taken from the latency distribution (for example, slower than the 99.9th percentile of the baseline), because many outages start as latency, not as errors.
- Choose the cool-down and the probes from the real recovery time. If the dependency usually recovers in 30 seconds after a restart, a 2-second cool-down only produces repeated failing trial calls; one of several minutes keeps the caller degraded after the dependency is back. Allow a few trial calls in the half-open state, not one, so a single lucky or unlucky response does not decide, and let the breaker close gradually.
- Replay and shadow before you enforce. Run the candidate settings over recorded production metrics or a traffic replay and count how often and when they would have opened, compared with the real incidents. Better still, deploy the breaker in observe-only mode (log and count the "would open" decisions without acting) for a week or two. Then enable it, per dependency and with a gradual rollout.
- Keep watching. Review the metrics that matter: the number of opens per instance (an open that corresponds to no incident is a false positive), the time from the start of an incident to the open, the time from recovery to the close, and the calls rejected while open. Traffic grows, dependencies change and thresholds go stale, so tune after incidents and after large changes in load.
5. Bulkheads
A ship is divided into watertight compartments so that a hole floods one of them and not the hull. A bulkhead gives each dependency (or each kind of work) its own pool of threads, connections or concurrency slots, so that a slow dependency can only exhaust its own pool.
I tested it with two dependencies: 30 calls to a slow one (0.5 s each) arrive together with 30 calls to a fast one (0.01 s), with 10 slots available.
| Setup | Median latency of the fast calls | Slowest fast call |
|---|---|---|
| One shared pool of 10 slots | 1523 ms | 1534 ms |
| Separate pools of 10 | 21 ms | 31 ms |
With a shared pool, the fast calls queue behind the slow ones and take about 70 times longer, although the fast dependency was perfectly healthy. With separate pools they are barely affected. The same idea applies at larger scale: separate connection pools per downstream, separate worker pools for interactive and batch work, and even separate deployments for important and unimportant tenants (see multi-tenant systems).
The cost is capacity you cannot share: ten idle slots in one pool cannot help the other. Size pools from the real concurrency each dependency needs (arrival rate times latency, from Little's law) and not from a round number.
6. Backpressure and load shedding
A service that accepts more work than it can finish does not process more; it builds a queue, and queued requests wait. Two ideas keep this under control.
Backpressure pushes the limit back to the sender: a bounded queue, a full pipe that makes write block, a 429 Too Many Requests or 503 with Retry-After, a consumer that pulls at its own speed instead of being pushed to. The sender then slows down or chooses what to do.
Load shedding is deliberately refusing some requests to protect the rest, for example when the queue is longer than a limit, or when the CPU is above a threshold. Rejecting early is cheap; accepting a request and then answering after the caller has timed out is the most expensive outcome.
I simulated a service that can do 10 requests per tick while 15 arrive per tick, for 60 ticks:
| Queue | Requests served | Rejected | Median wait | Longest wait |
|---|---|---|---|---|
| Unbounded | 600 | 0 | 15.6 ticks | 31 ticks |
| Limited to 20 | 600 | 290 | 1.5 ticks | 2 ticks |
The service did the same amount of work (600 requests) in both cases. The unbounded queue did not help anyone: it only made everybody wait up to 31 ticks, probably longer than their timeouts, so much of that work would have been for callers who had gone. The bounded queue rejected 290 requests at once and served the rest in about two ticks. Rejecting some work is what keeps the rest useful. Choose what to shed: low-priority traffic first, requests that are already close to their deadline, or requests from the client that is sending the most. For the algorithms that limit request rates per client, see designing a rate limiter; and a queue between services is the subject of message queues.
7. Putting the pieces together
Diagram: Where each tool sits on a call
The order matters. Shedding and bulkheads decide whether to start work at all. The breaker decides whether to call. The timeout bounds the call. The retry policy sits outside the timeout and uses the breaker, so that when the breaker is open, a retry fails at once instead of waiting; and it stops when the request's deadline is used up.
Two more principles tie these together.
- Fail fast, degrade gracefully. For each dependency, decide in advance what the user sees when it is unavailable: stale data, a feature switched off, a queue that accepts the request for later. This is a product decision as much as a technical one.
- Test the failure paths. A breaker that has never opened and a fallback that has never run are unproven. Inject latency and errors in staging (and carefully in production: chaos experiments), then watch your dashboards: the open/closed state of breakers, retry rates, queue depth, rejected requests and the saturation of each pool are the metrics of this guide.
8. Common mistakes
- No timeout, or a default of 30 seconds or more on a call in a request path.
- Retries at every layer, multiplying the load by the product of the retry counts.
- Retrying non-idempotent writes without an idempotency key.
- Backoff without jitter, so the crowd stays together.
- Retrying errors that will not change: 400, 401, 403, 404 and 422 are the caller's problem and will fail again (see "Which errors to retry").
- A timeout longer than the caller's own timeout, so work continues for nobody.
- An unbounded queue or thread pool "so that nothing is lost": it moves the failure from a fast rejection to a slow, total one.
- A circuit breaker without a fallback or without monitoring, so that nobody notices when it is open, or one whose thresholds were never checked against production traffic.
- Assuming that a timeout cancels the work. The caller stops waiting; the server keeps going unless cancellation is propagated and checked.
- Shared pools for critical and optional work.
- Treating health checks as a resilience strategy. A load balancer that removes unhealthy instances (see load balancing) helps, but it also concentrates the load on the instances that remain.
9. Things to mention in an interview
- Start from the failure mode: a slow dependency exhausts the caller's workers, so you bound time, count and concurrency.
- Say that a timeout bounds damage but does not fix anything, and that you set it from the latency percentiles and pass a deadline downstream.
- Give the retry formula and the amplification across layers, and say that retries need idempotency and backoff with jitter, a cap and a budget; that the decision depends on the error type; and that a per-client budget needs server-side limits and an attempt marker to work across a fleet.
- Say that a timeout does not cancel the work on the other side, and how cancellation is propagated and checked.
- Describe the breaker's three states and its fallback, that it is per dependency (and often per host), and that its thresholds should come from measured production traffic, not defaults.
- Explain the bulkhead with the shared-pool example, and load shedding with "the same work is done, but with a short wait instead of a long one".
- Mention metastable failures and retry storms: a system can stay down after the trigger is gone.
Quick recap
- A slow dependency is more dangerous than a dead one; timeouts limit how long you wait and deadlines carry the remaining time downstream.
- Retries help transient faults and hurt persistent ones: at 90% failure with 3 retries the load is 3.4 times as high, and layers multiply. Use idempotency, backoff, jitter, a cap and a retry budget, choose what to retry by error type, and enforce limits on the server too.
- A timeout stops the caller waiting, not the work: in the test a server that ignored the closed connection did 100% of the abandoned work, and one that checked did 20%.
- A circuit breaker (closed, open, half-open) fails fast to protect both sides, but needs a fallback and thresholds tuned on real traffic (a 3-in-a-row breaker opened about 3 times an hour by chance at a 2% background error rate).
- Bulkheads isolate pools so a slow dependency cannot take the fast ones with it (1523 ms against 21 ms in the experiment).
- Backpressure and load shedding keep queues short: the same 600 requests were served, with a 31-tick longest wait against 2.
Appendix: the experiment code
The numbers in this guide come from the three scripts below (Python 3, standard library only; the timing results use real asyncio sleeps, so the milliseconds vary slightly between runs, and the random choices use fixed seeds). The two animations are small deterministic models of a client and a server, not measurements: they show the shape of the behaviour, and the simulated capacity and the point at which an overloaded server stops serving anybody are assumptions of the model. The cancellation test uses a local TCP service on one machine, and its step counts are exact because the work is a fixed number of steps; the breaker simulation is a random model (20 seeded runs) of a constant background error rate, which real traffic does not have: errors cluster, and that is a reason to measure your own.
Timeouts, retries, jitter, bulkheads and queues
import asyncio, random, time, math
# ---- 1. timeouts: without one, a slow dependency ties up every worker
async def slow_dep(delay): await asyncio.sleep(delay)
async def handler(sem, delay, timeout):
async with sem: # a worker slot
try:
await asyncio.wait_for(slow_dep(delay), timeout)
return "ok"
except asyncio.TimeoutError:
return "timeout"
async def exp_timeout(timeout):
sem = asyncio.Semaphore(10) # 10 workers
t0 = time.perf_counter()
# the dependency now takes 1.0 s; 50 requests arrive at once
res = await asyncio.gather(*[handler(sem, 1.0, timeout) for _ in range(50)])
return round(time.perf_counter() - t0, 2), res.count("ok"), res.count("timeout")
# ---- 2. retries multiply load: expected attempts when each attempt fails with probability p
def attempts(p, retries): # 1 + p + p^2 ... (retries extra tries)
return sum(p**i for i in range(retries + 1))
# ---- 3. backoff schedules
def schedule(kind, n, base=0.1, cap=10, rnd=None):
out = []
for k in range(n):
ceiling = min(cap, base * 2**k)
out.append({"fixed": base, "exp": ceiling, "full": rnd.uniform(0, ceiling),
"equal": ceiling/2 + rnd.uniform(0, ceiling/2)}[kind])
return out
def collisions(kind, clients=1000, n=6, bucket=0.05):
rnd = random.Random(7)
worst = []
times = [0.0]*clients
for k in range(n):
for c in range(clients):
times[c] += schedule(kind, k+1, rnd=rnd)[k]
counts = {}
for t in times: counts[int(t/bucket)] = counts.get(int(t/bucket), 0) + 1
worst.append(max(counts.values()))
return worst
# ---- 4. bulkhead: one slow dependency, two pools vs one shared pool
async def call(pool, delay):
async with pool:
await asyncio.sleep(delay)
async def exp_bulkhead(shared):
pools = {"slow": asyncio.Semaphore(10), "fast": asyncio.Semaphore(10)}
if shared: pools["fast"] = pools["slow"]
t0 = time.perf_counter()
# 30 slow calls (0.5 s) arrive together with 30 fast calls (0.01 s)
fast_lat = []
async def fast():
s = time.perf_counter(); await call(pools["fast"], 0.01); fast_lat.append(time.perf_counter()-s)
tasks = [asyncio.create_task(call(pools["slow"], 0.5)) for _ in range(30)]
await asyncio.sleep(0)
tasks += [asyncio.create_task(fast()) for _ in range(30)]
await asyncio.gather(*tasks)
fast_lat.sort()
return round(fast_lat[len(fast_lat)//2]*1000), round(fast_lat[-1]*1000)
# ---- 5. load shedding / queue: Little's law, bounded queue vs unbounded
def queue_sim(limit, ticks=60, arrivals=15, service=10):
q = 0; waits = []; shed = 0; served = 0
for t in range(ticks):
for _ in range(arrivals):
if limit is not None and q >= limit: shed += 1
else: q += 1; waits.append(q/service) # ticks until this one is served
s = min(q, service); q -= s; served += s
waits.sort()
return served, shed, round(waits[len(waits)//2],1), round(waits[-1],1)
if __name__ == "__main__":
for to in (None, 0.2):
print("timeout", to, asyncio.run(exp_timeout(to)))
for p in (0.1, 0.5, 0.9):
print("p", p, "3 retries ->", round(attempts(p, 3), 2), "attempts per request")
for kind in ("fixed","exp","full","equal"):
print(kind, "peak clients in one 50 ms bucket per round:", collisions(kind))
print("bulkhead shared (median, max fast ms):", asyncio.run(exp_bulkhead(True)))
print("bulkhead isolated(median, max fast ms):", asyncio.run(exp_bulkhead(False)))
for lim in (None, 20):
print("queue limit", lim, "(served, shed, median wait, max wait):", queue_sim(lim))
Cancellation: does the server keep working after the caller gave up?
"""When the caller gives up, does the server keep working? A tiny TCP service whose request takes 1 s (20 steps of 50 ms).
Clients time out after 0.2 s and close the connection. Variant A ignores that; variant B checks whether the client is
still connected before each step (cancellation propagation). Standard library only."""
import asyncio
STEPS, STEP_S, CLIENTS, TIMEOUT = 20, 0.05, 20, 0.2
async def run(check_client):
steps_done = 0
async def handle(reader, writer):
nonlocal steps_done
await reader.readline() # the request
for _ in range(STEPS):
if check_client and reader.at_eof(): # the client has hung up: stop working
break
await asyncio.sleep(STEP_S)
steps_done += 1
try: writer.write(b"done\n"); await writer.drain()
except Exception: pass
writer.close()
server = await asyncio.start_server(handle, "127.0.0.1", 0)
port = server.sockets[0].getsockname()[1]
async def client():
r, w = await asyncio.open_connection("127.0.0.1", port)
w.write(b"req\n"); await w.drain()
try: await asyncio.wait_for(r.readline(), TIMEOUT)
except asyncio.TimeoutError: pass
w.close() # give up
await asyncio.gather(*[client() for _ in range(CLIENTS)])
await asyncio.sleep(STEPS * STEP_S + 0.3) # let the server finish whatever it is still doing
server.close()
return steps_done
for check in (False, True):
done = asyncio.run(run(check))
total = CLIENTS * STEPS
print(f"server {'checks whether the client is still there' if check else 'ignores the closed connection '}: {done:>3} of {total} work steps executed ({done/total*100:.0f}%), "
f"{CLIENTS} callers had given up after {TIMEOUT} s")
A breaker that opens by chance
"""A breaker that opens after 3 failures in a row, on a dependency that is healthy except for a steady background error rate.
Counts how often it opens by chance alone, averaged over 20 random runs of one simulated hour each."""
import random
for p, rate in ((0.02, 100), (0.005, 100), (0.02, 5)):
calls = 3_600 * rate; total = 0
for seed in range(20):
r = random.Random(seed); run = 0
for _ in range(calls):
if r.random() < p:
run += 1
if run == 3: total += 1; run = 0 # the breaker opens, then counting starts again from zero
else: run = 0
print(f"background error rate {p*100:.1f}%, {rate:>3} calls/s: opens {total/20:.1f} times per hour by chance alone (calculation: {calls * (1-p) * p**3:.1f})")
Sources
- Michael Nygard, Release It! (Pragmatic Bookshelf), the origin of the circuit breaker and bulkhead patterns in this form.
- Martin Fowler, CircuitBreaker.
- Marc Brooker, Timeouts, retries and backoff with jitter (Amazon Builders' Library), and his Exponential Backoff And Jitter comparison.
- Google, Site Reliability Engineering, chapters Handling Overload and Addressing Cascading Failures.
- Nathan Bronson et al., Metastable Failures in Distributed Systems (HotOS 2021).
- gRPC documentation, Deadlines.
Practise this topic: System Design System Design › Reliability and scale