CS Fundamentals Trainer · Guides

Resilience patterns: timeouts, retries and circuit breakers

How a slow dependency spreads, and the tools that contain it: timeouts and deadlines, retries with backoff, jitter and budgets, circuit breakers, bulkheads, backpressure and load shedding, with measured experiments and two step-by-step animations.

30 min read

Every dependency your service calls will eventually be slow, wrong or gone. Resilience is what your service does next: whether one sick dependency stays a local problem or spreads until everything is down. The tools are few and old: timeouts, retries with backoff and jitter, circuit breakers, bulkheads, backpressure and load shedding. Each one is simple. The difficulty is that they interact: a retry policy that is reasonable for one client becomes an attack on the server when a thousand clients share it, and a timeout that is too generous is worse than none for the caller who is waiting.

This guide builds the picture in the order a request meets the tools. I ran small experiments for each number quoted here; the code is in the appendix.

1. How a failure spreads

A slow dependency is more dangerous than a dead one. A dead one refuses connections and the caller learns at once. A slow one holds on to the caller's resources (a thread, a connection, memory) for as long as it takes, and every new request that arrives meanwhile does the same.

Diagram: One slow dependency exhausts the callers' workers

I measured the effect with 10 workers and 50 requests that arrive together, while the dependency takes 1 second instead of a few milliseconds:

PolicyTime to answer all 50Result
No timeout5.0 s (five rounds of 10)all 50 succeed, but the last user waited five seconds
Timeout of 0.2 s1.0 sall 50 fail fast; workers are free after 0.2 s each

Notice what the timeout did and did not do. It did not make the requests succeed; the dependency really was too slow. It bounded the damage: no request occupies a worker for longer than 0.2 s, so the service keeps its capacity for requests that do not need the sick dependency, and the caller knows quickly and can use a fallback. That is the whole purpose of the rest of this guide: limit how long, how often and how many.

2. Timeouts

A timeout is the maximum time you will wait for an answer. Three rules cover most of the practice.

  1. Every call that leaves the process gets one. Many HTTP clients, database drivers and queue clients default to no timeout or to minutes. Check the default of every client you use; it is a common root cause of incidents.
  2. Choose it from the data. Look at the latency distribution of the dependency (see observability on percentiles) and set the timeout a little above the latency you consider normal, for example just above the 99th or 99.9th percentile. Too short and you turn slow successes into failures and, with retries, into load; too long and it protects nothing.
  3. Separate the kinds. A connect timeout (is anyone there?) can be short, a read timeout depends on the operation, and an overall deadline bounds the whole request.

Deadlines travel with the request

If a user-facing request has 2 seconds in total and service A spends 1.5 s before calling B, then B should not be given its usual 2 s timeout, because nobody will be waiting for the answer. Pass the remaining time budget along (gRPC calls this a deadline; in HTTP it is often a header you define) and have each hop use min(its own timeout, what is left). Without it, work continues for requests whose caller has already given up, and that wasted work is exactly what makes an overloaded system slower.

A timeout does not stop the work on the other side

When a caller's timeout fires, the caller stops waiting. It does not stop the dependency. Unless something tells it, the server keeps going: the query keeps running, the downstream calls are still made, the message is still processed, and a write may still commit after the caller has reported a failure. I measured the cost with a small TCP service whose request takes 1 second (20 steps of 50 ms) and 20 clients that give up after 0.2 s and close the connection:

Server behaviourWork steps executed (of 400 if every request ran to the end)
ignores the closed connection400 (100%), all of it for callers who had left
checks before each step whether the client is still connected80 (20%)

The timeouts in the first experiment of this guide did stop the simulated dependency, because it lived in the same process and asyncio cancelled it. Across a network that does not happen by itself. Propagating cancellation takes work at every layer:

3. Retries

A retry fixes transient failures: a dropped packet, a brief failover, a request that landed on a restarting instance. It makes persistent failures worse, because it multiplies the load on something that is already struggling.

The arithmetic is simple. If each attempt fails with probability p and you allow 3 retries, the expected number of attempts per request is 1 + p + p² + p³:

Failure rate pAttempts per request (3 retries)
0.11.11
0.51.88
0.93.44

At a 10% failure rate retries cost almost nothing, which is why they look free in testing. At 90% the dependency receives 3.4 times its normal load exactly when it can least handle it. And this is for one layer. If service A retries 3 times against B, and B retries 3 times against C, a single failure at C can cause 4 × 4 = 16 calls. Retry at one layer, usually the one closest to the user or the one that knows the operation is safe to repeat, and not at every layer.

What is safe to retry

Retrying is only safe when repeating the request cannot do harm. Reads are. Writes are if they are idempotent (an UPDATE ... SET status = 'paid' is; balance = balance - 10 is not) or carry an idempotency key that the server checks; see designing a REST API and the limits of idempotency keys in distributed transactions. A timeout is not a failure: the request may have succeeded and only the answer was lost. That is why the key matters.

Which errors to retry

"Retry on failure" is too coarse. What to do depends on what the failure says about the request, and on whether the request can be repeated at all (the previous section):

What happenedRetry?Why and how
Connection refused, DNS failure, connection reset before the request was sentYesThe request never reached the server, so repeating it is safe even for writes. Try another instance if you can.
Timeout, or connection lost after the request was sentOnly if idempotent (or with an idempotency key)The outcome is unknown: the server may have done the work.
429 Too Many Requests, gRPC RESOURCE_EXHAUSTEDYes, after the wait the server asks for (Retry-After), with backoffThe server is telling you to slow down. Retrying sooner makes it worse, and a retry budget should count these.
503 Service Unavailable, gRPC UNAVAILABLEYes, with backoff and jitter, within the budgetUsually transient (restart, overload, failover). Respect Retry-After.
502 Bad Gateway, 504 Gateway TimeoutLike a timeoutA proxy failed or gave up; the upstream may still have processed the request.
500 Internal Server Error, gRPC INTERNAL, UNKNOWNRarely, once or twice, and only for idempotent requestsOften a deterministic bug that will fail again; repeated retries are only load.
408 Request TimeoutYes (idempotent)The server did not receive the whole request in time.
409 Conflict, 412 Precondition FailedNo, not as-isThe state changed. Re-read, decide again, then send a new request.
400, 401, 403, 404, 422, gRPC INVALID_ARGUMENT, PERMISSION_DENIED, NOT_FOUNDNoThe request itself is wrong or not allowed. Exception: one retry of a 401 after refreshing an expired token.
gRPC DEADLINE_EXCEEDED, CANCELLED (by you)No, the deadline is spentThe caller's budget is used up (see "Deadlines travel with the request").

Treat the table as a starting point and check your own services' conventions: some APIs return 500 for transient problems and 400 for conditions that would resolve by waiting, and different providers disagree on 429 versus 503. Make the retry decision in one place (a policy object or the client library), keep the list of retryable errors explicit, and log the error class of every retry so you can see which kinds actually succeed on the second attempt: an error that is almost never fixed by retrying should be removed from the list.

Backoff and jitter

Waiting a growing time between attempts (exponential backoff: 0.1 s, 0.2 s, 0.4 s, ...) gives the dependency room to recover. But if a thousand clients failed at the same moment, they will all retry at the same moment, after the same delay, again and again. Backoff alone does not break up the crowd. Jitter (randomness) does.

I simulated 1,000 clients that all fail at time zero and retry six times, and counted the largest number of clients that retry within the same 50 ms window in each round:

StrategyLargest crowd in one 50 ms window, rounds 1 to 6
Fixed delay1000, 1000, 1000, 1000, 1000, 1000
Exponential, no jitter1000, 1000, 1000, 1000, 1000, 1000
Equal jitter (half fixed, half random)1000, 478, 267, 133, 81, 42
Full jitter (random from 0 to the backoff ceiling)519, 260, 138, 71, 36, 24

Exponential backoff without jitter keeps the crowd intact: it is a thousand clients retrying together, only less often. Full jitter spreads the first round by half and keeps going. Add a cap on the delay and a limit on the number of attempts, and respect a Retry-After header when the server sends one.

Interactive animation: retry storm

The animation above shows the same effect in a server with limited capacity. When every client retries at once, the server stays overloaded and, in this model, serves nobody: it spends its capacity on requests that are about to be abandoned. This is a retry storm (or metastable failure): the system stays broken after the original cause has gone, because the retries themselves sustain the overload. With backoff and jitter the load falls away and everyone is served.

A retry budget

A fixed "retry up to 3 times" rule cannot tell a healthy period from a sick one. A retry budget limits retries as a share of normal traffic, for example "retries may be at most 10% of the requests sent in the last minute". During a small blip the budget allows the retries; during an outage it runs out and the client stops adding load. Several client libraries and service meshes offer this.

Budgets in a distributed system

A budget kept inside each client process is a good start, but it has limits in a fleet:

4. Circuit breakers

If a dependency has failed five times in a row, the sixth attempt will probably fail too. A circuit breaker remembers this and fails fast: it does not call the dependency at all for a while.

Diagram: The three states of a circuit breaker

Step through a breaker below. It opens after 3 failures in a row, fails the next 3 requests at once, tries a trial call, and closes again once the service has recovered. In the run with a recovering service the dependency received 10 of the 16 requests and 6 failed immediately without waiting; if the service does not recover, only 7 of 16 reached it and 9 failed fast.

Interactive animation: circuit breaker

Design questions that decide whether a breaker helps:

Tuning a breaker with production traffic

The numbers used above (3 in a row, a cool-down of a few requests) are for teaching. A threshold that works is derived from how the real dependency behaves, and a threshold copied from a blog post or a library default is a guess. A procedure that works well:

  1. Measure the baseline. From production metrics, take the normal error rate, the latency percentiles and the request rate per instance for each dependency, including their daily pattern (a quiet night and a busy afternoon give different statistics) and their normal incidents (deploys, failovers).
  2. Account for background errors. A dependency is rarely at 0% errors. I simulated a breaker that opens after 3 failures in a row, calling a dependency that is healthy except for a steady background error rate. By chance alone it opened about 3 times an hour at a 2% error rate and 100 calls per second (about 3 for each hour and each instance, so a fleet of 50 instances opens a few times every minute), and almost never at 0.5%. At 5 calls per second, 2% errors opened it about once in ten hours. "N in a row" is sensitive to the traffic volume, and it makes a healthy dependency look sick whenever volume is high.
  3. Prefer a rate over a window with a minimum volume for busy services: for example open when more than 50% of at least 20 calls in the last 10 seconds failed. Set the minimum so that a handful of unlucky calls cannot trip it, and set the window long enough to smooth the noise and short enough to react before the callers' workers are exhausted (the 50 requests and 10 workers of the first experiment gave a few seconds). For services with little traffic, consider a longer window or no breaker at all (a timeout may be enough).
  4. Count slow calls as failures with a threshold taken from the latency distribution (for example, slower than the 99.9th percentile of the baseline), because many outages start as latency, not as errors.
  5. Choose the cool-down and the probes from the real recovery time. If the dependency usually recovers in 30 seconds after a restart, a 2-second cool-down only produces repeated failing trial calls; one of several minutes keeps the caller degraded after the dependency is back. Allow a few trial calls in the half-open state, not one, so a single lucky or unlucky response does not decide, and let the breaker close gradually.
  6. Replay and shadow before you enforce. Run the candidate settings over recorded production metrics or a traffic replay and count how often and when they would have opened, compared with the real incidents. Better still, deploy the breaker in observe-only mode (log and count the "would open" decisions without acting) for a week or two. Then enable it, per dependency and with a gradual rollout.
  7. Keep watching. Review the metrics that matter: the number of opens per instance (an open that corresponds to no incident is a false positive), the time from the start of an incident to the open, the time from recovery to the close, and the calls rejected while open. Traffic grows, dependencies change and thresholds go stale, so tune after incidents and after large changes in load.

5. Bulkheads

A ship is divided into watertight compartments so that a hole floods one of them and not the hull. A bulkhead gives each dependency (or each kind of work) its own pool of threads, connections or concurrency slots, so that a slow dependency can only exhaust its own pool.

I tested it with two dependencies: 30 calls to a slow one (0.5 s each) arrive together with 30 calls to a fast one (0.01 s), with 10 slots available.

SetupMedian latency of the fast callsSlowest fast call
One shared pool of 10 slots1523 ms1534 ms
Separate pools of 1021 ms31 ms

With a shared pool, the fast calls queue behind the slow ones and take about 70 times longer, although the fast dependency was perfectly healthy. With separate pools they are barely affected. The same idea applies at larger scale: separate connection pools per downstream, separate worker pools for interactive and batch work, and even separate deployments for important and unimportant tenants (see multi-tenant systems).

The cost is capacity you cannot share: ten idle slots in one pool cannot help the other. Size pools from the real concurrency each dependency needs (arrival rate times latency, from Little's law) and not from a round number.

6. Backpressure and load shedding

A service that accepts more work than it can finish does not process more; it builds a queue, and queued requests wait. Two ideas keep this under control.

Backpressure pushes the limit back to the sender: a bounded queue, a full pipe that makes write block, a 429 Too Many Requests or 503 with Retry-After, a consumer that pulls at its own speed instead of being pushed to. The sender then slows down or chooses what to do.

Load shedding is deliberately refusing some requests to protect the rest, for example when the queue is longer than a limit, or when the CPU is above a threshold. Rejecting early is cheap; accepting a request and then answering after the caller has timed out is the most expensive outcome.

I simulated a service that can do 10 requests per tick while 15 arrive per tick, for 60 ticks:

QueueRequests servedRejectedMedian waitLongest wait
Unbounded600015.6 ticks31 ticks
Limited to 206002901.5 ticks2 ticks

The service did the same amount of work (600 requests) in both cases. The unbounded queue did not help anyone: it only made everybody wait up to 31 ticks, probably longer than their timeouts, so much of that work would have been for callers who had gone. The bounded queue rejected 290 requests at once and served the rest in about two ticks. Rejecting some work is what keeps the rest useful. Choose what to shed: low-priority traffic first, requests that are already close to their deadline, or requests from the client that is sending the most. For the algorithms that limit request rates per client, see designing a rate limiter; and a queue between services is the subject of message queues.

7. Putting the pieces together

Diagram: Where each tool sits on a call

The order matters. Shedding and bulkheads decide whether to start work at all. The breaker decides whether to call. The timeout bounds the call. The retry policy sits outside the timeout and uses the breaker, so that when the breaker is open, a retry fails at once instead of waiting; and it stops when the request's deadline is used up.

Two more principles tie these together.

8. Common mistakes

9. Things to mention in an interview

Quick recap

Appendix: the experiment code

The numbers in this guide come from the three scripts below (Python 3, standard library only; the timing results use real asyncio sleeps, so the milliseconds vary slightly between runs, and the random choices use fixed seeds). The two animations are small deterministic models of a client and a server, not measurements: they show the shape of the behaviour, and the simulated capacity and the point at which an overloaded server stops serving anybody are assumptions of the model. The cancellation test uses a local TCP service on one machine, and its step counts are exact because the work is a fixed number of steps; the breaker simulation is a random model (20 seeded runs) of a constant background error rate, which real traffic does not have: errors cluster, and that is a reason to measure your own.

Timeouts, retries, jitter, bulkheads and queues

import asyncio, random, time, math

# ---- 1. timeouts: without one, a slow dependency ties up every worker
async def slow_dep(delay): await asyncio.sleep(delay)

async def handler(sem, delay, timeout):
    async with sem:                      # a worker slot
        try:
            await asyncio.wait_for(slow_dep(delay), timeout)
            return "ok"
        except asyncio.TimeoutError:
            return "timeout"

async def exp_timeout(timeout):
    sem = asyncio.Semaphore(10)          # 10 workers
    t0 = time.perf_counter()
    # the dependency now takes 1.0 s; 50 requests arrive at once
    res = await asyncio.gather(*[handler(sem, 1.0, timeout) for _ in range(50)])
    return round(time.perf_counter() - t0, 2), res.count("ok"), res.count("timeout")

# ---- 2. retries multiply load: expected attempts when each attempt fails with probability p
def attempts(p, retries):  # 1 + p + p^2 ... (retries extra tries)
    return sum(p**i for i in range(retries + 1))

# ---- 3. backoff schedules
def schedule(kind, n, base=0.1, cap=10, rnd=None):
    out = []
    for k in range(n):
        ceiling = min(cap, base * 2**k)
        out.append({"fixed": base, "exp": ceiling, "full": rnd.uniform(0, ceiling),
                    "equal": ceiling/2 + rnd.uniform(0, ceiling/2)}[kind])
    return out

def collisions(kind, clients=1000, n=6, bucket=0.05):
    rnd = random.Random(7)
    worst = []
    times = [0.0]*clients
    for k in range(n):
        for c in range(clients):
            times[c] += schedule(kind, k+1, rnd=rnd)[k]
        counts = {}
        for t in times: counts[int(t/bucket)] = counts.get(int(t/bucket), 0) + 1
        worst.append(max(counts.values()))
    return worst

# ---- 4. bulkhead: one slow dependency, two pools vs one shared pool
async def call(pool, delay):
    async with pool:
        await asyncio.sleep(delay)

async def exp_bulkhead(shared):
    pools = {"slow": asyncio.Semaphore(10), "fast": asyncio.Semaphore(10)}
    if shared: pools["fast"] = pools["slow"]
    t0 = time.perf_counter()
    # 30 slow calls (0.5 s) arrive together with 30 fast calls (0.01 s)
    fast_lat = []
    async def fast():
        s = time.perf_counter(); await call(pools["fast"], 0.01); fast_lat.append(time.perf_counter()-s)
    tasks = [asyncio.create_task(call(pools["slow"], 0.5)) for _ in range(30)]
    await asyncio.sleep(0)
    tasks += [asyncio.create_task(fast()) for _ in range(30)]
    await asyncio.gather(*tasks)
    fast_lat.sort()
    return round(fast_lat[len(fast_lat)//2]*1000), round(fast_lat[-1]*1000)

# ---- 5. load shedding / queue: Little's law, bounded queue vs unbounded
def queue_sim(limit, ticks=60, arrivals=15, service=10):
    q = 0; waits = []; shed = 0; served = 0
    for t in range(ticks):
        for _ in range(arrivals):
            if limit is not None and q >= limit: shed += 1
            else: q += 1; waits.append(q/service)   # ticks until this one is served
        s = min(q, service); q -= s; served += s
    waits.sort()
    return served, shed, round(waits[len(waits)//2],1), round(waits[-1],1)

if __name__ == "__main__":
    for to in (None, 0.2):
        print("timeout", to, asyncio.run(exp_timeout(to)))
    for p in (0.1, 0.5, 0.9):
        print("p", p, "3 retries ->", round(attempts(p, 3), 2), "attempts per request")
    for kind in ("fixed","exp","full","equal"):
        print(kind, "peak clients in one 50 ms bucket per round:", collisions(kind))
    print("bulkhead shared  (median, max fast ms):", asyncio.run(exp_bulkhead(True)))
    print("bulkhead isolated(median, max fast ms):", asyncio.run(exp_bulkhead(False)))
    for lim in (None, 20):
        print("queue limit", lim, "(served, shed, median wait, max wait):", queue_sim(lim))

Cancellation: does the server keep working after the caller gave up?

"""When the caller gives up, does the server keep working? A tiny TCP service whose request takes 1 s (20 steps of 50 ms).
Clients time out after 0.2 s and close the connection. Variant A ignores that; variant B checks whether the client is
still connected before each step (cancellation propagation). Standard library only."""
import asyncio

STEPS, STEP_S, CLIENTS, TIMEOUT = 20, 0.05, 20, 0.2

async def run(check_client):
    steps_done = 0
    async def handle(reader, writer):
        nonlocal steps_done
        await reader.readline()                      # the request
        for _ in range(STEPS):
            if check_client and reader.at_eof():     # the client has hung up: stop working
                break
            await asyncio.sleep(STEP_S)
            steps_done += 1
        try: writer.write(b"done\n"); await writer.drain()
        except Exception: pass
        writer.close()
    server = await asyncio.start_server(handle, "127.0.0.1", 0)
    port = server.sockets[0].getsockname()[1]
    async def client():
        r, w = await asyncio.open_connection("127.0.0.1", port)
        w.write(b"req\n"); await w.drain()
        try: await asyncio.wait_for(r.readline(), TIMEOUT)
        except asyncio.TimeoutError: pass
        w.close()                                    # give up
    await asyncio.gather(*[client() for _ in range(CLIENTS)])
    await asyncio.sleep(STEPS * STEP_S + 0.3)        # let the server finish whatever it is still doing
    server.close()
    return steps_done

for check in (False, True):
    done = asyncio.run(run(check))
    total = CLIENTS * STEPS
    print(f"server {'checks whether the client is still there' if check else 'ignores the closed connection         '}: {done:>3} of {total} work steps executed ({done/total*100:.0f}%), "
          f"{CLIENTS} callers had given up after {TIMEOUT} s")

A breaker that opens by chance

"""A breaker that opens after 3 failures in a row, on a dependency that is healthy except for a steady background error rate.
Counts how often it opens by chance alone, averaged over 20 random runs of one simulated hour each."""
import random
for p, rate in ((0.02, 100), (0.005, 100), (0.02, 5)):
    calls = 3_600 * rate; total = 0
    for seed in range(20):
        r = random.Random(seed); run = 0
        for _ in range(calls):
            if r.random() < p:
                run += 1
                if run == 3: total += 1; run = 0   # the breaker opens, then counting starts again from zero
            else: run = 0
    print(f"background error rate {p*100:.1f}%, {rate:>3} calls/s: opens {total/20:.1f} times per hour by chance alone (calculation: {calls * (1-p) * p**3:.1f})")

Sources

Practise this topic: System Design System Design › Reliability and scale

More guides