CS Fundamentals Trainer · Guides
Capacity estimation: load, storage, bandwidth and memory on the back of an envelope
The constants worth memorising, what one small machine really does (measured), a seven-step method with Littles law, worked examples for a URL shortener, photo sharing and chat, how long data takes to move, and the mistakes that make estimates wrong by ten.
34 min read
"How many users? How many requests per second? How much data?" Those are the first questions in almost every design discussion, and they are not small talk. A system for a hundred requests a day and a system for a hundred million need different architectures: the first fits on one cheap server, and the second needs caches, partitioned data and a CDN. Capacity estimation (the "back-of-the-envelope calculation") is the skill of getting from a few stated assumptions to the order of magnitude of load, storage, bandwidth and memory, quickly enough to do it out loud, and accurately enough to choose between designs.
It is not about precision. If the answer is 3 TB, nobody cares whether it is 2.7 or 3.4. They care whether it is 3 TB or 3 PB, because that is the difference between "one database" and "a distributed storage system". This guide gives you the constants worth memorising, a method that works every time, three worked examples, measured numbers for what one small machine can actually do, and the mistakes that make estimates wrong by a factor of ten.
A note on measurements. The numbers marked "measured" were produced for this guide on a small cloud VM (2 vCPUs of an Intel Xeon at 2.1 GHz, 8 GB of memory, Linux, Python 3.13, Go 1.24, SQLite 3.45); the code is in the appendix. Your hardware will differ, sometimes by a factor of ten, which is itself a lesson: use such numbers to know whether you are in the right range, and measure your own system before you rely on them. The arithmetic in the worked examples comes from the calculator in the appendix, so every figure can be checked.
1. What an estimate is for
An estimate answers a question about the design:
| If the estimate says… | …the design question is |
|---|---|
| a few hundred requests per second | does a single well-built service already do it? (usually yes) |
| tens of thousands per second | a stateless tier behind a load balancer, plus a cache and read replicas |
| writes beyond what one database commits | batching, queues, or splitting the data across nodes |
| data in the terabytes | one node with good backups, or sharding? How long does a restore take? |
| data in the petabytes, or bandwidth in the hundreds of Gbit/s | object storage, tiering, and a CDN |
| millions of open connections | a dedicated connection tier; memory per connection becomes the constraint |
The estimate also finds the dominant constraint. Every system is limited by one thing first: queries per second (a search box), storage (photos), bandwidth (video), memory (a chat connection tier) or write rate (metrics). Knowing which one saves you from designing a beautiful solution to the wrong problem.
2. The numbers worth memorising
Time
| Seconds | Memorise as | |
|---|---|---|
| 1 day | 86,400 | about 10^5 (round to 100,000 when you need speed) |
| 1 month (30 days) | 2,592,000 | about 2.6 million |
| 1 year | 31,536,000 | about 3 × 10^7 |
Two conversions are worth knowing by heart: 1 million events a day is about 12 per second (1,000,000 ÷ 86,400 ≈ 11.6), and so 100 million a day is about 1,200 per second. Almost every "per day" figure in a problem needs turning into "per second", and these two cover most cases.
Sizes
Powers of two are close to powers of ten, which makes the arithmetic easy: 2^10 = 1,024 (about a thousand, 1 KB), 2^20 ≈ a million (1 MB), 2^30 ≈ a billion (1 GB), 2^40 ≈ a trillion (1 TB), 2^50 ≈ a quadrillion (1 PB). Typical item sizes: a character is 1 byte (more in UTF-8 for non-ASCII), a UUID is 16 bytes binary or 36 as text, a tweet-sized message with metadata about 200 to 500 bytes, a web page 1 to 3 MB, a phone photo 2 to 5 MB, a minute of compressed HD video tens of MB.
Bits are not bytes. Network speeds are quoted in bits per second, storage in bytes: a 1 Gbit/s link moves at most 125 MB/s, and 10 Gbit/s at most 1.25 GB/s. Mixing them up is a factor-of-eight error.
Availability
The "nines" translate to downtime you are allowed:
| Availability | Per year | Per 30-day month |
|---|---|---|
| 99% | 87.6 hours | 7.2 hours |
| 99.9% | 8.8 hours | 43.2 minutes |
| 99.99% | 52.6 minutes | 4.3 minutes |
| 99.999% | 5.3 minutes | 26 seconds |
A system made of several parts in series is only as available as the product of its parts: three components at 99.9% each give about 99.7%. That is why a requirement of four nines rules out a single database in a single zone, whatever else you do.
Latency
The classic list "Latency Numbers Every Programmer Should Know", credited to Jeff Dean and compiled by Peter Norvig (the widely copied version is dated about 2012), gives the shape of the hierarchy:
| Operation | Time (circa 2012) |
|---|---|
| L1 cache reference | 0.5 ns |
| Main memory reference | 100 ns |
| Compress 1 KB | 3 µs |
| Send 1 KB over a 1 Gbit/s network | 10 µs |
| Read 4 KB randomly from SSD | 150 µs |
| Read 1 MB sequentially from memory | 250 µs |
| Round trip within the same datacenter | 500 µs |
| Read 1 MB sequentially from SSD | 1 ms |
| Disk seek | 10 ms |
| Read 1 MB sequentially from a spinning disk | 20 ms |
| Packet from California to the Netherlands and back | 150 ms |
Treat the individual values as outdated but the ratios as durable: memory is about a thousand times faster than a disk seek, a datacenter round trip is hundreds of times slower than a memory access, and a round trip across an ocean is a hundred or so milliseconds that no engineering can remove (the speed of light is the limit). Those ratios are what tell you to cache, to batch small operations and to put data near users.
What one machine does (measured)
The same numbers measured on the VM described above:
| Operation | Measured |
|---|---|
SHA-256, one core (Python's hashlib) | about 1.2 GB/s |
| JSON encode / decode of a 620-byte object, Python | about 190,000 / 120,000 per second |
| zlib compression, JSON text, level 1 / level 6 | about 51 / 18 MB/s, output about 48% / 43% of the input |
| Sequential read from the page cache | about 7 GB/s |
| Sequential read after dropping the cache | about 2.9 GB/s |
| Random 4 KB read after dropping the cache | about 37 µs each |
write of 4 KB followed by fsync | about 0.12 ms, about 8,400 per second |
| Loopback TCP round trip, 1 byte, Python at both ends | about 37 µs |
| Opening and closing a local TCP connection | about 74 µs |
Go net/http "hello world", 2 vCPUs shared with the load generator | 25,000 requests/s with 1 client, about 40,000 to 43,000 with 8 to 256 clients |
| Latency of that server at 64 concurrent clients | p50 1.3 ms, p99 5.1 ms |
| Memory per idle TCP connection in Go (a goroutine and a 4 KB buffer) | about 4.6 KB of process memory |
On the same machine, SQLite with a table of one million rows (an id, a number and 100 bytes of text, with two indexes, about 130 bytes per row on disk):
| Operation | Measured |
|---|---|
| Bulk load in one transaction | about 205,000 rows/s |
| Primary-key lookups, one connection, warm | about 200,000 per second |
| Lookups through a secondary index (about 10 rows each) | about 154,000 per second |
One full scan with a LIKE '%…%' filter | about 119 ms for 1,000,000 rows |
One insert per commit, synchronous=FULL | about 4,400 commits/s (0.23 ms each) |
One insert per commit, WAL with synchronous=NORMAL | about 27,000 commits/s |
Read these carefully, because they hold several of the lessons of this guide:
- A cheap machine does a lot, if the work is simple. Forty thousand trivial requests a second on two shared CPUs means that a small service is rarely bottlenecked by the web framework. It is bottlenecked by what each request does: a handler that queries a database, calls other services and renders a response is a hundred times heavier than an empty one. The honest figure for planning is hundreds to a few thousand requests per second per server for typical work, and you find your own by load testing.
- Durability dominates write rates. The same table commits 4,400 independent transactions a second when every commit waits for the disk and 27,000 when it does not, and a bulk load (many rows per transaction) reaches 205,000. "How many writes per second can a database take?" has no single answer; it depends on batching and on how strict the durability is.
- An index is the difference between microseconds and a scan. The lookup through an index costs 5 to 7 µs; scanning the same table for a substring costs 119 ms, about 20,000 times more.
- Absolute numbers vary by hardware, and so does the ranking. The 37 µs cold random read is far below the classic 150 µs, because a modern local NVMe disk (or the cache of the host behind a virtual one) is much faster than the 2012 SSD. A network-attached volume in a cloud can be slower than both. This is why the table above is a calibration, not a specification.
3. The method
The same seven steps work for any system. Write down each assumption and its unit, and round aggressively.
- Clarify the scope and the mix. What does the product do, for how many users, with what read/write ratio, what retention, what quality targets? Ask, and when you cannot, state your assumption out loud. A wrong assumption is acceptable; a hidden one is not.
- Traffic.
DAU × actions per user per day ÷ 86,400gives the average requests per second. Multiply by a peak factor (2 to 3 for a steady consumer product, 5 to 10 or more for spiky ones such as ticket sales or news) to get the peak. Size for the peak, not the average. - Storage.
records × size per record × retention, then multiply for replication (typically 3 copies, or about 1.5 with erasure coding for cold objects) and for overhead (indexes, row headers and free space: databases are rarely run above 70 to 80% full). In my SQLite test the data and two indexes took 130 bytes for a row with about 105 bytes of fields, so count on a noticeable overhead; measure it for your schema. - Bandwidth.
requests per second × bytes per responsefor outgoing traffic, and the same for incoming. Convert to bits and compare against what a network interface carries (1, 10 or 25 Gbit/s). - Memory. For caches, estimate the working set (the data that is actually hot) rather than the whole data set. A common assumption is that 20% of the items receive 80% of the reads. That is a rule of thumb and not a law; real access logs often show a different skew, so check them when you have them. For connection tiers,
connections × memory per connection. - Servers.
peak load ÷ capacity of one server, where the capacity is a measured number and is derated to a target utilisation (50 to 70% leaves room for spikes, deploys and failures), and add redundancy (N+1, or N+2 for large pools, or enough to survive the loss of a whole zone). - Sanity-check and find the bottleneck. Cross-check with a second route (for example, bottom-up from the machines you would need against top-down from the users), and name the resource that runs out first.
Little's law: how many requests are in flight
One formula is worth keeping beyond the seven steps. Little's law says that the average number of items in a stable system equals the arrival rate times the average time each spends in it: L = λ × W. For a service, that is requests in flight = requests per second × seconds per request.
| Rate | Time per request | Requests in flight |
|---|---|---|
| 1,000 per second | 50 ms | 50 |
| 12,000 per second | 50 ms | 600 |
| 12,000 per second | 500 ms | 6,000 |
| 100,000 per second | 20 ms | 2,000 |
It tells you how many concurrent connections, threads or database connections a given load needs, and it shows the cruel effect of slowness: if the same service gets ten times slower, ten times as many requests pile up in it, so the memory, threads and connection-pool slots it holds also grow ten times, which is how a slowdown becomes an outage.
4. Worked example 1: a URL shortener
Assumptions. 100 million new short links a month, 100 reads for every write, 500 bytes stored per link (the short code, the long URL, a creation time and a few counters), links kept 5 years, 3 copies, peak at 3 times the average.
| Quantity | Calculation | Result |
|---|---|---|
| Writes per second, average | 100,000,000 ÷ (30 × 86,400) | 39 per second |
| Writes per second, peak | × 3 | about 120 |
| Reads per second, average | × 100 | about 3,900 |
| Reads per second, peak | × 3 | about 11,600 |
| Links after 5 years | 100 M × 12 × 5 | 6 billion |
| Raw storage | 6 × 10^9 × 500 B | 3 TB |
| With 3 copies | × 3 | 9 TB |
| Reads per day | 3,858 × 86,400 | 333 million |
| Cache for the hot links | assume 1 distinct link per 100 reads: 3.3 M distinct per day, the hottest 20% of them, at 500 B | about 333 MB |
| App servers | 11,600 ÷ 5,000 per server = 3, plus one spare | 4 |
What it tells you. The write rate is tiny (tens per second), so write scaling is not the problem. The read rate is moderate, and the hot data fits in a few hundred MB of cache, so a cache removes most database reads. Storage, 3 TB for the primary copy, fits on one modern database server's disks, so partitioning is optional at first and becomes a growth question, not a day-one requirement. The interesting design work is elsewhere (generating unique short codes without collisions, and keeping redirects fast and available). The estimate has told you where the effort should not go.
5. Worked example 2: photo sharing
Assumptions. 50 million daily active users; 10% post one photo a day; each upload is a 2 MB original plus 0.2 MB of thumbnails; each user views 50 photos a day at an average of 300 KB each (feed-sized images).
| Quantity | Calculation | Result |
|---|---|---|
| Uploads per day | 50 M × 10% | 5 million (58 per second on average) |
| New storage per day | 5 M × 2.2 MB | 11 TB |
| New storage per year | × 365 | 4 PB |
| With 3 full copies | × 3 | 12 PB per year |
| With erasure coding (about 1.5×) | × 1.5 | 6 PB per year |
| Outgoing data per day | 50 M × 50 × 300 KB | 750 TB |
| Average outgoing bandwidth | 750 TB × 8 ÷ 86,400 | 69 Gbit/s |
| At 3× peak | × 3 | about 208 Gbit/s |
| 10 Gbit/s links needed | average, peak | 7, 21 |
What it tells you. The request rate (58 uploads a second) is trivial. Storage (petabytes a year) and above all bandwidth (hundreds of Gbit/s at peak) dominate. No origin server farm should serve that: the images belong in object storage, delivered by a CDN that absorbs most of the traffic near users, with tiering that moves old photos to cheaper storage. Compare with the previous example, where the same method pointed at a cache and a modest database. The method is the same, and the conclusion is entirely different.
6. Worked example 3: chat
Assumptions. 100 million daily active users, each sending 40 messages a day; 200 bytes stored per message (text and metadata); 20% of the users online at peak; each online user keeps one open connection costing about 8 KB in total; a gateway holds up to 100,000 connections.
| Quantity | Calculation | Result |
|---|---|---|
| Messages per day | 100 M × 40 | 4 billion |
| Messages per second, average | ÷ 86,400 | about 46,000 |
| Messages per second, peak | × 3 | about 139,000 |
| Storage per day | 4 B × 200 B | 800 GB |
| Storage per year | × 365 | 292 TB (876 TB with 3 copies) |
| Connections at peak | 20% of 100 M | 20 million |
| Memory to hold the connections | 20 M × 8 KB | 160 GB |
| Gateways | 20 M ÷ 100,000 | 200 (plus spare capacity) |
| If one gateway fails | 100,000 clients reconnect; over 30 s | about 3,300 connections per second on the rest |
The 8 KB per connection is a measured figure: holding 18,000 idle connections in a Go server took about 4.6 KB of process memory each (a goroutine and a 4 KB read buffer); the whole machine used about 12 KB per connection counting both the client and server ends of each one, which leaves roughly 4 KB of kernel memory per end (an estimate: I assumed both ends cost the same). A real chat connection also carries per-user state, so 8 KB is a floor.
What it tells you. The message rate is high but ordinary for a partitioned log or a sharded database. The distinctive constraint is connections: 20 million of them, 160 GB of memory just to hold them open, spread over hundreds of machines. That is the reason chat designs have a separate gateway tier that only holds connections, and a lookup of "which gateway holds this user". It is also why a failing gateway matters: its 100,000 clients reconnect at once, so reconnects need jitter and the remaining gateways need headroom. See the guide on designing a chat system for how those parts fit together.
7. Moving data takes time
Capacity is not only "does it fit" but also "how long does it take to copy". A new replica, a restore from backup or a migration to new hardware has to move the whole data set:
| Data | At 1 Gbit/s | At 10 Gbit/s |
|---|---|---|
| 1 TB | 2.2 hours | 13 minutes |
| 10 TB | 22 hours | 2.2 hours |
| 100 TB | 9.3 days | 22 hours |
(Assuming the link is fully used, which it never quite is.) These are the numbers behind advice such as "keep each database node to a few terabytes". Past that, a failed replica takes a day to rebuild, during which you are running with less redundancy, and a restore after a disaster is measured in days. When an estimate gives you tens of terabytes in one place, ask not only whether it fits but also how you would rebuild it.
8. Rules of thumb and traps
- Peak is not average. Traffic follows the day, the week and events. A product that averages 1,000 requests a second may see 3,000 at the evening peak and 10,000 in a launch. Say which one you are sizing.
- Reads and writes are different problems. Reads scale with caches and replicas; writes need partitioning or batching. The ratio (100:1 for a shortener, 1:1 for a chat, write-heavy for metrics) decides which one is hard.
- Fan-out multiplies. One post by an account with a million followers is one write and a million feed updates if you push on write. Estimate the fan-out, not just the number of posts.
- Replication and overhead are multipliers, not footnotes. Three copies, indexes, free space and backups can turn 3 TB into 15 TB.
- Growth. Estimate for the retention period and for one to three years of user growth, and say what happens when the estimate is exceeded.
- Averages hide the tail. If one request in a hundred is ten times slower, a user who makes fifty requests a page sees a slow one on about 40% of pages (1 − 0.99^50). Latency targets are percentiles, and the arithmetic of fan-out makes the tail more important, not less.
- Utilisation is not capacity. A server driven to 100% has no room for a spike, a deploy or a lost neighbour, and queues grow sharply as utilisation approaches 100%. Plan for 50 to 70%.
- Failure domains. Capacity is what is left when something fails. Ask whether the system still carries the peak after losing a server, a rack, or a zone.
- Measure, then trust. Every per-server figure in an estimate is a guess until it is measured under realistic load. The measured figures here show how far a "hello world" is from a real handler.
9. Presenting an estimate
In an interview or a design review, the estimate is also communication.
- State assumptions first, with units: "I'll assume 100 million daily users, 40 messages each, 200 bytes a message."
- Round to make the arithmetic easy (86,400 becomes 100,000 when you need to do it in your head) and say that you rounded. One significant digit is enough.
- Go from small to large and show the result's meaning: "That's about 46,000 messages a second on average, so roughly 140,000 at peak, which one database will not take but a partitioned log will."
- Finish with the design consequence, not the number: which constraint dominates and what you will therefore do.
- Cross-check: "From the machine side, 200 gateways at 100,000 connections each gives the same 20 million."
- Invite correction. If the interviewer changes an assumption, redo the one line it affects. That is the point of writing assumptions down.
10. Common mistakes
- Estimating before understanding the requirements, so the numbers answer a question nobody asked.
- Using the average where the peak matters, or the reverse.
- Mixing bits and bytes, or MB and MiB when the precision matters, or months of 30 and 31 days where it does not.
- Forgetting replication, indexes, free space and retention when sizing storage.
- Counting users instead of concurrent users, or concurrent users instead of requests.
- Assuming a server's capacity without a measurement or a clear source, then building a plan on top of it.
- False precision: "47,917 requests per second" tells the listener the speaker does not understand the uncertainty of the inputs.
- Stopping at the number and never saying what it means for the design.
- Over-engineering for an imaginary scale or under-engineering for a real one. The estimate is there to settle which.
11. Things to mention in an interview
- Why you estimate: to find the dominant constraint and decide between designs, not to be exact.
- The constants: 86,400 seconds in a day (1 million a day ≈ 12 per second), powers of two and ten, bits versus bytes, the nines of availability.
- The latency hierarchy and its ratios; what a cache, a CDN and batching change.
- The seven steps: scope, traffic with a peak factor, storage with replication and overhead, bandwidth, memory/working set, servers with headroom, sanity check.
- Little's law for concurrency, and the effect of a slowdown on in-flight requests.
- That data movement (rebuilds, restores) has a time cost that bounds node size.
- That estimates should be cross-checked, rounded, stated with assumptions, and followed by a design consequence.
Quick recap
- An estimate decides the architecture by finding the dominant constraint (requests, storage, bandwidth, memory, connections or write rate). Precision is not the goal; the order of magnitude is.
- Memorise 86,400 s per day (1 M per day ≈ 12 per second), the powers of two, bits versus bytes, the nines, and the latency ratios. Measured on a small VM: about 40,000 trivial HTTP requests a second, 200,000 indexed lookups a second, 4,400 fsynced commits a second, and about 4.6 KB of memory per idle connection.
- The method: scope → traffic (× peak factor) → storage (× replicas × overhead × retention) → bandwidth → memory → servers (÷ target utilisation, + spare) → sanity check.
- The same method gave three different answers: the URL shortener (3 TB, 4 servers: a cache and a database), photo sharing (4 PB a year, 208 Gbit/s peak: object storage and a CDN) and chat (20 million connections, 160 GB: a connection tier of about 200 gateways).
- Little's law (
L = λW) gives requests in flight; copy time (10 TB takes 22 hours at 1 Gbit/s) bounds how big a node can be. - State assumptions with units, round, cross-check, and end with what the number means for the design.
Appendix: the code
The numbers come from four programs. The calculator estimate.py prints every worked example above. bench.py produced the CPU, disk, SQLite and loopback figures (run python3 bench.py cpu disk db net; it needs a few GB of free disk space and tries to drop the page cache, which needs root). The two Go programs measured the HTTP rate and the cost of idle connections (build each with go build; start conns_bin server in one terminal and conns_bin client 18000 in another, with the file-descriptor limit raised if you want more than about 18,000 connections).
estimate.py
"""Back-of-the-envelope calculator for the capacity-estimation guide. Every number printed in the worked examples comes from here."""
import math
DAY = 86_400
def fmt(n, unit=""):
for div, name in ((1e15, "P"), (1e12, "T"), (1e9, "G"), (1e6, "M"), (1e3, "k")):
if abs(n) >= div: return f"{n / div:.3g} {name}{unit}"
return f"{n:.3g} {unit}".strip()
def rule(title): print(f"\n== {title}")
def constants():
rule("time")
for name, s in (("minute", 60), ("hour", 3600), ("day", DAY), ("month (30 days)", 30 * DAY), ("year", 365 * DAY)):
print(f" {name:16s} {s:>12,} s ~ {s:.1e}")
rule("availability: downtime allowed per year and per 30-day month")
for nines in (0.99, 0.999, 0.9999, 0.99999):
y = (1 - nines) * 365 * DAY; m = (1 - nines) * 30 * DAY
h = lambda s: f"{s / 3600:.1f} h" if s >= 3600 else (f"{s / 60:.1f} min" if s >= 60 else f"{s:.0f} s")
print(f" {nines * 100:g}% {h(y):>9} per year {h(m):>9} per month")
rule("powers of two")
for e, n in ((10, "KB"), (20, "MB"), (30, "GB"), (40, "TB"), (50, "PB")):
print(f" 2^{e} = {2 ** e:>20,} ~ 10^{e * 0.30103:.1f} (1 {n})")
def littles_law():
rule("Little's law: requests in flight = rate x time in the system")
for rps, ms in ((1_000, 50), (12_000, 50), (12_000, 500), (100_000, 20)):
print(f" {rps:>7,} requests/s x {ms:>3} ms = {rps * ms / 1000:>7,.0f} requests in flight")
def example_url_shortener():
rule("Example 1: URL shortener")
new_per_month, read_write, record, years, replicas, peak = 100e6, 100, 500, 5, 3, 3
w = new_per_month / (30 * DAY); r = w * read_write
print(f" writes: {new_per_month:,.0f} / month = {w:.1f} per second on average; peak x{peak} = {w * peak:.0f} per second")
print(f" reads: x{read_write} = {r:,.0f} per second on average; peak = {r * peak:,.0f} per second")
total = new_per_month * 12 * years
raw = total * record
print(f" links after {years} years: {total:,.0f}; {record} bytes each = {fmt(raw, 'B')} raw, {fmt(raw * replicas, 'B')} with {replicas} copies")
reads_day = r * DAY
print(f" reads per day: {reads_day:,.0f}; if 20% of links get 80% of the reads, cache the hot 20% of one day's links:")
print(f" assume 1 distinct link per 100 reads -> {reads_day / 100:,.0f} distinct per day; 20% of that = {reads_day / 100 * 0.2:,.0f} x {record} B = {fmt(reads_day / 100 * 0.2 * record, 'B')} of cache")
per_server = 5_000
print(f" app servers: peak {r * peak:,.0f} / {per_server:,} per server = {math.ceil(r * peak / per_server)}, plus one spare = {math.ceil(r * peak / per_server) + 1}")
def example_photos():
rule("Example 2: photo sharing")
dau, post_share, orig, thumb, views, avg_view = 50e6, 0.10, 2e6, 0.2e6, 50, 0.3e6
posts = dau * post_share
per_day = posts * (orig + thumb)
print(f" uploads: {dau:,.0f} DAU x {post_share:.0%} post once = {posts:,.0f} photos/day = {posts / DAY:.0f} per second on average")
print(f" storage: {posts:,.0f} x ({orig / 1e6:g} MB + {thumb / 1e6:g} MB thumbnails) = {fmt(per_day, 'B')} per day, {fmt(per_day * 365, 'B')} per year")
print(f" with 3 full copies {fmt(per_day * 365 * 3, 'B')} per year; with erasure coding ~1.5x: {fmt(per_day * 365 * 1.5, 'B')} per year")
egress = dau * views * avg_view
print(f" reads: {dau:,.0f} x {views} views x {avg_view / 1e3:.0f} KB = {fmt(egress, 'B')} per day = {egress * 8 / DAY / 1e9:.0f} Gbit/s on average, {egress * 8 / DAY / 1e9 * 3:.0f} Gbit/s at 3x peak")
print(f" one 10 Gbit/s server link carries {10 / 8:.2f} GB/s; the average needs {egress * 8 / DAY / 1e9 / 10:.0f} of them, the peak {egress * 8 / DAY / 1e9 * 3 / 10:.0f}")
def example_chat():
rule("Example 3: chat")
dau, msgs, size, online, bytes_conn, per_gw = 100e6, 40, 200, 0.2, 8_000, 100_000
per_day = dau * msgs; rate = per_day / DAY
print(f" messages: {dau:,.0f} x {msgs} = {per_day:,.0f} per day = {rate:,.0f} per second on average, {rate * 3:,.0f} at 3x peak")
print(f" storage: {per_day:,.0f} x {size} B = {fmt(per_day * size, 'B')} per day, {fmt(per_day * size * 365, 'B')} per year, x3 copies = {fmt(per_day * size * 365 * 3, 'B')}")
conns = dau * online
print(f" connections at peak: {online:.0%} of DAU = {conns:,.0f}; at {bytes_conn / 1e3:g} KB each = {fmt(conns * bytes_conn, 'B')} of memory just to hold them")
print(f" gateways: {conns:,.0f} / {per_gw:,} per gateway = {math.ceil(conns / per_gw)} (plus spare capacity so one lost gateway's {per_gw:,} reconnects can be absorbed)")
print(f" if a gateway fails, {per_gw:,} clients reconnect at once; spread over 30 s that is {per_gw / 30:,.0f} connections per second to the rest")
def copy_time():
rule("how long it takes to move data (a new replica, a restore, a migration)")
for size_tb in (1, 10, 100):
row = []
for name, gbit in (("1 Gbit/s", 1), ("10 Gbit/s", 10)):
secs = size_tb * 1e12 / (gbit * 1e9 / 8)
row.append(f"{name}: {secs / 3600:6.1f} h" if secs < 3 * DAY else f"{name}: {secs / DAY:6.1f} days")
print(f" {size_tb:>4} TB " + " ".join(row) + " (link fully used, no overhead)")
def headroom():
rule("headroom: how many servers for a load, at a target utilisation")
peak, per_server = 12_000, 4_000
for util in (1.0, 0.7, 0.5):
n = math.ceil(peak / (per_server * util))
print(f" peak {peak:,} rps, {per_server:,} rps per server at 100% -> at {util:.0%} target utilisation: {n} servers; with N+1: {n + 1}; losing one of {n + 1} leaves {(n) * 100 / (n + 1):.0f}% of the pool")
if __name__ == "__main__":
constants(); littles_law(); example_url_shortener(); example_photos(); example_chat(); copy_time(); headroom()
bench.py
"""Calibration numbers for the capacity-estimation guide: what one small machine does. Standard library only."""
import hashlib, json, os, random, socket, sqlite3, threading, time, uuid, zlib
D = "/var/tmp/g3/bench"
os.makedirs(D, exist_ok=True)
def timeit(f, n):
t = time.perf_counter(); f(n); return time.perf_counter() - t
def part_cpu():
print("CPU-bound work in one Python thread (the interpreter is slower than C or Go, so read these as floors)")
buf = os.urandom(1 << 20)
n = 200; dt = timeit(lambda n: [hashlib.sha256(buf).digest() for _ in range(n)], n)
print(f" SHA-256 of 1 MB blocks: {n / dt:7.0f} MB/s")
random.seed(5)
words = [hashlib.md5(str(i).encode()).hexdigest()[:random.randint(3, 9)] for i in range(3000)]
def make():
return {"id": str(uuid.UUID(int=random.getrandbits(128), version=4)), "user": random.randrange(10**7),
"text": " ".join(random.choice(words) for _ in range(60)), "tags": random.sample(words, 3), "n": [random.randrange(1000) for _ in range(20)]}
doc = make(); raw = json.dumps(doc); n = 100_000
print(f" JSON object of {len(raw)} bytes: {n / timeit(lambda n: [json.dumps(doc) for _ in range(n)], n):7.0f} encodes/s, "
f"{n / timeit(lambda n: [json.loads(raw) for _ in range(n)], n):7.0f} decodes/s")
text = ("\n".join(json.dumps(make()) for _ in range(3000))).encode()
for lvl in (1, 6):
t = time.perf_counter(); c = zlib.compress(text, lvl); dt = time.perf_counter() - t
print(f" zlib level {lvl} on JSON text: {len(text) / dt / 1e6:7.0f} MB/s, size {len(c) / len(text):.0%} of the original")
def part_disk():
print("Disk and memory")
p = f"{D}/seq.bin"; size = 1 << 30
with open(p, "wb") as f:
blk = os.urandom(1 << 20)
t = time.perf_counter()
for _ in range(size >> 20): f.write(blk)
f.flush(); os.fsync(f.fileno())
print(f" write 1 GB sequentially + fsync: {size / (time.perf_counter() - t) / 1e6:7.0f} MB/s")
for label in ("page cache (warm)", "after dropping caches (cold)"):
if "cold" in label:
try:
os.sync(); open("/proc/sys/vm/drop_caches", "w").write("3")
except Exception as e:
print(" (could not drop caches:", e, ")"); continue
t = time.perf_counter(); got = 0
with open(p, "rb", buffering=0) as f:
while True:
b = f.read(1 << 20)
if not b: break
got += len(b)
print(f" read 1 GB sequentially, {label:28s}: {got / (time.perf_counter() - t) / 1e6:7.0f} MB/s")
# random 4 KB reads, cold
try:
os.sync(); open("/proc/sys/vm/drop_caches", "w").write("3")
except Exception: pass
fd = os.open(p, os.O_RDONLY); n = 2000; random.seed(1)
t = time.perf_counter()
for _ in range(n): os.pread(fd, 4096, random.randrange(0, size - 4096) & ~4095)
dt = time.perf_counter() - t
print(f" random 4 KB reads, cold: {dt / n * 1e6:7.0f} us each ({n / dt:,.0f} per second, one at a time)")
os.close(fd); os.remove(p)
# fsync latency
fd = os.open(f"{D}/sync.bin", os.O_WRONLY | os.O_CREAT); n = 500
t = time.perf_counter()
for i in range(n): os.pwrite(fd, b"x" * 4096, 0); os.fsync(fd)
dt = time.perf_counter() - t; os.close(fd)
print(f" write 4 KB + fsync: {dt / n * 1e3:7.2f} ms each ({n / dt:,.0f} per second)")
def part_db():
print("SQLite, one table of 1,000,000 rows (id, user, 100-byte payload) with indexes on id and user")
p = f"{D}/db.sqlite"
for f in (p, p + "-wal", p + "-shm"):
if os.path.exists(f): os.remove(f)
c = sqlite3.connect(p, isolation_level=None); c.execute("PRAGMA journal_mode=WAL")
c.execute("CREATE TABLE t (id INTEGER PRIMARY KEY, user INTEGER, payload TEXT)")
c.execute("CREATE INDEX t_user ON t (user)")
pay = "p" * 100; N = 1_000_000
t = time.perf_counter(); c.execute("BEGIN")
c.executemany("INSERT INTO t (user, payload) VALUES (?, ?)", ((random.randrange(100_000), pay) for _ in range(N)))
c.execute("COMMIT"); dt = time.perf_counter() - t
size = os.path.getsize(p)
print(f" bulk load in one transaction: {N / dt:8,.0f} rows/s; file {size / 1e6:.0f} MB = {size / N:.0f} bytes per row (data + 2 indexes)")
n = 100_000; ids = [random.randrange(1, N) for _ in range(n)]
t = time.perf_counter()
for i in ids: c.execute("SELECT payload FROM t WHERE id=?", (i,)).fetchone()
print(f" primary-key lookups, warm: {n / (time.perf_counter() - t):8,.0f} per second, one connection")
n = 20_000; t = time.perf_counter()
for _ in range(n): c.execute("SELECT COUNT(*) FROM t WHERE user=?", (random.randrange(100_000),)).fetchone()
print(f" lookups via the user index: {n / (time.perf_counter() - t):8,.0f} per second (about {N // 100_000} rows each)")
n = 50; t = time.perf_counter()
for _ in range(n): c.execute("SELECT COUNT(*) FROM t WHERE payload LIKE '%zzz%'").fetchone()
dt = (time.perf_counter() - t) / n
print(f" one full scan (no index): {dt * 1e3:8.0f} ms for {N:,} rows")
# commits with fsync
c.execute("PRAGMA synchronous=FULL"); n = 300; t = time.perf_counter()
for _ in range(n): c.execute("INSERT INTO t (user, payload) VALUES (1, 'x')")
dt = time.perf_counter() - t
print(f" one insert per commit (synchronous=FULL): {n / dt:6,.0f} commits/s ({dt / n * 1e3:.2f} ms each)")
c.execute("PRAGMA synchronous=NORMAL"); t = time.perf_counter()
for _ in range(n * 10): c.execute("INSERT INTO t (user, payload) VALUES (1, 'x')")
dt = time.perf_counter() - t
print(f" one insert per commit (synchronous=NORMAL, WAL): {n * 10 / dt:6,.0f} commits/s")
def part_net():
print("Loopback TCP, one byte back and forth (this is the floor: a real network adds distance)")
srv = socket.socket(); srv.bind(("127.0.0.1", 0)); srv.listen()
def echo():
s, _ = srv.accept(); s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
while True:
b = s.recv(1)
if not b: return
s.sendall(b)
threading.Thread(target=echo, daemon=True).start()
c = socket.create_connection(srv.getsockname()); c.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
n = 20_000; t = time.perf_counter()
for _ in range(n): c.sendall(b"x"); c.recv(1)
dt = time.perf_counter() - t
print(f" round trip: {dt / n * 1e6:6.1f} us (includes Python's own overhead on both ends)")
srv2 = socket.socket(); srv2.bind(("127.0.0.1", 0)); srv2.listen(1024)
threading.Thread(target=lambda: [srv2.accept()[0].close() for _ in range(2000)], daemon=True).start()
t = time.perf_counter()
for _ in range(2000):
s = socket.create_connection(srv2.getsockname()); s.close()
print(f" opening and closing a connection: {(time.perf_counter() - t) / 2000 * 1e6:6.0f} us")
if __name__ == "__main__":
import sys
for p in (sys.argv[1:] or ["cpu", "disk", "db", "net"]): globals()["part_" + p]()
http/main.go
// A "hello world" HTTP server and a closed-loop client on the same machine: how many requests per second one small VM serves.
package main
import (
"fmt"
"io"
"net"
"net/http"
"runtime"
"sort"
"sync"
"sync/atomic"
"time"
)
func main() {
ln, _ := net.Listen("tcp", "127.0.0.1:0")
go http.Serve(ln, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { io.WriteString(w, "ok") }))
url := "http://" + ln.Addr().String() + "/"
for _, workers := range []int{1, 8, 64, 256} {
client := &http.Client{Transport: &http.Transport{MaxIdleConnsPerHost: workers, MaxConnsPerHost: workers}}
var n atomic.Int64
var mu sync.Mutex
var lat []time.Duration
stop := time.Now().Add(4 * time.Second)
var wg sync.WaitGroup
for i := 0; i < workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
var mine []time.Duration
for time.Now().Before(stop) {
t := time.Now()
resp, err := client.Get(url)
if err != nil {
continue
}
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
mine = append(mine, time.Since(t))
n.Add(1)
}
mu.Lock()
lat = append(lat, mine...)
mu.Unlock()
}()
}
wg.Wait()
sort.Slice(lat, func(i, j int) bool { return lat[i] < lat[j] })
fmt.Printf(" %3d concurrent clients: %7.0f requests/s, p50 %v, p99 %v\n", workers, float64(n.Load())/4, lat[len(lat)/2].Round(time.Microsecond), lat[len(lat)*99/100].Round(time.Microsecond))
client.CloseIdleConnections()
}
fmt.Println(" (GOMAXPROCS =", runtime.GOMAXPROCS(0), "; the client and the server share the same CPUs)")
}
conns/main.go
// Memory cost of holding many idle TCP connections, one goroutine per connection (the usual Go server shape).
// Run "server" in one process and "client N" in another.
package main
import (
"bufio"
"fmt"
"net"
"os"
"strconv"
"strings"
"time"
)
func meminfo(key string) int64 {
b, _ := os.ReadFile("/proc/meminfo")
for _, l := range strings.Split(string(b), "\n") {
if strings.HasPrefix(l, key+":") {
f := strings.Fields(l)
v, _ := strconv.ParseInt(f[1], 10, 64)
return v // kB
}
}
return 0
}
func rss() int64 {
b, _ := os.ReadFile("/proc/self/status")
for _, l := range strings.Split(string(b), "\n") {
if strings.HasPrefix(l, "VmRSS:") {
v, _ := strconv.ParseInt(strings.Fields(l)[1], 10, 64)
return v
}
}
return 0
}
const ports = 6
func main() {
if os.Args[1] == "server" {
base := int(meminfo("MemAvailable"))
r0 := rss()
var count int64
for p := 0; p < ports; p++ {
ln, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", 9100+p))
if err != nil {
panic(err)
}
go func() {
for {
c, err := ln.Accept()
if err != nil {
return
}
count++
go func() { // one goroutine and one 4 KB read buffer per connection
r := bufio.NewReaderSize(c, 4096)
r.ReadByte()
c.Close()
}()
}
}()
}
fmt.Println("server ready")
last := int64(-1)
for {
time.Sleep(2 * time.Second)
if count != last && count > 0 {
last = count
fmt.Printf("server: %6d connections, process RSS +%6.1f MB (%.2f KB each), system memory used +%6.1f MB (%.2f KB each)\n",
count, float64(rss()-r0)/1024, float64(rss()-r0)/float64(count),
float64(int64(base)-meminfo("MemAvailable"))/1024, float64(int64(base)-meminfo("MemAvailable"))/float64(count))
}
}
}
n, _ := strconv.Atoi(os.Args[2])
var keep []net.Conn
for i := 0; i < n; i++ {
c, err := net.Dial("tcp", fmt.Sprintf("127.0.0.1:%d", 9100+i%ports))
if err != nil {
fmt.Println("client stopped at", i, err)
break
}
keep = append(keep, c)
}
fmt.Println("client holds", len(keep), "connections")
time.Sleep(6 * time.Second)
}
Sources
- Jeff Dean, "Numbers Everyone Should Know" (Google), as compiled and updated in the community gist Latency Numbers Every Programmer Should Know, and Peter Norvig, Teach Yourself Programming in Ten Years (the original list).
- Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly), chapter 1, "Reliable, Scalable, and Maintainable Applications": describing load, percentiles and the effect of fan-out.
- John D. C. Little, "A Proof for the Queuing Formula: L = λW", Operations Research 9(3), 1961 (Little's law).
- Alex Xu, System Design Interview: An Insider's Guide (2020), chapter 2, "Back-of-the-envelope estimation".
- Donne Martin, The System Design Primer, appendix on powers of two, latency numbers and handy metrics.
- Wikipedia, High availability, the table of availability percentages and allowed downtime.
- Beyer, Jones, Petoff, Murphy (eds.), Site Reliability Engineering (O'Reilly, 2016), on service level objectives and headroom.
Practise this topic: System Design System Design › Reliability and scale