CS Fundamentals Trainer · Guides
Design a file storage and sync service (Dropbox-style)
A case study in large files and many devices: splitting files into content-addressed blocks, a metadata service separate from block storage, delta sync and deduplication, change notification to other devices, conflict handling, sharing, and the numbers that show why the block store, not the database, is the big cost.
4 min read
The problem. Users keep files in a folder on several devices. Changes on one device must appear on the others, quickly and without re-uploading whole files, even over poor connections; files can be gigabytes; and no change may ever be lost.
Requirements and numbers
Functional: upload, download, automatic sync across devices, version history, sharing, offline edits that sync later. Non-functional: reliability and durability above all (losing a file is the worst failure), efficient bandwidth, eventual consistency across devices with clear conflict handling.
Assume 50 million users with 10 GB stored on average: 50,000,000 x 10 GB = 500 PB of logical data. With deduplication and compression it may be less, but it is petabytes either way. Metadata is tiny in comparison: 10 billion files of 1 KB of metadata is 10 TB. This asymmetry is the first design decision: metadata and file contents are separate systems with different needs.
Blocks, not files
Uploading a 2 GB file again because one slide changed is unacceptable. So the client splits files into blocks (for example 4 MB; some systems use content-defined chunking so an insertion does not shift every following block), and computes a hash (SHA-256) of each block. The hash is the block's id: the storage is content-addressed.
This gives three things for free:
- Delta sync. After an edit the client compares block hashes with the previous version and uploads only the blocks that changed.
- Deduplication. The server already has a block with that hash (from this user or another)? Then it is not uploaded at all. Deduplication across users has privacy implications (a client can test whether a file exists elsewhere), so large services limit or scope it.
- Resumable transfers. An interrupted upload continues with the missing blocks.
Architecture
Client (watcher, chunker, local DB)
| 1. commit: file path + ordered list of block hashes
v
Metadata service <----> Metadata DB (SQL, sharded by user/namespace)
| 2. returns which blocks are missing
v
Client uploads missing blocks --> Block service --> Object storage (replicated / erasure-coded)
|
Notification service (long poll / WebSocket) --> other devices: "namespace changed, version N"
- Client. Watches the folder, chunks and hashes changed files, keeps a local database of what the server has, and syncs in the background. Retries are routine, so every operation is idempotent.
- Metadata service. Holds the tree: namespaces, file names, versions, and for each version the ordered list of block hashes. It needs transactions (a rename or a commit is atomic), so a relational database sharded by namespace fits. It never sees file contents.
- Block service and object storage. Stores blocks by hash. Durability comes from replication across zones or erasure coding (cheaper per byte than three copies), and cold blocks move to cheaper tiers. A block is immutable, which makes caching and replication simple.
- Notification service. When the commit succeeds, other devices are told that a namespace changed. Devices keep a long-lived connection (long poll or WebSocket); on reconnect they ask "what changed since cursor N?", so a missed message is never a lost change.
The sync protocol
- The client sends a commit: path, base version, list of block hashes.
- The server replies with the hashes it does not have, the client uploads them (in parallel), then confirms.
- The server creates the new file version atomically (all blocks present), advances the namespace journal cursor and notifies other devices.
- Other devices fetch the journal entries after their cursor, download only the blocks they lack and update their folder.
A journal (an ordered log per namespace) is the backbone: it gives every device the same total order of changes, and a way to catch up after being offline.
Conflicts
Two devices edit the same file from the same base version. The commit contains the base version, so the second one is detected as stale: the server accepts it as a new file ("report (conflicted copy from Ana's laptop).docx") instead of overwriting anyone. Automatic merging is possible only for formats the system understands. Deletes versus edits follow the same principle: never destroy data silently; keep a version history and trash for a retention period.
Sharing and permissions
A shared folder is a namespace mounted in several users' trees, with access control lists on it. Every metadata operation checks permissions; block downloads use short-lived signed URLs so the block store needs no user logic, and large files go straight from the object store or a CDN, bypassing application servers.
Other points
- Large files and bandwidth: parallel block uploads, compression where useful, bandwidth throttling.
- Consistency: strong for metadata inside a namespace, eventual across devices.
- Garbage collection: a block may be removed only when no version references it; keep reference counts or run a mark-and-sweep, with a grace period to avoid racing an upload.
- Encryption: in transit (TLS) and at rest; end-to-end encryption conflicts with server-side deduplication and previews, so it is a product trade-off.
- Security: scan uploads, rate-limit, audit access.
Sources
- Alex Xu, System Design Interview Vol. 1, chapter "Design Google Drive".
- Dropbox: Streaming File Synchronization and Rewriting the heart of our sync engine.
- rsync algorithm (Tridgell and Mackerras).
Practise this topic: System Design System Design › Case studies