CS Fundamentals Trainer · Guides

Design a file storage and sync service (Dropbox-style)

A case study in large files and many devices: splitting files into content-addressed blocks, a metadata service separate from block storage, delta sync and deduplication, change notification to other devices, conflict handling, sharing, and the numbers that show why the block store, not the database, is the big cost.

4 min read

The problem. Users keep files in a folder on several devices. Changes on one device must appear on the others, quickly and without re-uploading whole files, even over poor connections; files can be gigabytes; and no change may ever be lost.

Requirements and numbers

Functional: upload, download, automatic sync across devices, version history, sharing, offline edits that sync later. Non-functional: reliability and durability above all (losing a file is the worst failure), efficient bandwidth, eventual consistency across devices with clear conflict handling.

Assume 50 million users with 10 GB stored on average: 50,000,000 x 10 GB = 500 PB of logical data. With deduplication and compression it may be less, but it is petabytes either way. Metadata is tiny in comparison: 10 billion files of 1 KB of metadata is 10 TB. This asymmetry is the first design decision: metadata and file contents are separate systems with different needs.

Blocks, not files

Uploading a 2 GB file again because one slide changed is unacceptable. So the client splits files into blocks (for example 4 MB; some systems use content-defined chunking so an insertion does not shift every following block), and computes a hash (SHA-256) of each block. The hash is the block's id: the storage is content-addressed.

This gives three things for free:

Architecture

Client (watcher, chunker, local DB)
   |  1. commit: file path + ordered list of block hashes
   v
Metadata service  <---->  Metadata DB (SQL, sharded by user/namespace)
   |  2. returns which blocks are missing
   v
Client uploads missing blocks --> Block service --> Object storage (replicated / erasure-coded)
   |
Notification service (long poll / WebSocket) --> other devices: "namespace changed, version N"

The sync protocol

  1. The client sends a commit: path, base version, list of block hashes.
  2. The server replies with the hashes it does not have, the client uploads them (in parallel), then confirms.
  3. The server creates the new file version atomically (all blocks present), advances the namespace journal cursor and notifies other devices.
  4. Other devices fetch the journal entries after their cursor, download only the blocks they lack and update their folder.

A journal (an ordered log per namespace) is the backbone: it gives every device the same total order of changes, and a way to catch up after being offline.

Conflicts

Two devices edit the same file from the same base version. The commit contains the base version, so the second one is detected as stale: the server accepts it as a new file ("report (conflicted copy from Ana's laptop).docx") instead of overwriting anyone. Automatic merging is possible only for formats the system understands. Deletes versus edits follow the same principle: never destroy data silently; keep a version history and trash for a retention period.

Sharing and permissions

A shared folder is a namespace mounted in several users' trees, with access control lists on it. Every metadata operation checks permissions; block downloads use short-lived signed URLs so the block store needs no user logic, and large files go straight from the object store or a CDN, bypassing application servers.

Other points

Sources

Practise this topic: System Design System Design › Case studies

More guides