CS Fundamentals Trainer · Guides
Design a notification system: push, email and SMS at scale
A case study in fan-out and reliability: requirements and back-of-envelope numbers, an architecture with a request API, per-channel queues and workers, templates and user preferences, how to get at-least-once delivery without sending twice, retries with backoff, rate limits and priority, and what to measure.
4 min read
The problem. Applications need to tell users things: an order shipped, a password reset, a friend's comment, a marketing campaign. The channels are different (mobile push, email, SMS, in-app), each has its own provider and limits, and the volume swings from a trickle to millions in minutes. Design the system that accepts "tell user U about event E" and gets it to the right places reliably.
Requirements
Functional: send via push, email, SMS and in-app; per-user preferences and opt-outs; templates with localisation; scheduled and triggered sends; delivery status.
Non-functional: reliable (a password reset must not be lost), no duplicates (two SMS for one event is a bug that costs money), fast for transactional messages, tolerant of provider outages, and able to absorb campaign bursts without delaying the urgent ones.
Numbers
Assume 10 million users and 5 notifications per user per day on average: 50 million a day, which is 50,000,000 / 86,400 ≈ 580 per second on average. Traffic is bursty: a campaign to 10 million users that must finish in 10 minutes is 10,000,000 / 600 ≈ 17,000 per second, thirty times the average. So the design must treat the burst, not the average, as the load, and queues are the tool that absorbs it.
Architecture
Services --> Notification API --> validate, dedupe, load preferences
|
priority topics (high / normal / bulk)
|
+-------------+------+-------+-------------+
push queue email queue sms queue in-app queue
| | | |
push workers email workers sms workers in-app writer
| | | |
APNs/FCM email provider SMS gateway database
|
delivery log + status
- API. A producer service calls
POST /notificationswith a user id (or a segment), a template id, parameters and an idempotency key (for exampleorder-1042-shipped). The API checks the schema, rejects keys it has seen recently and writes the request durably to a queue before answering202 Accepted. It does not talk to providers in the request path. - Preferences and routing. A router service reads the user's channel preferences, quiet hours and opt-outs, device tokens and locale, and decides which channels to use. Legal categories matter: a password reset is transactional and bypasses marketing opt-outs, while a promotion must respect them.
- Templates. Messages are rendered from versioned templates with parameters and the user's locale, so producers never format text, and copy can change without a deploy.
- Queues per channel and priority. One queue per channel isolates failures (an email outage does not delay pushes), and separate priority queues keep a bulk campaign from starving password resets. Workers read the high-priority queue first.
- Workers. Each worker calls the provider, records the result and acknowledges the message only afterwards.
- Delivery log. Every attempt and callback (delivered, bounced, opened, unsubscribed) is stored for support, analytics and cleanup of dead device tokens.
Reliability without duplicates
Queues give at-least-once delivery: after a worker crash the message is redelivered. Exactly-once is not available end to end, so make sends idempotent: before sending, a worker does INSERT … ON CONFLICT DO NOTHING (or a Redis SET NX) on (notification id, channel), and sends only if the insert was new. A crash between "send" and "record" can still cause a duplicate (the rare case is a trade-off you accept); many providers also accept an idempotency key of their own, which closes the gap.
Retries. Failures are either permanent (invalid address, uninstalled app: stop, and clean up the token) or transient (timeout, provider 5xx or rate limit: retry). Retry with exponential backoff and jitter (1 s, 2 s, 4 s, … with random spread), up to a limit, then park the message in a dead-letter queue that someone is alerted about. Use a circuit breaker per provider, and fail over to a second provider for critical channels such as SMS.
Rate limits and protection
- Respect provider limits with a token bucket per provider and per channel; slow the workers down rather than getting blocked.
- Limit per user (no more than N marketing messages per day) so that a bug upstream cannot spam anyone.
- Protect the API with per-producer quotas.
Scale
- Partition queues by user id so that one user's messages keep their order and workers can scale horizontally.
- Fan-out for a campaign: expand a segment into per-user messages in batches inside a worker, not in the API.
- Delivery log writes are the heavy part: append-only, partitioned by time, with old data archived.
Trade-offs to discuss
- Push vs pull for in-app: store and let the client poll or open a WebSocket.
- Batching or digests reduce noise ("5 new comments") but delay messages.
- Ordering across channels is not guaranteed; keep per-user ordering only inside a channel.
- Scheduled sends need a time-ordered store (a delayed queue, or a table scanned by a scheduler with a lock).
What to measure
Queue depth and age of the oldest message (the real latency indicator), send success rate per provider, end-to-end latency percentiles for transactional messages, bounce and unsubscribe rates, and the dead-letter count.
Sources
- Alex Xu, System Design Interview, chapter "Design a notification system".
- Amazon SQS dead-letter queues and Exponential backoff and jitter (AWS Architecture Blog).
- Firebase Cloud Messaging documentation.
Practise this topic: System Design System Design › Case studies