CS Fundamentals Trainer · Guides

Design a notification system: push, email and SMS at scale

A case study in fan-out and reliability: requirements and back-of-envelope numbers, an architecture with a request API, per-channel queues and workers, templates and user preferences, how to get at-least-once delivery without sending twice, retries with backoff, rate limits and priority, and what to measure.

4 min read

The problem. Applications need to tell users things: an order shipped, a password reset, a friend's comment, a marketing campaign. The channels are different (mobile push, email, SMS, in-app), each has its own provider and limits, and the volume swings from a trickle to millions in minutes. Design the system that accepts "tell user U about event E" and gets it to the right places reliably.

Requirements

Functional: send via push, email, SMS and in-app; per-user preferences and opt-outs; templates with localisation; scheduled and triggered sends; delivery status.

Non-functional: reliable (a password reset must not be lost), no duplicates (two SMS for one event is a bug that costs money), fast for transactional messages, tolerant of provider outages, and able to absorb campaign bursts without delaying the urgent ones.

Numbers

Assume 10 million users and 5 notifications per user per day on average: 50 million a day, which is 50,000,000 / 86,400 ≈ 580 per second on average. Traffic is bursty: a campaign to 10 million users that must finish in 10 minutes is 10,000,000 / 600 ≈ 17,000 per second, thirty times the average. So the design must treat the burst, not the average, as the load, and queues are the tool that absorbs it.

Architecture

Services --> Notification API --> validate, dedupe, load preferences
                                      |
                              priority topics (high / normal / bulk)
                                      |
                 +-------------+------+-------+-------------+
              push queue   email queue    sms queue    in-app queue
                 |             |             |             |
             push workers  email workers  sms workers   in-app writer
                 |             |             |             |
              APNs/FCM     email provider  SMS gateway   database
                                      |
                              delivery log + status
  1. API. A producer service calls POST /notifications with a user id (or a segment), a template id, parameters and an idempotency key (for example order-1042-shipped). The API checks the schema, rejects keys it has seen recently and writes the request durably to a queue before answering 202 Accepted. It does not talk to providers in the request path.
  2. Preferences and routing. A router service reads the user's channel preferences, quiet hours and opt-outs, device tokens and locale, and decides which channels to use. Legal categories matter: a password reset is transactional and bypasses marketing opt-outs, while a promotion must respect them.
  3. Templates. Messages are rendered from versioned templates with parameters and the user's locale, so producers never format text, and copy can change without a deploy.
  4. Queues per channel and priority. One queue per channel isolates failures (an email outage does not delay pushes), and separate priority queues keep a bulk campaign from starving password resets. Workers read the high-priority queue first.
  5. Workers. Each worker calls the provider, records the result and acknowledges the message only afterwards.
  6. Delivery log. Every attempt and callback (delivered, bounced, opened, unsubscribed) is stored for support, analytics and cleanup of dead device tokens.

Reliability without duplicates

Queues give at-least-once delivery: after a worker crash the message is redelivered. Exactly-once is not available end to end, so make sends idempotent: before sending, a worker does INSERT … ON CONFLICT DO NOTHING (or a Redis SET NX) on (notification id, channel), and sends only if the insert was new. A crash between "send" and "record" can still cause a duplicate (the rare case is a trade-off you accept); many providers also accept an idempotency key of their own, which closes the gap.

Retries. Failures are either permanent (invalid address, uninstalled app: stop, and clean up the token) or transient (timeout, provider 5xx or rate limit: retry). Retry with exponential backoff and jitter (1 s, 2 s, 4 s, … with random spread), up to a limit, then park the message in a dead-letter queue that someone is alerted about. Use a circuit breaker per provider, and fail over to a second provider for critical channels such as SMS.

Rate limits and protection

Scale

Trade-offs to discuss

What to measure

Queue depth and age of the oldest message (the real latency indicator), send success rate per provider, end-to-end latency percentiles for transactional messages, bounce and unsubscribe rates, and the dead-letter count.

Sources

Practise this topic: System Design System Design › Case studies

More guides