Track 2 · Products people use
mediumNotification System
Take events from every service and turn them into push, email and SMS — through providers you do not control, without sending anything twice.
Suggested architecture
Scenario
Your own parts are cheap; the providers are the limits. Shift the mix toward SMS and watch a 500-per-account cap arrive long before anything of yours is busy — then slow a provider down and watch the workers run out of slots.
Notification requests from all services per second.
Share of events that go out by SMS — the slowest, most rate-limited channel.
How long the email provider takes per call. A slow provider holds a worker slot for the whole wait.
Each holds 2,000 calls in flight. Size by provider latency × rate, not by CPU.
Click a component for its role, common technology choices and tradeoffs, and what it is carrying at this scale. Hover a connection to see what flows along it. Drag to rearrange — layout changes are local and reset on reload.
Every figure here is a rough estimate from simple capacity arithmetic, not a benchmark. Each part carries its own assumption about what one copy can do — real numbers depend on your hardware, payloads and access pattern. The point is which component moves first as you turn the dials, not the digits themselves.
In plain words
A notification system takes "something happened" from any service and turns it into a push, an email or a text — the right one, in the right language, not too often, and never twice. Its real constraints are outside your control: the providers that actually deliver have quotas and latencies you cannot change, so the design is built around waiting for them gracefully.
The shape of the problem
Every service in a company eventually needs to tell a user something, and every one of them will, left alone, integrate a push SDK and an email API badly. A notification system exists so that happens once. It takes events — "order 123 shipped" — from anyone, and owns everything after: which channels this user allows, what the message says in their language, whether they have had too many today, and the actual sending through providers you do not run.
Assume 3 000 events/s average across all services, five times that when a marketing campaign fires, and a mix of roughly 70% push, 25% email, 5% SMS.
Those provider numbers are the design. Push is fast and nearly free. Email has a sending quota per account. SMS is slow, expensive, and capped per phone number. Nothing you own is the bottleneck; the third parties are.
Accept fast, send later
The API does three things and returns: validate the event, check its idempotency key (a caller-chosen id that makes a repeated request harmless) against a short-lived set in Redis, and put it on a queue. The caller gets an acknowledgement in a few milliseconds and never waits on a provider.
Everything slow happens in the delivery workers, consuming the queue: load the user's preferences, render the template, check the per-user rate limit, call the provider, record the attempt (retrying with backoff and jitter). The queue is what turns a provider outage into a delay. Depth rises, nothing is lost, and when the provider recovers the backlog drains — which is also why queue depth is the one metric to alarm on.
One queue topic (a named lane) per channel. A slow SMS provider should hold up SMS, not push. Shared queues turn the slowest provider into everyone's latency.
Never twice
Duplicates come from two places: the caller retrying the API call, and the worker retrying the provider call. Both are handled the same way — an idempotency key that travels the whole way.
The caller supplies a key (order-123-shipped). The API records it with a 24-hour
TTL; a second call with the same key is acknowledged and dropped. The worker
passes the same key to the provider as its client-side message id; the good
providers deduplicate on it, so a retry after a timeout does not send twice. When
a provider offers no such id, the delivery log is the last line: check for a
successful attempt before sending, and accept that a crash between send and log
can still produce a duplicate.
Workers wait, they do not work
A delivery worker spends almost all of its time waiting on a provider: 200 ms for push, 300 ms for email, half a second for SMS. Its capacity is therefore not CPU but concurrent calls — how many provider requests it can have in flight at once. Six workers holding 2 000 calls each is 12 000 in flight, which at 300 ms a call is 40 000 sends per second of headroom.
Slow the email provider down with the slider and watch what happens: the rate does not change, but every call holds its slot longer, and the workers fill up from the inside. This is the failure that surprises people, because CPU graphs look fine the whole way down.
Where this design breaks
- Per-user rate limits across workers. "No more than 5 an hour" needs a shared counter, which is one more Redis round trip per event. Fold it into the preferences read.
- Fan-out events. "Everyone in this group" is one event and ten thousand sends. Expand at enqueue time, not in the worker, or one event blocks a partition (one lane of the queue) for everyone behind it.
- Provider quotas at peak. An email account allowed 2 000/s does not care that it is Black Friday. Warm up limits ahead of campaigns, or spread across accounts.
- Templates in the hot path. Rendering with a locale (language and region) lookup per event is fine at 3 000/s and not at 100 000/s; cache compiled templates per locale.
Take this with you
- The one idea: accept fast, send later. A queue turns a slow or broken provider into a delay instead of an outage.
- In an interview, cover the idempotency key end to end, one queue per channel, and why workers are sized by calls in flight rather than CPU.
- At work, alarm on queue depth, not on worker CPU — that graph looks fine all the way down.