Track 2 · Products people use
mediumChat
Messages between people who are online now, and delivery for the ones who are not. The interesting part is that a connection is a resource you hold, not a request you serve.
Suggested architecture
Scenario
Messages are cheap; connections and fan-out are not. A gateway that holds 50,000 idle sockets is normal — the question is what happens when a fraction of them all send at once, and how many deliveries each message becomes.
Messages arriving from all connected users per second.
Conversations opened per second — each a range read from the store.
Recipients per message. 1 is a direct chat; a 50-person group is 50 deliveries — and 50 push sends when they are offline.
Each holds up to 50,000 sockets. Size for connections first, messages second.
Click a component for its role, common technology choices and tradeoffs, and what it is carrying at this scale. Hover a connection to see what flows along it. Drag to rearrange — layout changes are local and reset on reload.
Every figure here is a rough estimate from simple capacity arithmetic, not a benchmark. Each part carries its own assumption about what one copy can do — real numbers depend on your hardware, payloads and access pattern. The point is which component moves first as you turn the dials, not the digits themselves.
In plain words
A chat system delivers a message from one phone to another in under a second, whether or not the other app is open. The unusual part is that servers have to push messages down to clients, which means holding a connection open for every user — and holding a million connections is a different job from answering a million requests.
The shape of the problem
A chat system is two delivery systems that have to look like one. While the app is open, a message should arrive in well under a second, which means the server has to push — the client cannot poll (ask "anything new?" over and over) every 200 ms from a phone. When the app is closed, there is nothing to push down, and the platform's notification service is the only way in. Almost every part of the design exists to make those two paths agree about what "delivered" means.
Assume 10 M daily users, each sending 40 messages a day, and 1 M connected at any moment.
- Messages: 400 M / 86 400 ≈ 5 000 msg/s average, four times that in the evening.
- Connections: 1 M open sockets, idle almost all the time.
- Deliveries: messages × recipients — a 50-person group turns one send into fifty.
The connection number is the one that sizes the gateways. Five thousand messages a second is a modest service; a million open sockets is not, and it costs memory and file descriptors (the operating system's per-connection bookkeeping, which has a hard cap) whether or not anyone types.
Connections are held, not served
An HTTP request is served and forgotten. A WebSocket is held: the gateway keeps state per socket (which user, which device, a send buffer) for as long as the connection lives. That changes the sizing unit. A gateway instance is limited by open connections — call it 50 000 — long before message throughput matters, and a restart drops every one of them at once.
Two consequences follow. First, the gateways must be stateless (holding nothing a user would miss if the instance vanished) in every way except the socket map, so any of them can take any reconnecting user. Second, clients must reconnect with backoff and jitter; a deploy that restarts eight gateways in sequence is eight small thundering herds (everyone reconnecting at the same instant), and without jitter they arrive as one large one.
Keep the gateways dumb. They read frames and write frames. Everything that needs to understand a conversation lives one hop back, in the chat service, where it can be scaled by messages rather than by sockets.
Fan-out through a queue, not gateway to gateway
The tempting design is for the gateway that receives a message to find the recipient's gateway and forward directly. It works for one-to-one chat with one device per user, and falls apart the moment either of those stops being true.
Instead the chat service persists the message, publishes it to a queue split into partitions (independent lanes) by conversation, and a fan-out worker does the delivery: look up each recipient in the presence cache, write down the right socket on the right gateway, or hand off to push for anyone not connected. The queue buys three things: acceptance is fast ("sent" means "queued"), delivery is retried by the consumer rather than the sender, and ordering within a conversation comes from the partition key.
The diagram deliberately does not draw the delivery edge from the fan-out worker back to the gateways. It is the same gateway fleet writing instead of reading; drawn as a connection it would loop traffic back into the chat service, and the model would count every delivery as a new message.
Presence is a guess with a TTL
"Online" cannot be known; it can only be assumed until proven otherwise. Each
connected client heartbeats every 30 s and the gateway refreshes a Redis key,
user → gateway, with a 60 s TTL (time-to-live: the key deletes itself unless
refreshed). Presence is whatever keys have not expired.
That means a phone that lost signal stays "online" for up to a minute, and a message sent in that window is written to a socket that no longer exists. The fix is not better presence; it is that delivery is not the source of truth. The message is in the store, the client fetches what it missed on reconnect, and the push notification covers the gap in between.
Where this design breaks
- Big groups. A 10 000-member channel makes every message 10 000 deliveries and 10 000 push sends. Past a few hundred members, switch to pull: notify "new messages", let clients fetch.
- Hot conversations. One partition key per conversation means one consumer per conversation; a single viral thread is capped at what one worker can do.
- The push provider. APNs and FCM (Apple's and Google's push services) rate-limit per app. A large offline audience at peak is a third-party limit you cannot raise on the day.
- History reads. "Last 50 messages" is one partition scan. "Search my messages" is not, and needs a separate index.
Take this with you
- The one idea: connections are held, not served. Size gateways by open sockets, keep them dumb, and put the logic one hop back where it scales by messages.
- In an interview, explain WebSocket vs push notification, why fan-out goes through a queue, and why presence is a guess with a TTL.
- At work, test what a rolling restart of the gateways does to reconnects. Without jitter, a deploy is a self-inflicted traffic spike.