Server-sent events across pods relay, never echo

Updated 14 September 2026 with production counts, figures, the rolling-deploy guard, the named heartbeat and the late-joiner replay.

The dashboard has one long-lived connection to the backend, over server-sent events, and everything that changes while you watch comes down it: a call being created, ringing, ending, its cost landing, the balance moving, a campaign's progress, and — for a call you open — the transcript word by word while the caller is still speaking. The transport is plain server-sent events, the thing the browser gives you for free as EventSource. What makes server-sent events across pods work — the backend runs on several — is that any pod can produce an event, but only the pod holding your connection can deliver it, so every event is delivered locally and relayed to the sibling pods with a tag that stops it coming back to its origin. Over the 30 days to 13 September 2026 the relay delivered 125,728 events across pods for 448 dashboard connections; one connection fell behind enough to be told so, and one ticket was presented after it had expired. The same call events reach your own code over a WebSocket, as in place a call from code and get the record back.

The dashboard moved from WebSockets to server-sent events in May 2025, seven months before the current backend existed. The reasons have held: a single direction is all a dashboard needs, and the browser handles the framing, the parsing and the reconnect.

EventSource authentication is a ticket, not a bearer token

The dashboard keeps its JWT in memory and sends it as an Authorization header on every API call. EventSource has no way to do that. The whole constructor, as the HTML standard defines it, is "source = new EventSource( url [, { withCredentials: true } ])" — a URL and one boolean (WHATWG HTML Living Standard). No headers.

So the connection is authenticated in two steps. The dashboard makes an ordinary authenticated POST /api/v1/auth/sse-ticket, naming the account it wants to watch; the backend checks the user has access to that account, mints a ticket, and stores it in Redis with the user, account and session bound to it, for 30 seconds. The dashboard then opens GET /api/v1/sse/connect?ticket=…. The backend validates the ticket with a single atomic get-and-delete — Redis GETDEL — so the first presenter gets a stream and a second presenter gets nothing. An expired ticket and a used one look identical to the server, which is the point: there is no state to replay. The account the stream serves is the one bound at minting; the stream endpoint never takes an account from the client.

The client owns reconnection, not the browser

A single-use ticket has a consequence the standard did not anticipate: the browser's built-in reconnect is useless. MDN describes the default: "By default, if the connection between the client and server closes, the connection is restarted" — with the server able to tune the delay through the retry field, "the browser will wait for the specified time before attempting to reconnect" (MDN, Using server-sent events). When the stream drops, EventSource retries the same URL, and the ticket in that URL is already gone, so every native retry is a guaranteed failure with an error event attached. The dashboard therefore closes the EventSource on any error and runs its own loop: fetch a new ticket, open a new stream, with exponential backoff from one second to a cap of thirty, and a limit of ten attempts before it stops and waits for something to change.

The things that change are watched explicitly. A tab coming back to the foreground checks whether the stream is still open and whether a heartbeat has arrived in the last 45 seconds, and reconnects if not — a suspended tab loses its socket silently. The browser's online event reconnects immediately. A token refresh reconnects only if the stream is down, because refreshes happen every eleven minutes or so and must not churn a healthy stream. A ten-second watchdog covers the rest.

Three tiers, one connection

Not every event is for every screen. Account-wide events — call created, status, ended, balance updated, payment required, campaign and schedule changes, notifications, export status — arrive on every connection for that account, with no subscription. Per-call streams — transcript, interim words, the bot's own speech, turn changes, node transitions, recording state — are heavy and only wanted by whoever has that call open, so the dashboard subscribes with POST /calls/{id}/subscribe, after verifying it owns the call and the connection. A third tier carries events addressed to a user rather than an account.

The subscription request is where the multi-pod reality shows. The load balancer sends that POST to any pod, and the pod that receives it may not be the one holding your stream. When it finds it does not, it publishes the subscription as a command to all pods; the one that owns the connection re-checks that the user owns it and registers the subscription. The user id travels with the command so ownership is never taken on trust from another pod.

Deliver locally, relay to siblings, never echo

When a pod produces an event — a call ended on the pod that ran its finalization, a balance changed on the pod that processed the payment — it delivers to its own connections first, then publishes the event over NATS core pub/sub, wrapped in an envelope carrying the originating pod's id. Every pod holds one wildcard subscription to that relay and filters in memory by account, user or call; there is no per-account subscription to create or tear down. A pod that receives an envelope with its own id in it drops it. That is the whole dedup: no sequence numbers, no shared state.

The subtle bug was at startup. NATS core pub/sub does not buffer for a subscriber that is not yet there, so a pod that started accepting HTTP connections before its relay subscription was up would take on dashboards and silently miss everything the other pods produced until the subscription caught up. The server now waits, up to ten seconds, for the relay subscriber to report ready before it opens the port.

A slow dashboard is told, not dropped

Each connection has a buffer of 100 events. A browser that stops draining it — a laptop lid closing, a debugger paused — does not get disconnected; it receives a lag-warning event carrying the number of events it missed, and stays connected. In the 30 days to 13 September 2026 that fired once. The dashboard treats it as a reason to refetch rather than a fault.

Liveness runs in both directions. The server sends a named ping event every 20 seconds, comfortably under the 60-second idle timeout of the load balancer in front of it. It has to be a named event: the SSE comment line that servers conventionally use as a keep-alive is swallowed by EventSource and never reaches page code, so only a real event can drive a client-side dead-connection check. Two things we learned the hard way sit in the response headers: without X-Accel-Buffering: no and Cache-Control: no-transform, the first event got through the proxy and every later one was buffered — the dashboard would show "connected" and then nothing. And on a rolling deploy, a draining pod refuses new tickets — it would otherwise consume a ticket and die, leaving the browser with a dead stream and no ticket — and sends its existing connections a retry: 1000 and a shutdown event, so they reconnect at once and land on a pod that is staying.

Joining a call that is already talking

The hardest event to get right is the first one after you open a live call. The bot publishes its events to a durable log — transcript, speech and status wait for the log's acknowledgement; interim words do not, because at a word every 250 to 400 milliseconds the acknowledgement wait was visible in the pipeline. The backend's consumer reads that log, persists what needs persisting, and broadcasts each event to the subscribed connections.

A dashboard that opens the call thirty seconds in wants the thirty seconds it missed, then live events, with nothing lost at the seam. Subscribe first, then replay: the replay reads the call's events from the log, and because the subscription is already registered, everything broadcast during the replay arrives too — as duplicates, which the connection filters through a 512-entry window of content hashes. The order matters. Replay first and every event between the end of the replay and the registration of the subscription is gone; a greeting's word events sit exactly in that window. The rule written into the code is that duplicates are benign and missing events are the bug the mechanism exists to prevent, so if the dedup structure is ever in a bad state it delivers rather than drops.

The log keeps seven days of events, so a call opened a week later replays the same way.

What the dashboard does with an event

Two different things, depending on the event. Anything about an active call patches state directly — a new call is inserted into the active list, a status change updates it, an ended call removes it — with no refetch. An ended call also invalidates the cached call history and detail so the next read is fresh, and patches the campaign's contact list in place. The balance event invalidates the balance query. There is no Last-Event-ID, no gap replay at the account level; on every reconnect the dashboard refetches the active-call list, including when that list comes back empty, because a call that ended during the gap would otherwise sit at "ringing" until the page reloaded.