How a call gets a bot from the warm pool

Updated 14 September 2026 with production numbers, the busy-reply retry, the August scaling incident, and figures.

When a call starts on Talkif, the backend does not start a bot for it. It sends one request to a warm pool of bots that are already running, and one bot that is free claims the call, answers with the address of its media socket, and begins preparing before the phone has rung. The pool is sized by an autoscaler that watches the number of active calls and the number waiting, and keeps a fraction of a bot spare at all times. Over the 30 days to 12 September 2026, 325 calls were assigned a bot this way in production; 243 of the 305 with a recorded attempt number were claimed on the first request. What the bot does with that head start is in why our bots start talking in under half a second.

The claim is a question asked of the whole pool

The assignment message goes out on a request-reply channel that every idle bot in the region is subscribed to, in a group — the broker delivers each request to exactly one member of the group. The bot that receives it checks three things in order: that it is not shutting down, that its circuit breaker has not opened after repeated warm-up failures, and that it is not already on a call. If all three pass, it loads the flow, opens its provider connections, pre-generates the greeting (the warm-up described in an earlier post), marks itself assigned, and replies with its socket URL. The backend puts that URL in the instructions it hands to the carrier, and the media stream, when it arrives, lands on a bot that already knows the call.

There is no central scheduler deciding which bot gets which call, and no lock. A bot's own busy flag is the only state that matters, and it lives in the bot. This is simpler than a registry with atomic claims, and it has one weakness that took us until July to close. It is also a different shape from platforms that route from the middle: LiveKit's agent dispatch, for instance, keeps capacity knowledge on the server and promises that it "optimizes dispatch for high concurrency and low latency, typically supporting hundreds of thousands of new connections per second with a max dispatch time under 150 ms" (LiveKit docs, Dispatching agents). Our pool is a few dozen bots per region, not hundreds of thousands of connections, and the trade we took is no coordinator to keep consistent — at the price of the next section.

A busy bot can be asked, and says so

The broker's group delivery is not capacity-aware. It picks a subscriber, and a bot that is mid-call is still a subscriber, so it can receive the request and reply busy while idle bots on the same channel sit unasked. With a small pool, one busy bot can absorb request after request. In the 30-day window, 209 requests were answered busy — against 325 successful assignments — and the attempt distribution shows what that costs:

Attempts before a bot was claimed	Calls
1	243
2	35
3	11
4 or more	16

Three hundred and five calls recorded an attempt number; four in five were claimed on the first request, and the tail reaches a tenth attempt.

The fix is to treat a busy reply as what it is: a fast round-trip of a few milliseconds, cheap to repeat. Since July 2026 a busy or draining reply is retried on the same channel up to four more times with a 150 ms pause — the pause is long enough for the bot that just answered to be past the point where it would be picked again — before the backend falls back to the region-wide channel and tries there. A request that gets no reply is treated differently: nobody answered within the 10-second window, and another attempt on that channel would burn another ten seconds, so the phase ends at once. Re-rolling the dice is free; waiting is not.

What happens when nobody is free

The two cases are different because the caller is in a different place.

An outbound call that finds no bot is not failed. It is put back on the queue with a bot-wait marker, and the queue processor re-drives it; the budget for that wait is 90 seconds, measured from the first failed attempt, and the number is not arbitrary — it is the time it takes the autoscaler to notice the backlog and bring a new bot from nothing to registered. Nobody is on the line for a call that has not been dialled, so the wait is invisible to the person being called. If the budget runs out, the call gets one final attempt and then fails honestly with a bot-unavailable outcome rather than being retried forever. In the 30-day window, 29 calls entered that wait and 9 exhausted it — all nine within the same second on 25 August, which is the incident below.

An inbound call has a person on the line, and there is nothing to wait with. If the region has no healthy bot, or the account is at its concurrent-call ceiling, the call is rejected at the carrier with a busy signal; an account can enable a callback on rejection, so the caller is rung back when a bot is free instead of being asked to try again.

The warm pool is sized by concurrent calls, with headroom

The bots run under KEDA, which drives a Kubernetes autoscaler from a Prometheus query. The query is the number of active calls in the region, plus the number of direct calls waiting for a bot at full weight, plus the scheduled backlog — clamped, for a reason explained next. The target is 0.7 units per bot: one active call means two bots, two calls means three, five calls means eight. The previous setting, one call per bot, only added capacity after every bot was busy, which for a phone call is already too late. The scaler polls every five seconds and may double the pool, or add ten bots, every fifteen seconds; it scales down by at most a quarter or five bots a minute, after three minutes of stability. The floor is three bots per region, so a quiet region never pays a cold start, and the ceiling is fifty.

The clamp exists because of 25 August 2026. A campaign queued 109 calls with a concurrency cap of five. The scaler saw 109 waiting, grew the pool to the ceiling of fifty bots across ten machines, and the campaign — capped at five — used a tenth of them. Campaign backlog is now counted only up to the active count plus four: a campaign can never use more bots than its cap, so the raw backlog overstates what the pool needs. Active calls and direct backlog are never clamped; a call in progress occupies exactly one bot.

Scaling down must not hang up on anyone

An autoscaler that removes bots has to choose which. Kubernetes lets a workload say. The pod-deletion-cost annotation "represents the cost of deleting a pod compared to other pods belonging to the same ReplicaSet. Pods with lower deletion cost are preferred to be deleted before pods with higher deletion cost." The docs also say the annotation "is honored on a best-effort basis, so it does not offer any guarantees on pod deletion order" (Kubernetes documentation, ReplicaSet).

A background task in the backend watches the bot registry and writes that cost onto each pod as its state changes: high while a bot is assigned or on a call, low when it is idle. When the scaler removes five bots, it removes idle ones. Best-effort turned out to be the operative phrase. The annotation only steers the replica controller; the cluster's node consolidator, which drains under-used machines, does not read it, and with a ten-minute termination grace it could drain a machine holding live calls. Since August 2026 a busy bot also carries a do-not-disrupt annotation that the consolidator does honour, and it is removed the moment the call ends so that empty machines can still be reclaimed.

What this does not do

Assignment is regional and stays regional: a call that finds no bot in its region is not sent across the ocean to another, because the media path would follow it. A bot that is assigned but whose media socket never arrives — the callee was busy, the carrier gave up — releases itself after a timeout; before July that timeout was the only release, and a small pool could be starved by calls nobody answered. Now a terminal outcome from the carrier cancels the assignment at once. And the busy-reply retry is a mitigation, not a design: directed assignment, where the backend asks a specific idle bot rather than the group, is the version that removes the dice altogether.