the latency problem in real-time ai calls
Why pauses and turn-taking speed are the difference between a call and an interrogation.
a phone call has a clock a chat window does not
Text interfaces can pause for a second or two without anyone noticing — a spinner absorbs the wait. A phone call cannot. Human conversational turn-taking runs on gaps of a few hundred milliseconds; anything much longer reads as either a dropped line or, worse, a machine "thinking", which breaks the illusion of a conversation immediately and permanently for that call.
This is a harder constraint than it sounds, because a voice agent on a real call is doing several things in sequence for every turn: transcribing what was just said, deciding what to say back, generating that response, and synthesising it as audio. Each step adds latency, and the pieces do not parallelise cleanly.
where the time actually goes
Speech-to-text has to wait for enough silence to be confident the person finished speaking, which is itself a latency-accuracy trade-off — cut it too fine and the agent interrupts; wait too long and every turn feels sluggish. Language generation adds its own delay, especially for longer responses. Text-to-speech synthesis is usually the fastest step but still not free.
None of these steps is the villain individually. The problem is that they stack, and a system tuned for accuracy on each step in isolation can still produce a call that feels unbearably slow end to end.
designing around the constraint rather than fighting it
Some of the fix is engineering — streaming partial transcripts instead of waiting for a full utterance, starting speech synthesis before the full response text is finalised, keeping response length short so there is less to generate and speak. Some of the fix is conversational design — short acknowledgements ("samajh gaya", "theek hai") that buy processing time without feeling like dead air, the same way a human filler word does.
The two approaches work together. Neither alone gets a call to feel like a call rather than a series of questions read off a script with pauses in between.
what this means for a production system
Latency is not a metric you fix once. It moves with every model swap, every added qualification field, every language added to the mix. Treating it as a permanent constraint to design around — not a bug to eliminate — is what keeps a voice agent feeling like a conversation months after the first version shipped.