Voice agents, part 1 · 4 min read
What is inside a voice AI agent
Six parts, one second, and about ten cents a minute. Press play to watch where the time and the money go.
One turn, as a waterfall
The clock starts the moment the caller stops talking and stops when they hear the agent's first word. Every bar is one component. Each one's latency stacks on the last, and each one bills by the minute.
Latency is a sum. No single part is slow; six reasonable parts add up to more than a second. Cost is a sum too. The raw stack is under a dime a minute; the platform fee, if you use one, can double it.
The six parts
| Part | What it does | The number that matters | Typical | $ / min |
|---|---|---|---|---|
| Phone line + transport | Carries the caller's audio in and the agent's audio out. Buffers a little to smooth the network. | Jitter buffer and hop time | 60–100 ms | $0.008–0.014 |
| End-of-turn detection | Decides the user has finished. Guessing early means interrupting them; guessing late is dead air. | Silence timer or turn-detector delay | 250–500 ms | bundled, $0 |
| Speech to text | Turns audio into words while the user is still talking, then finalises the last few after they stop. | Final-transcript lag after the last word | 200–300 ms | $0.0077 |
| Language model | Writes the reply. In voice only the first token matters, because speech can start before the sentence is finished. | Time to first token | 120–500 ms | $0.006–0.020 |
| Text to speech | Turns the reply into audio, streamed out syllable by syllable as tokens arrive. | Time to first byte | 75–100 ms | $0.04 |
| Orchestration + hops | The glue that routes audio and text between the four services. Each hop is small; four of them are not. | Inter-service hops | 40–80 ms | $0.008 self-hosted, $0.05–0.14 on a platform |
Why "first token" and "first byte", not total time
In a chat window you wait for the whole answer. In voice, speech synthesis starts on the first few tokens and keeps pace with generation, so the caller hears the reply while the model is still writing it. The only part of generation on the critical path is the wait for the first token. The same logic applies to text to speech: what matters is how fast the first audio bytes start streaming, not how long the whole sentence takes to synthesise. A pipeline where every stage streams into the next is why the well-configured preset is so much shorter without any stage being individually dramatic.
The hidden part: end-of-turn detection
Before transcription can finalise, something has to decide the user is done. The simplest version is a silence timer: no speech for 450 to 500 milliseconds means "your turn". That single choice is often the longest bar in the waterfall, and it is invisible in most architecture diagrams. Smarter systems use a small model that reads pitch, pacing, and the words so far to guess whether a pause is a full stop or a comma, and can commit in around 250 milliseconds. Get it wrong the other way and the agent talks over people, which callers hate far more than a slow reply. Part 2 of this series is about that trade-off.
Where the numbers come from
Latency: Deepgram documents Nova-3 streaming at under 300 ms and reported a 198 ms median first-token time self-hosted in May 2026. ElevenLabs quotes about 75 ms to first audio for Flash v2.5. Time-to-first-token figures of roughly 120 to 160 ms for Groq and Cerebras, about 280 ms for Gemini 2.5 Flash, and slower for GPT-4o-mini come from 2026 provider latency benchmarks. Transport, hop, and end-of-turn figures are typical values from voice-platform engineering write-ups rather than a single measurement.
Cost: Deepgram Nova-3 streaming is $0.0077 per minute pay-as-you-go. ElevenLabs on the major agent platforms is $0.04 per minute after the March 2026 pricing change. LLM cost per minute depends on how much history you send each turn; $0.006 to $0.02 covers a fast open-weight model through GPT-4o-mini class pricing at typical conversation lengths. Telephony is $0.008 to $0.014 per minute, orchestration $0.008 self-hosted or $0.05 to $0.14 per minute on hosted platforms. All list prices, checked September 2026; they will drift.
What this series covers
- Anatomy and metrics. This piece.
- End-of-turn detection. Three real detectors on one clip: which ones talk over you.
- Interruptions. Barge-in with real audio: two cut-ins, three stop policies.
- Why it sounds robotic. Streaming, chunking, and prosody.
- When it hears wrong. Transcription errors and how they cascade.
- Knowing it works at scale. Evals, tail latency, regression tests.
- Speech-to-speech models. Does the pipeline go away?