Voice Agents: Real-Time Speech In and Out
Build agents people talk to: speech-to-text, text-to-speech, real-time voice models, latency budgets, and conversation design for voice.
Talking is the oldest interface there is. A voice agent can answer the phone at 2 a.m., take a booking while someone is driving, or walk a technician through a repair while their hands are busy. When it works, it feels like magic. When it doesn't, it feels like the worst phone menu you've ever used: long silences, robotic monologues, and an agent that talks over you and then mishears your email address.
The difference is rarely the model's intelligence. It's latency, turn-taking, and conversation design, three things text agents never have to think about. In this lesson you'll learn the two main architectures for voice agents, where the milliseconds go and how to win them back, how to handle interruptions, and how to design conversations that work when nobody can see the screen.
Two architectures for voice agents
Every voice agent has to hear, think, and speak. There are two broad ways to wire that up.
The pipeline: speech-to-text, model, text-to-speech
A pipeline (sometimes called cascaded) chains three specialized components:
- Speech-to-text (STT), also called automatic speech recognition, turns the caller's audio into a transcript, ideally streaming partial words as they're spoken.
- A language model reads the transcript, calls tools, and writes a text reply. This is the same agent you already know how to build.
- Text-to-speech (TTS) turns the reply into audio, ideally streaming the first sound before the whole reply is written.
Around those three sit two more pieces: voice activity detection (VAD), which notices when someone is speaking, and turn detection (or endpointing), which decides when they've finished.
Speech-to-speech: one realtime model
A speech-to-speech model (often called a realtime model) takes audio in and produces audio out directly, handling listening, reasoning, and speaking in one model over a persistent connection. OpenAI's Realtime API and Google's Gemini Live API are well-known examples. Because nothing is converted to text and back in between, these models can respond quickly and pick up on tone, hesitation, and emphasis that a transcript loses.
Choosing between them
| Pipeline | Speech-to-speech | |
|---|---|---|
| Latency | Good with careful streaming; each hop adds time | Usually lower and more natural out of the box |
| Choice of model | Any text model, including your strongest reasoning and tool-use model | Limited to the provider's realtime models |
| Control and debugging | A transcript at every step; easy to log, test, and swap parts | Harder to inspect; behavior lives inside one model |
| Voice options | Pick any TTS voice, including custom brand voices | The provider's voices |
| Tone and emotion | Mostly lost in transcription | Preserved in both directions |
| Cost | Three bills, each predictable | Audio tokens, often priced well above text |
A practical default: start with a pipeline if your agent does serious tool work, needs strict compliance, or must be easy to audit, because you get a text record of everything and can use whichever model is best at the task. Try speech-to-speech when conversational feel matters most and the task is simple, such as a friendly front desk or language practice. Many teams prototype both. Vendors in this space move quickly, so check current docs for model names, voices, and pricing before you commit.
You rarely build the plumbing yourself. Open-source frameworks such as Pipecat and LiveKit Agents handle audio transport, VAD, turn detection, interruption, and the wiring between STT, model, and TTS vendors, and several hosted platforms do the same for phone lines. Your job is mostly the agent: its prompt, its tools, and its conversation design.