How Speech-to-Speech Voice Agents Work: Chained Pipelines, Native Audio and Full Duplex
A voice agent turns speech into action in one of three ways: a chain of speech recognition, a text agent and speech synthesis; one native audio model that listens, reasons and speaks; or a full-duplex voice model that keeps talking while a separate backend works. This explainer follows each design and how its timing is judged.
Checked against OpenAI's voice agents and Realtime guides, Google's Live API overview, xAI's Voice API docs (Grok Voice 3.0) and Artificial Analysis's Speech to Speech methodology
- A model or component that processes the conversation
- A separate backend that runs reasoning and tools
As of All entries as described by the labs and Artificial Analysis
| Attribute | Chained pipeline | Native speech-to-speech session | Full duplex with delegation | Figure |
|---|---|---|---|---|
| How speech reaches reasoning | Three stages managed by your application: speech-to-text, the agent workflow itself, then text-to-speech1 | One model interprets the audio, decides what to do and responds in speech, in one session12 | GPT-Live listens and speaks at the same time (full duplex) and handles the spoken interaction, delegating reasoning and tool use to a separate backend1 | Vendor-stated |
| What you can inspect or swap | The text between stages can be stored, checked or transformed, and each component can be replaced independently1 | The session handles audio turns, tools, interruptions and handoffs; tools, handoffs and guardrails are attached to the agent as they would be for a text agent12 | You can keep an existing text workflow as the backend; with client delegation your application runs the work and returns results, with Responses delegation an OpenAI-hosted model does it1 | Vendor-stated |
| Turn-taking and interruptions | Not specified in the guide's description of the pattern; it lists overlap and yielding to interruptions among the timing behaviors to test in voice agents1 | Documented as built-in session behavior: barge-in, natural turn taking and realtime tool use; xAI's API exposes turn_detection with server_vad for automatic detection24 |
Users can keep talking while backend work runs1 | Vendor-stated |
| How the labs describe the interface | OpenAI: a pipeline you control stage by stage1 | OpenAI Realtime API sessions; Google's Live API, which processes continuous streams of audio, images and text for spoken responses; xAI's Voice API, which streams audio and text both ways over a WebSocket134 | OpenAI's GPT-Live, as a voice front end for an existing agent1 | Vendor-stated |
| What the lab recommends it for | When each stage needs to be visible or replaceable, for example storing a transcript or running policy checks before speech is generated1 | Speech, reasoning and tool use in one session; conversations that should feel immediate, with low first-audio latency12 | Full-duplex conversations with a separate backend1 | Vendor-stated |
| What independent benchmarks cover | Not covered: Artificial Analysis benchmarks "native audio models", those that take audio in and give audio out5 | Speech reasoning (Big Bench Audio), conversational dynamics (a subset of Full Duplex Bench), agentic performance (τ-Voice) and human preference5 | Same four dimensions where the model is benchmarked as a native audio model5 | Vendor-stated |
The three designs are not rival products so much as architecture choices, and the labs' documentation is the source for each cell; it describes what each lab offers, not a comparison across labs. The chained column cites OpenAI's guide because it is the clearest description of the pattern; other vendors support chaining too. For measured latency of native audio models, see Fastest Time to First Audio Among Speech-to-Speech Models. Figure labels: how they work.
How it works
The design choice comes first
OpenAI's voice agents guide puts the decision plainly: "the key design choice is how speech connects to reasoning and tools: a continuous conversation with a separate backend, a single voice model, or a pipeline you control stage by stage", and its rule of thumb is to choose the audio architecture first, then design the rest of the agent workflow as you would for text.1 The rest of this post takes the three in turn.
Chained: speech-to-text, a text agent, text-to-speech
In a chained pipeline your application manages three stages: speech-to-text, the agent workflow itself, and text-to-speech.1 Because the text between stages is visible, you can store the transcript, run policy checks before the text agent responds, call internal systems, and generate speech only once the workflow reaches an approved answer; each component can be replaced independently.1
Native speech-to-speech: one model handles audio
A native speech-to-speech model takes audio in and produces audio out directly. OpenAI describes its Realtime API as using "one model to interpret audio, decide what to do, and respond in speech" in a single session, recommended where the interaction should feel conversational and immediate, with barge-in, low first-audio latency, natural turn taking and realtime tool use.12 Google's Live API likewise processes continuous streams of audio, images and text to give immediate spoken responses,3 and xAI's Voice API streams audio and text in both directions over a WebSocket, with a reasoning.effort setting, a tools list that includes MCP and function types, and a turn_detection setting where server_vad means the server detects turns automatically.4
Full duplex: talk while the backend works
OpenAI's GPT-Live can "listen and speak at the same time, a capability called full duplex": the live model handles the spoken interaction and delegates reasoning and tool use to a separate backend, so users can keep talking while backend work runs.1 The backend can be your own agent (client delegation, where your application runs the work and returns results) or an OpenAI-hosted Responses model (Responses delegation, where your application still runs custom functions); in both, the application controls permissions and business records.1 The guide advises putting speaking behavior in the live model's prompt and business rules in the backend prompt.1
How these designs are benchmarked
Artificial Analysis benchmarks "native audio models" on four dimensions: speech reasoning (its Big Bench Audio test), conversational dynamics, agentic performance and human preference.5 Its conversational-dynamics tests are a subset of Full Duplex Bench and measure four behaviors: pause handling (not interrupting a user's natural pause), turn taking (taking the turn when appropriate), user interruption handling and backchannel handling (continuing when the user says "mm-hmm" instead of treating it as a new turn).5 Agentic performance uses τ-Voice, a benchmark from Sierra of full-duplex voice agents on customer-service tasks, where each score is the mean of three independent trials.5
What to measure in your own agent
OpenAI's guide suggests testing both the conversation and the completed task, for example checking that a booking assistant's spoken confirmation matches the saved appointment.1 It lists conversational timing (audible response timing, unwanted silence, overlap and yielding to interruptions), speech and language (accents, noise, names and numbers) and session reliability as separate areas, and advises tracking median and 95th-percentile latency while recording backend time separately.1 The practical consequence of all three designs is the same: a voice agent can speak quickly and still fail the task, so latency and task success need to be read together.
Sources
- OpenAI, Voice agents (API guide), retrieved 2026-10-31.
- OpenAI, Realtime API (API guide), retrieved 2026-10-31.
- Google, Gemini Live API overview, last updated 2026-10-06, retrieved 2026-10-31.
- xAI, Voice Agent API (Grok Voice 3.0), retrieved 2026-10-31.
- Artificial Analysis, Speech to Speech benchmarking methodology, retrieved 2026-10-31.