How to Build a Voice AI Agent That Handles Real Calls

⚡What you are actually building

A voice agent is a loop, not a model. Audio arrives from a phone call, gets transcribed to text in near-real-time, the text goes to an LLM that decides what to say (and which tools to call — look up an order, book an appointment, check a balance), the response text is synthesized to speech, and audio streams back to the caller. The loop repeats until the call ends. Every stage is a network hop with its own latency and failure modes, and the caller experiences the sum of all of them as "how smart does this thing feel."

The demo version of this is a weekend project; the production version is one of the more demanding real-time systems in applied AI. Real callers mumble, have accents, call from cars, interrupt mid-sentence, spell names, read out card numbers, and get angry. Phone networks add jitter, echo, and 8 kHz audio. Your agent has to be correct, fast, and compliant under all of it, on every call, at a per-call cost your unit economics survive.

Scope discipline matters more here than in most categories. A voice agent that books appointments for one business type, with a constrained set of tools and a clean escalation path to a human, is a shippable product. "An agent that handles any call for any business" is a research program. Define the call types, the tools, the success criteria, and the handoff conditions before choosing a single vendor.

Our voice AI agent development services→

🔗The pipeline: telephony, STT, LLM, TTS

Telephony is the front door. You need a provider that gives you programmable calls plus raw audio streaming: Twilio (Media Streams over WebSocket), Vonage, Plivo, Telnyx, and SignalWire all offer this, with per-minute pricing and regional number availability as the main axes. The audio itself is typically narrowband 8 kHz mu-law — a constraint your STT choice must handle well, because models tuned on clean microphone audio degrade noticeably on phone audio.

Speech-to-text runs streaming, not batch: partial transcripts flow while the caller is still talking, so the system can start thinking early. Deepgram and AssemblyAI are the common dedicated choices, with OpenAI and Google speech models as credible alternatives — as of writing, all publish per-minute pricing; verify current rates before modelling costs. What matters beyond accuracy is endpointing quality (detecting that the caller finished) and latency of final transcripts on telephony audio, with your vocabulary — test with your actual domain words, names, and account-number formats, not vendor demos.

The LLM is the brain and usually the latency bottleneck. It receives the running transcript plus conversation state, produces a response, and calls tools against your systems. Favor models with fast time-to-first-token and stream everything; keep the system prompt tight; and design tool calls to return in well under a second, because a slow CRM lookup reads to the caller as the agent freezing. Response style needs its own engineering: spoken answers are short, number formatting is verbal ("back on Tuesday the third", not a date string), and the model must never read out raw IDs or URLs.

Text-to-speech closes the loop, streaming audio sentence by sentence so playback starts before the full response is synthesized. ElevenLabs, Cartesia, OpenAI, and others compete here; evaluate on time-to-first-byte, voice quality at phone bandwidth, and how naturally the voice handles interruptions. An increasingly viable alternative architecture is speech-to-speech models (the OpenAI Realtime API and Gemini Live class), which collapse STT, LLM, and TTS into one streaming model — lower latency and better prosody, at the price of less control over each stage and, as of writing, a younger tooling ecosystem.

⏱️The latency budget, stage by stage

Humans read pauses. In natural conversation, gaps of roughly 200 to 500 milliseconds signal a normal turn; once the gap after a caller finishes speaking stretches past about a second, the agent feels slow, and past two seconds callers start repeating themselves or hanging up. Your entire engineering goal is keeping the caller-perceived response gap inside that window, and the way to do it is a budget: assign each stage a maximum, measure each stage in production, and treat regressions as outages.

The numbers below are labelled planning ranges from typical production systems as of writing, not benchmarks — actual figures depend on providers, regions, and load, so measure your own stack. The sum is what matters: if your stages add to 1,800 ms, no amount of prompt work will make the agent feel responsive.

The tactics that buy back budget: speculative response generation while the caller is finishing (start the LLM on a partial transcript, discard if the final transcript diverges), sentence-level streaming through the whole chain (the TTS starts on the first clause while the LLM is still generating the second), keeping a filler-and-acknowledgment layer for slow tool calls ("Let me check that for you" — generated instantly, while the lookup runs), and colocating your orchestrator with your STT and TTS providers to shave network hops.

✋Interruption handling and barge-in

Barge-in is the feature that separates a phone tree with a neural voice from an agent: when the caller starts talking while the agent is speaking, the agent must stop, discard its remaining queued audio, and listen. Implementation-wise this is a media-plane concern — voice activity detection on the inbound stream triggers cancellation of the outbound TTS stream, buffer flushes down the telephony leg, and a rollback of conversation state so the half-spoken sentence is not treated as said. Every stage of the pipeline needs a cancellation path, and most early builds are missing at least one.

The subtlety is false positives. Background noise, the caller saying "mm-hmm" while listening, and echo of the agent own voice all look like speech to a naive VAD. You need echo cancellation (telephony providers offer it; verify it is actually enabled on your media path), configurable VAD sensitivity, and backchannel handling — short affirmations should not interrupt the agent, which usually means requiring sustained speech of a few hundred milliseconds before triggering barge-in.

Turn-taking is the other half of the same problem. Endpointing too aggressively cuts callers off mid-thought ("my number is four-one..." pause "...five"); too conservatively adds dead air. The robust pattern combines acoustic endpointing with semantic hints from the transcript — an unfinished list, a partial phone number, a trailing conjunction all predict the caller is not done. As of writing, several STT and voice-platform vendors ship semantic endpointing features; they are worth evaluating before building your own heuristic stack.

Test this like a load test, not a demo: scripted callers that interrupt at fixed offsets, noisy audio tracks, hold-music and side-conversation scenarios, and double-talk where both parties speak simultaneously. The agents that feel magical in production are the ones whose teams have a regression suite of adversarial audio.

📞Telephony and platform choices, honestly

You have three architectural options, and the right one depends on how much of the problem you want to own. Option one: voice-agent platforms (Vapi, Retell, Bland, and similar) bundle telephony, orchestration, STT/LLM/TTS wiring, and interruption handling behind an API, priced per minute on top of the underlying model costs. They are the fastest path to production and the right choice for validating a use case; the trade-offs are per-minute platform margins, less control over the pipeline internals, and dependency on their roadmap.

Option two: open orchestration frameworks — LiveKit Agents and Pipecat are the leading open-source choices as of writing — give you the real-time media plumbing, VAD, turn-taking, and provider integrations as libraries you host yourself, wired to a telephony provider for calls. You own latency tuning, model choices, and data flow (which matters for compliance), at the cost of operating a stateful real-time media service.

Option three: fully custom on raw telephony APIs. Justified mainly when compliance or unit economics at scale demand total control; it means owning echo, jitter, reconnection, and every vendor edge case yourself. Most teams that choose this underestimate the media-layer engineering by a wide margin.

⚖️Compliance: recording, consent, and disclosure

Call recording and disclosure law is jurisdictional, and getting it wrong is a legal problem, not an engineering ticket. In the United States, most states allow one-party consent, while a minority — including California, Florida, and Pennsylvania — require all-party consent for recording. Because you cannot reliably know where every caller is, the practical standard for a business operating nationally is to disclose recording and AI usage at the start of every call, in plain language, before any data is collected. This is not legal advice; have counsel review your disclosure language and call flows for every market you operate in.

Outbound calling adds a second regime: the TCPA and related rules govern autodialed and prerecorded or artificial-voice calls, and as of writing, regulators have been explicit that AI-generated voices fall under artificial-voice provisions — consent requirements, opt-out handling, calling-hour restrictions, and Do-Not-Call list hygiene all apply. Verify the current rules before any outbound campaign; enforcement here is active and penalties are per call.

Data handling completes the picture. If calls touch payment card data, PCI-DSS applies — the standard pattern is pausing transcription and recording during card capture or using DTMF masking so digits never enter your audio pipeline at all. Health information pulls in HIPAA; EU callers pull in GDPR with its own consent and retention rules. Operationally: define retention periods for recordings and transcripts, encrypt at rest, restrict access, and log who listened to what — because a voice platform is a repository of everything your customers said out loud.

💰Cost per call: the honest math

Voice agent economics are per-minute, and the stack has four meters running simultaneously. The ranges below are labelled planning ranges assembled from published provider pricing as of writing — verify current rates before committing, because this market reprices frequently. The point of the table is not precision; it is the shape of the math so you can model your own deal.

Worked example, five-minute support call at mid-range assumptions: telephony around $0.05–$0.15, STT $0.03–$0.10, LLM $0.05–$0.20 (highly sensitive to model tier and prompt length), TTS $0.15–$0.60 — landing in a broad band of roughly $0.30 to $1.00 per call before any platform margin. Voice-agent platforms add their own per-minute fee on top, and speech-to-speech models reprice the whole middle of the stack into one per-minute model cost. The conclusion that survives every repricing: TTS and model choice dominate, and long calls are expensive, so average handle time is a cost lever, not just a service metric.

Against that, the comparison buyers actually make: a human-handled call in a US contact center commonly costs several dollars all-in. The economics work — provided containment is real. An agent that resolves 40 percent of call types and escalates the rest cleanly is a strong business; an agent that mishandles the hard 20 percent and generates callbacks is a cost center with a voice. Model cost per resolved call, not cost per attempted call.

🧭The build plan and when to hire a team

Sequence the build so the risky parts are proven early. First: the pipeline skeleton with one call type and no tools, measured for latency end to end. Second: tool integration for the real workflow (booking, lookup, ticket creation), with timeouts and fillers. Third: interruption handling and the adversarial audio test suite. Fourth: compliance plumbing — disclosure, recording controls, retention. Fifth: escalation to humans with full context transfer, which is a feature callers judge you on. Launch narrow, measure containment and cost per resolved call, then expand call types.

Realistic timelines for a focused production agent — one business function, clean escalation, US market: six to ten weeks on a voice platform, ten to sixteen weeks on a self-hosted framework, assuming an existing API for the tools it calls. Labelled build-cost ranges at US or senior blended rates: $25,000 to $60,000 for a platform-based production agent, $60,000 to $150,000 for a self-hosted framework build with custom tooling and compliance depth, and $150,000 and up when custom telephony, multiple languages, or deep contact-center integration enter scope.

When to hire a team: when the agent will touch payments or regulated data, when containment targets are contractual, or when your unit economics require owning the pipeline. The failure modes of this category — dead air, talking over callers, mishandled escalations, compliance gaps — are all systems failures that experienced teams design out up front rather than discovering in call recordings.

Voice AI agent development at Codazz→What AI agents actually cost to run→