Voice AI Agents for Call Centres over VoIP and SIP: Architecture, Risks and a Pilot Plan

ApexlyTech Team 11 min read
Voice AI Agents for Call Centres over VoIP and SIP: Architecture, Risks and a Pilot Plan

TL;DR: A voice AI agent answers or places phone calls and holds a spoken conversation, using speech recognition (ASR), a large language model (LLM) and speech synthesis (TTS), or a single speech-to-speech model. In a VoIP contact centre it connects like any other endpoint: calls arrive over a SIP trunk, your PBX or a cloud voice platform streams the call audio to the AI over a WebSocket or RTP, and the AI's audio is streamed back. What makes it feel natural is latency and barge-in, letting the caller interrupt. What makes it safe is everything around the model: disclosure that the caller is talking to AI, lawful recording, keeping card data out of transcripts, tight limits on what the agent can do, and a fast hand-off to a person. Start with one narrow call type, measure it against your current numbers, and keep a human route open on every call.

Where voice agents work, and where they do not

Voice agents do best on calls that are frequent, follow a predictable shape, and end in something the system can check or do:

  • Answering and routing. Understanding why someone is calling in their own words, instead of a menu of eight options, and passing them to the right team with a summary.
  • Status and information. Order and delivery status, appointment times, opening hours, balances, and policy questions answered from an approved knowledge base.
  • Simple transactions. Booking, rescheduling or cancelling appointments; confirming deliveries; taking a callback request.
  • Overflow and out-of-hours. Handling the simple calls when queues are long, so people take the complex ones.
  • After-call work. Transcribing and summarising calls handled by people, filling the CRM record and tagging the reason for the call. This is often the safest place to start, because the AI never speaks to the customer.

They fit poorly where the caller is distressed or vulnerable, where the conversation involves complex judgement or negotiation, where a wrong answer has legal or safety consequences, or where accents, languages or line quality are outside what your chosen speech models handle well. Test with real recordings from your own callers before deciding.

The architecture

Telephony: SIP, RTP and the trunk

Most contact centres already run on SIP (RFC 3261) for call signalling, with audio carried as RTP (RFC 3550) and the media described in SDP (RFC 8866). Calls reach you over a SIP trunk from a carrier, into a PBX or softswitch such as Asterisk, FreeSWITCH or Kamailio-based platforms, or into a cloud voice provider. Encrypt both halves: SIP over TLS for signalling and SRTP (RFC 3711) for media. Keypad presses usually travel as RTP events (RFC 4733), which matters if callers enter account numbers. Telephone audio is typically narrowband, 8 kHz G.711; wideband codecs such as Opus (RFC 6716) are used on WebRTC and some SIP legs. Speech models trained on wideband audio may need testing on 8 kHz calls specifically.

Getting the audio to the AI

There are three common integration patterns:

  • Media streaming from the PBX or cloud platform. Asterisk's AudioSocket is a simple TCP protocol that sends and receives real-time audio, as signed linear 16-bit, 8 kHz mono PCM, plus DTMF. Asterisk's ARI External Media forwards a bridge's media to a destination you specify over RTP. On Twilio, Media Streams open a WebSocket per stream, carrying base64-encoded 8 kHz μ-law audio, and a bidirectional stream lets you send audio back into the call.
  • A managed speech layer. Some platforms handle speech recognition and synthesis for you and send your application text. Twilio's ConversationRelay, for example, converts the caller's speech to text for your application and speaks your text replies, with settings for interruption and for ignoring short backchannel sounds such as "uh-huh".
  • SIP straight to a speech-to-speech model. Some AI providers now accept SIP calls directly. OpenAI's Realtime API, for example, documents pointing a SIP trunk at its SIP endpoint, receiving a webhook for each incoming call, accepting it via the API, and transferring the call with a SIP REFER.

Each pattern trades control for convenience. Streaming to your own service gives you the most control over models, data handling and logging; a direct SIP connection to a model provider is the quickest to stand up but puts more of the call path in one vendor's hands.

The conversation loop

A cascaded pipeline looks like this:

  1. Voice activity detection (VAD) decides when the caller is speaking and when they have finished a turn. Open-source detectors such as Silero VAD process a 30 ms chunk in well under a millisecond on a CPU, so detection itself is not the bottleneck; choosing how long a pause counts as "finished" is the hard part.
  2. Streaming ASR turns speech into text, with partial results while the caller is still talking.
  3. The LLM decides what to say or do, with the conversation history, retrieved knowledge and a short list of tools (look up an order, book a slot, transfer).
  4. Streaming TTS starts speaking the first words of the reply before the whole sentence is generated.

Speech-to-speech models collapse these steps into one, which can reduce latency and keep tone of voice, but makes it harder to inspect and filter what is said before it is spoken. Many teams keep a text step so that responses can be checked against rules.

Barge-in

People interrupt, and a voice agent that keeps talking over them feels broken. Barge-in means: when VAD detects the caller speaking during playback, stop the TTS immediately, flush any audio already queued in the call path, and treat what the caller said as the next turn. Platforms provide the mechanics; Twilio's Media Streams, for instance, accept a clear message that interrupts audio already sent. The tuning is the hard part: coughs, background noise and "mm-hm" should not stop the agent, while "no, wait" should.

The latency budget

Conversation has tight timing. A study of turn-taking across ten languages, Stivers and colleagues in PNAS (2009), found that the most common gap between one speaker finishing and the next starting falls between 0 and 200 milliseconds. Telephone networks add their own delay: ITU-T Recommendation G.114 notes that keeping one-way delay below 150 ms gives most applications essentially transparent interactivity, and recommends not exceeding 400 ms for general network planning. A voice agent cannot reach human-level gaps, but every stage adds to the silence the caller hears. Measure and budget each one:

StageWhat adds delayHow to reduce it
Network and telephonyCarrier path, jitter buffers, transcoding, distance to the AI regionHost the AI close to your media servers; avoid unnecessary transcoding
End-of-turn detectionThe silence threshold before the agent decides the caller has finishedTune per use case; use semantic end-of-turn signals where available
ASRTime to the final transcriptStreaming recognition; start reasoning on stable partial results
LLMTime to first token, tool calls, retrievalSmaller or faster models for routine turns; short prompts; cache lookups
TTSTime to first audioStreaming synthesis; speak a short acknowledgement while a slow tool call runs

Measure from the caller's side, on real phone lines, at the 95th percentile rather than the average. A pilot that feels fine on a laptop over Wi-Fi can feel slow on a mobile call routed through two carriers.

Risks and the controls that address them

Telling callers they are speaking to AI

In the EU, Article 50(1) of the AI Act has applied since 2 August 2026: AI systems intended to interact directly with people must be designed so that people are informed they are interacting with an AI system, unless that is obvious from the context. In the US, the Federal Communications Commission ruled in February 2024 (FCC 24-17) that the Telephone Consumer Protection Act's restrictions on "artificial or prerecorded voice" cover AI-generated voices, so outbound AI calls covered by the TCPA need the called party's prior express consent unless an exemption applies. The FCC has also proposed, but at the time of writing not finalised, rules on disclosure for AI-generated calls. The simplest safe practice everywhere: say at the start of the call that the caller is speaking with an automated assistant, and offer a person.

Recording and transcription

A voice agent creates recordings or transcripts by design, so recording law applies. Some US states require the consent of all parties to record a confidential communication; California's Penal Code section 632 is a well-known example. Under the GDPR and UK GDPR you need a lawful basis for processing call audio and transcripts, a clear notice, and retention limits. Decide what you store (audio, transcript, summary), for how long and who can access it, and announce recording at the start of the call. Where recordings must be shared between systems, SIPREC (RFC 7866) is the standard way to send call media and metadata to a recorder.

Card data and other sensitive information

Never let card numbers or security codes into prompts, transcripts or model logs. The PCI Security Standards Council's information supplement on protecting telephone-based payment card data says that if sensitive authentication data is received and recorded, it must be rendered unrecoverable once authorisation completes, and treats preventing its recording as a best practice. Hand payment steps to a PCI-compliant DTMF capture or secure payment IVR, and pause transcription during them.

Wrong answers and manipulation

LLMs can state wrong things confidently; OWASP lists misinformation (LLM09) and prompt injection (LLM01) among its top risks, and excessive agency (LLM06) when a model has more tools and permissions than it needs. On a phone line, a caller can try to talk the agent into a refund or into revealing another customer's details. Controls: answer only from approved sources, give the agent a small set of tools with server-side checks (identity verification, amount limits), make refunds and account changes require verification that the model cannot bypass, and log every tool call.

Escalation to people

Design the hand-off first, not last. Escalate when the caller asks for a person, when confidence is low, after repeated misunderstandings, when sentiment turns negative, and for any topic on a "never automate" list such as complaints, bereavement or vulnerability. Transfer with context: a short summary and the data already collected, passed to the agent's screen, so the caller does not repeat themselves. In SIP, a transfer is typically a REFER to the human queue or a bridge in your PBX.

Caller identity and spoofing

Caller ID can be spoofed, so a phone number alone is weak proof of identity. STIR/SHAKEN (RFC 8224, RFC 8225 and RFC 8588) lets carriers sign and verify caller identity in some markets, but coverage varies. Do not use voice or caller ID alone to authorise sensitive actions; use one-time codes or knowledge checks your systems verify.

How to run a pilot

  1. Choose one call type with high volume and clear success criteria, for example "where is my order?" or appointment rescheduling.
  2. Measure the baseline from your own data: containment today, average handling time, abandonment, repeat calls within seven days and customer satisfaction for that call type.
  3. Write the boundaries: what the agent may do, what it must never do, when it must transfer, and what it says at the start of every call.
  4. Test offline first with real recordings (with appropriate consent and redaction): accents, noise, interruptions and edge cases.
  5. Go live on a slice of traffic, such as a percentage of calls or out-of-hours only, with an always-available "speak to a person" route.
  6. Review calls every day for the first weeks: transcripts, transfers, tool calls and complaints. Fix prompts, knowledge and thresholds from what you find.
  7. Decide with numbers: compare against the baseline, including the cost per resolved call and the effect on human agents' workload, then expand, adjust or stop.

For guardrails on AI agents in general, see our guide to AI agents for business, and for the regulatory timeline in the EU, our EU AI Act guide.

How ApexlyTech can help

Apex Telephony is our VoIP and SIP platform, with smart routing, visual IVR and AI-powered features, and we build integrations between telephony, CRM and AI services. We can help you connect a voice agent to your SIP infrastructure, design the hand-off to your team, and set up recording, redaction and logging that respect the rules above. See our AI and automation and cloud solutions services, or contact us.

Frequently asked questions

Do I have to tell callers they are talking to an AI?

In the EU, yes: since 2 August 2026, Article 50(1) of the AI Act requires people to be informed they are interacting with an AI system unless it is obvious. Elsewhere, rules vary, but disclosing at the start of the call and offering a person is the safe default.

Can a voice AI agent work with our existing PBX?

Usually, yes. Most PBXs and cloud voice platforms can stream call audio to an external service, for example Asterisk's AudioSocket or External Media, or Twilio's Media Streams, and some AI providers accept SIP calls directly.

What is barge-in?

Letting the caller interrupt the agent. When the caller starts speaking, the agent stops its speech, clears queued audio and listens. Good tuning ignores coughs and short sounds like "mm-hm".

How fast does a voice agent need to respond?

As fast as you can make it. In human conversation the most common gap between turns is 0 to 200 milliseconds, so every stage of the pipeline matters. Measure the silence callers actually hear, on real phone lines, at the 95th percentile.

Can a voice agent take card payments?

It should not handle card numbers itself. Hand the payment step to a PCI-compliant DTMF capture or secure payment system and pause recording and transcription during it.

Sources