Use case
Voice agent
A system that answers or places phone calls and holds a spoken conversation — booking, triage, reminders, first-line support. The hardest stack in this ontology: telephony, speech recognition, a model, speech synthesis and interruption handling, all under a latency budget.
- Source
- Editorial ontology entry — no fetched document behind this page
- Evidence
- none on this page — it links to the pages that hold it
- Category
- Communications
- Typical risk
- high
- Entry
- editorial · reviewed 25 Aug 2026
01What this is
A voice agent joins telephony to streaming speech-to-text, a model that decides what to say and what to do, and text-to-speech, with barge-in so the caller can interrupt. End-to-end latency below roughly a second is what separates a usable agent from an irritating one.
A good deployment states at the start that the caller is speaking to an automated system, offers a route to a human at any point, keeps an auditable log of what it said and did, constrains its actions to a defined set, and degrades to a person rather than improvising when confidence drops.
Pitfalls: latency budgets blown by a single synchronous lookup; recognition that fails on accents or code-switching; and agents that take actions — refunds, bookings, medical triage — beyond their sanctioned scope. Recording, consent and disclosure duties all apply, and several jurisdictions treat voice as biometric data.
- Typical data
- call audio, customer data, transcripts, order data
- Solution classes it admits
- Enterprise SaaS, Private cloud, Self-hosted
02Deployment options
- ASSESSMENT
On a neutral reading of this use case, Self-hosted is a strong alternative, Private cloud is a strong alternative and Enterprise SaaS is conditional.
- ASSESSMENT
Read as a generic reading of this page, not a recommendation: no organisation, size, jurisdiction, budget or technical capability has been supplied, so wherever an option depends on one of those, it says "unknown". Ask your own question to get a verdict that accounts for them.
Private cloud
STRONG ALTERNATIVEManaged model API or private model deployment in your cloud account
- ASSESSMENT
Two patterns fit inside one account in a region you name: call a managed foundation-model API such as Amazon Bedrock, Microsoft Foundry / Azure OpenAI, or Vertex AI; or deploy an open-weight model on GPU compute you control. Your application, retrieval layer, storage, identity and logs remain in your cloud boundary in both patterns.
- ASSESSMENT
The managed-API pattern can use closed-source frontier models without buying or operating GPUs. It is usually the fastest way to build a custom workflow, but prompts and retrieved context are processed by the managed service, so model availability, retention, abuse monitoring and regional routing must be checked for the exact feature and endpoint.
- ASSESSMENT
The private-model pattern gives more control over weights, serving and network paths, and can use managed endpoints or your own containers. It also makes your team responsible for capacity, patches, model upgrades, evaluation and failover.
- ASSESSMENT
The cloud provider becomes a data processor in either pattern: you need a DPA, a documented region, and an answer on cross-region routing and where support staff can access the environment from.
- RECOMMENDATION
Start with the managed-API pattern when the workflow is custom but model operations are not the source of competitive advantage; move to private model serving only if evaluation, volume, portability or the data boundary justifies the extra operations. Your stated technical capability is "unknown".
Self-hosted
STRONG ALTERNATIVEOpen-weight models on infrastructure you operate
- ASSESSMENT
Documents, queries and embeddings stay on machines you own, using an open-weight model whose licence you review. For a brief that involves confidential documents and personal data, that removes a model-API vendor from the data path rather than governing that transfer by contract.
- ASSESSMENT
It costs you the operational work instead: a GPU server, Docker, Linux, backups and a patching routine. Your stated technical capability is "unknown", which is the attribute this option most depends on.
- ASSESSMENT
No processor agreement, subprocessor list or cross-border transfer assessment is needed for the model itself, because no third party processes the content.
- RECOMMENDATION
Recommended where local processing is preferred (you did not say so) and the content is sensitive (confidential documents and personal data).
Enterprise SaaS
CONSIDER IFFinished closed-source cloud product with enterprise controls
- ASSESSMENT
This is a complete vendor application, not a model API: examples include an enterprise assistant, coding copilot or document product with the workflow, interface, connectors and administration already built. It can use closed-source cloud models while requiring no model hosting or application engineering from your team.
- RECOMMENDATION
Choose it when the product already performs the actual workflow and its controls meet your requirements. Do not choose it only because its underlying model is strong: a finished SaaS product is less flexible than building against a managed API when your process, integrations or review steps are organisation-specific.
- ASSESSMENT
Vendor commitments are treated as unverified until we have fetched the page that makes them. Until then this option carries questions to ask, not assurances: a signed data processing agreement covering the data you will actually put in; a documented data residency commitment naming the region, in the contract rather than a blog post; a written no-training commitment for your content, including uploads and connected sources; stated retention periods and a deletion path you can exercise; an administrative audit log you can export, and SSO with group-based access control.
- RECOMMENDATION
No jurisdiction was named, so this is the check rather than the conclusion: compare the vendor's stated processing locations and subprocessor list against the cross-border transfer rules wherever you operate before uploading anything.
03Deployment stacks
04Tools by hosting option
Self-hosted3
- Libraryfaster-whisperReimplementation of Whisper on CTranslate2 with substantially lower memory use and faster inference, including int8 execution on CPU. The usual engine behind self-hosted transcription.
- HybridLiveKit AgentsFramework for real-time voice and video agents on LiveKit’s WebRTC infrastructure, with turn detection, interruption handling and pluggable speech and model providers.
- FrameworkPipecatOpen-source framework for real-time voice and multimodal agents, composing speech-to-text, model and speech synthesis services into a streaming pipeline with barge-in support.
Private cloud4
- SaaSDeepgramSpeech API for streaming and batch transcription with diarisation and keyword boosting, plus speech synthesis. Offers self-hosted deployment for regulated environments.
- Libraryfaster-whisperReimplementation of Whisper on CTranslate2 with substantially lower memory use and faster inference, including int8 execution on CPU. The usual engine behind self-hosted transcription.
- HybridLiveKit AgentsFramework for real-time voice and video agents on LiveKit’s WebRTC infrastructure, with turn detection, interruption handling and pluggable speech and model providers.
- FrameworkPipecatOpen-source framework for real-time voice and multimodal agents, composing speech-to-text, model and speech synthesis services into a streaming pipeline with barge-in support.
Vendor cloud5
- SaaSAssemblyAISpeech-to-text API with diarisation, summarisation and topic detection over recorded and streaming audio, with region options and per-second pricing.
- SaaSDeepgramSpeech API for streaming and batch transcription with diarisation and keyword boosting, plus speech synthesis. Offers self-hosted deployment for regulated environments.
- SaaSElevenLabsSpeech synthesis and voice cloning API with multilingual voices, plus a conversational agent product. Voice cloning carries consent and likeness obligations.
- HybridLiveKit AgentsFramework for real-time voice and video agents on LiveKit’s WebRTC infrastructure, with turn detection, interruption handling and pluggable speech and model providers.
- SaaSVapiHosted platform for building telephone voice agents, bundling telephony, speech recognition, model orchestration and synthesis behind one API with call analytics.
On-premise (enterprise plan)2
- SaaSDeepgramSpeech API for streaming and batch transcription with diarisation and keyword boosting, plus speech synthesis. Offers self-hosted deployment for regulated environments.
- Libraryfaster-whisperReimplementation of Whisper on CTranslate2 with substantially lower memory use and faster inference, including int8 execution on CPU. The usual engine behind self-hosted transcription.
05Compliance hot spots
This use case usually raises personal data, sensitive data, consent, transparency, human oversight, automated decision-making, sector rules, retention, vendor jurisdiction.
- Sensitive data
- No published jurisdiction page names this topic yet
- Consent
- China (mainland)
- Transparency
- United KingdomJapanSouth Korea
- Human oversight
- No published jurisdiction page names this topic yet
- Automated decision-making
- European UnionUnited KingdomSouth Korea
- Sector rules
- No published jurisdiction page names this topic yet
- Retention
- Hong Kong
- Vendor jurisdiction
- China (mainland)
06Example questions
Each of these opens the question box with the text already in it. The answer is researched for your organisation, not for this page.
07Related use cases
- CommunicationsTranslationTranslating documents, messages and interfaces between languages while keeping terminology and formatting intact. In East Asian and European operations this is often the highest-volume AI workload in the company.
- CommunicationsMeeting transcriptionTurning internal meetings into a transcript, a summary and a list of actions. Usually a bot that joins the call, or an app that records the room. Value comes from the follow-up, not the transcript.
- CommunicationsEmail draftingDrafting replies in the inbox from the thread, the CRM record and the company style. Small per message, large in aggregate — and the place where a wrong statement leaves the building fastest.
- CommunicationsCRM call summariesWriting the call back into the CRM: a summary, the next step, the fields a rep would otherwise type. The point is not the transcript but that the record exists at all and is consistent across the team.
- CommunicationsCall transcriptionTranscribing telephone calls — support lines, sales calls, advice sessions — usually from a contact-centre or telephony platform rather than a meeting app. Narrow-band audio and regulated recording rules make it a distinct problem.
- CommunicationsAI customer-support agentAn assistant that answers customer questions from your help content and account systems, resolves what it can, and hands the rest to a person with context attached. Deflection rate matters less than what happens on the cases it cannot close.
Improve this page
Sign in to contribute
From the field
0 deployments · 0 questions
Nobody has reported deploying this here yet, and no question has been opened against this page. Both appear once a reviewer accepts them.