Voice agent (telephony + STT + LLM + TTS)

Architecture and data flow

Organisation size

20–5000 employees

Data classes

personal, confidential

Constraints

high, repetitive call volume with a narrow set of intents; a SIP trunk or a telephony provider that offers one; recording consent and call-handling rules settled in advance; a 48 GB GPU or better; speech and language models compete for the same card; an engineering team — this is a build, not a configuration exercise

Industries

Telecommunications, Retail, Logistics, Healthcare, Hospitality

Jurisdictions

any

• On-premise single 48 GB GPU server

  GPU

  NVIDIA L40S 48 GB (rack) or NVIDIA RTX 6000 Ada 48 GB (office workstation)

  VRAM

  48 GB

  System RAM

  128 GB

  Storage

  4000 GB

  CPU

  24–32-core x86 server CPU (AMD EPYC or Intel Xeon Scalable)

  Form factor

  Rack server

  Indicative costUS$12,000 – US$22,000

  indicative build cost for the complete machine, USD, Aug 2026 — verify with a local supplier

  The step up when answer quality matters more than price: a 32B model at 4-bit with generous KV cache, or a 14B model at fp16 with a long context. ECC memory and a standard 2U chassis mean it belongs in a rack or comms room rather than under a desk.

• Cloud GPU instance — 1–2 × NVIDIA A100 80 GB

  GPU

  NVIDIA A100 80 GB (AWS P4d, Azure NDasr A100 v4, GCP A2)

  VRAM

  80 GB

  System RAM

  256 GB

  Storage

  2000 GB

  CPU

  24–48 vCPU

  Form factor

  Cloud instance

  Indicative costUS$2 – US$12

  indicative on-demand rental, USD per hour, Aug 2026 — verify against the provider price list for your region

  For 32B models with long context, or 70B-class models on two cards. The band covers one to two GPUs and the gap between committed-use and on-demand rates. Check that the GPU family you need exists in the region you are required to stay in before committing to a design.

API integrationapi-integration

development

Compliance and governancecompliance-governance

compliance

Dockerdocker

infrastructure

Linuxlinux

operations

LLM evaluationllm-evaluation

ml

LLM inferencellm-inference

ml

Networkingnetworking

networking

Pythonpython

development

Speech to textspeech-to-text

ml

Telephonytelephony

networking

Commands are copied from each project’s own documentation, and the page they came from is linked under the step. 0 of 7 steps currently open an evidence record. The rest are linked to their source; §10 says which of those documents were fetched and which were fetched without their anchor being found — two different states, named differently there.

1. 01

   Pick three intents and write the scripts

   Assessment

   Voice agents fail by trying to hold a conversation. Choose the three highest-volume calls with a deterministic shape — booking confirmation, delivery status, opening hours — and write the dialogue including the failure lines. Anything else is transferred immediately.

2. 02

   Build a browser prototype before touching telephonyversion-sensitive

   Assessment

   LiveKit's quickstart takes you from nothing to "a voice assistant that you can speak to in your terminal, browser, telephone, or native app" — start in the browser, where iteration is fast and there is no carrier in the loop. Pipecat's CLI scaffold does the same job.

   lk agent init my-agent --template agent-starter-python

   from the LiveKit quickstart

   uv tool install "pipecat-ai[cli]"
   pipecat init quickstart

   the Pipecat equivalent

   Source documentation

3. 03

   Replace the hosted services with local ones, one at a timeversion-sensitive

   Assessment

   Both frameworks ship with hosted STT, LLM and TTS wired in. Swap them individually and measure after each change: local speech to text first, then the model, then the voice. Doing all three at once makes a latency regression impossible to attribute.

   pip install faster-whisper

   docker run --runtime nvidia --gpus all \
       -v ~/.cache/huggingface:/root/.cache/huggingface \
       --env "HF_TOKEN=$HF_TOKEN" \
       -p 8000:8000 \
       --ipc=host \
       vllm/vllm-openai:latest \
       --model Qwen/Qwen3-0.6B

   from the vLLM docs — substitute the conversation model

   Source documentation

4. 04

   Budget the latency and measure every hop

   Assessment

   Aim for under a second from end of speech to start of audio. Measure each hop separately: endpointing, transcription, time to first token, first audio chunk. Streaming everywhere is what makes this achievable; any component that waits for a complete result will eat the entire budget on its own.

5. 05

   Connect the SIP trunk

   Assessment

   Add the phone number, inbound routing and a transfer path to a human queue that works on the first attempt. Test from real mobile networks, not only from a softphone — codec and packet loss on a mobile call change transcription accuracy noticeably.

   Source documentation

6. 06

   Make transfer the easy path

   Assessment

   "Let me put you through" must work on any signal of confusion: a repeated question, silence, a raised voice, or the word "agent". Pass the transcript and the caller id so the person does not start from nothing. Callers forgive a limited agent; they do not forgive a loop they cannot escape.

   docker volume create n8n_data
   
   docker run -it --rm \
    --name n8n \
    -p 5678:5678 \
    -e GENERIC_TIMEZONE="<YOUR_TIMEZONE>" \
    -e TZ="<YOUR_TIMEZONE>" \
    -e N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS=true \
    -e N8N_RUNNERS_ENABLED=true \
    -v n8n_data:/home/node/.n8n \
    docker.n8n.io/n8nio/n8n

   for post-call actions

   Source documentation

7. 07

   Pilot on a fraction of calls, with a way back

   Assessment

   Route five per cent of one intent to the agent, out of hours first. Watch containment, transfer rate and call abandonment together, and keep a switch that sends everything back to the queue. Transcribe every call for review — with consent handled as below.

   Source documentation

Structured issue-spotting to support your own review — not legal advice. Verify against the cited primary sources and your counsel.

• A hosted voice-agent platform

  Using the managed versions of these frameworks with hosted speech and model services removes most of the latency engineering and all of the GPU work.

  • — Materially faster to a working agent, and usually better voice quality
  • — Call audio and transcripts go to processors — consent, retention and residency questions
  • — Per-minute pricing that grows with call volume

  livekit-agentspipecat

• Deflect to messaging instead

  A chat assistant with a clean handoff solves much of the same demand at a fraction of the difficulty, because text has no latency budget and no consent problem.

  • — Far simpler and quicker
  • — Does not serve callers who will only use the phone

  dify

• Transcribe and summarise calls without answering them

  Much of the value people expect from a voice agent is really call notes. That recipe is two difficulty points easier and carries far less risk.

  • — No real-time constraint at all
  • — Does not reduce call volume

  faster-whisperwhisperx

• Self-hostedMeeting and call transcription to CRM notesfaster-whisper or WhisperX transcribes and separates speakers on your own GPU, a local model writes the summary and the action items against a fixed schema, and n8n files the result against the right CRM record with a person approving before it is saved.
• Self-hostedCustomer-support agent with knowledge base and handoffDify self-hosted as the agent builder and knowledge base, a local model on vLLM behind it, pgvector or Qdrant for retrieval, and n8n for the handoff into your helpdesk. Scoped deliberately: a defined topic list, a refusal path, and a human queue.

Not yet in the ledger

1. LiveKit Agents — Voice AI quickstart

   Docs

   not yet fetched

2. Pipecat — Quickstart

   Docs

   not yet fetched

3. faster-whisper — README (installation and usage)

   Readme

   not yet fetched

4. WhisperX — README (setup and diarisation)

   Readme

   not yet fetched

5. vLLM — Using Docker

   Docs

   not yet fetched

6. vLLM — Quickstart (online serving)

   Docs

   not yet fetched

7. n8n — Install with Docker

   Docs

   not yet fetched

Deployed this stack, or hit something this page does not cover? Corrections, sources and implementation reports are what keep a recipe worth reading.

If you would rather not build it, we can introduce a forward-deployed engineer who has deployed this stack before. The enquiry form starts from this recipe.