Local LLM inference server (Ollama / vLLM)

Architecture and data flow

Organisation size

1–500 employees

Data classes

confidential, personal, public

Constraints

at least one application that needs a model endpoint; a GPU sized for the model you intend to serve; a place to run containers and someone to keep them running

Industries

Technology, Professional services, Financial services, Other

Jurisdictions

any

• On-premise single 24 GB GPU server

  GPU

  NVIDIA RTX 4090 24 GB (or NVIDIA L4 24 GB for a rack-mounted, 72 W alternative)

  VRAM

  24 GB

  System RAM

  64 GB

  Storage

  2000 GB

  CPU

  16-core x86 server CPU (AMD EPYC 7003/9004 or Intel Xeon Scalable)

  Form factor

  Tower server

  Indicative costUS$4,000 – US$9,000

  indicative build cost for the complete machine, USD, Aug 2026 — verify with a local supplier

  The default box for a 20–60 person firm. Fits a 14B model at 4-bit with roughly 8 GB of KV cache left for concurrent chat, or an 8B model at fp16. NVMe storage sized for the model cache plus a document corpus and its embeddings. Add a UPS and an offsite backup target — this machine holds the whole knowledge base.

• Cloud GPU instance — 1 × NVIDIA L4 24 GB

  GPU

  NVIDIA L4 24 GB (AWS G6, Azure NVadsA10/NCads equivalents, GCP G2)

  VRAM

  24 GB

  System RAM

  64 GB

  Storage

  1000 GB

  CPU

  8–16 vCPU

  Form factor

  Cloud instance

  Indicative costUS$1 – US$2

  indicative on-demand rental, USD per hour, Aug 2026 — verify against the provider price list for your region

  The private-cloud counterpart of `onprem-small-24gb`: same model sizes, no capital outlay, and a region you choose explicitly. Running it continuously for a year usually costs more than buying the equivalent box, so it suits pilots, bursts and firms without a server room. The cloud provider becomes a data processor — a DPA and a documented region are required.

• On-premise single 48 GB GPU server

  GPU

  NVIDIA L40S 48 GB (rack) or NVIDIA RTX 6000 Ada 48 GB (office workstation)

  VRAM

  48 GB

  System RAM

  128 GB

  Storage

  4000 GB

  CPU

  24–32-core x86 server CPU (AMD EPYC or Intel Xeon Scalable)

  Form factor

  Rack server

  Indicative costUS$12,000 – US$22,000

  indicative build cost for the complete machine, USD, Aug 2026 — verify with a local supplier

  The step up when answer quality matters more than price: a 32B model at 4-bit with generous KV cache, or a 14B model at fp16 with a long context. ECC memory and a standard 2U chassis mean it belongs in a rack or comms room rather than under a desk.

• Apple Silicon workstation (unified memory)

  GPU

  Apple M4 Max (up to 128 GB unified) or M3 Ultra (up to 256 GB unified)

  VRAM

  96 GB

  System RAM

  128 GB

  Storage

  2000 GB

  CPU

  Apple M4 Max / M3 Ultra

  Form factor

  Workstation

  Indicative costUS$4,500 – US$12,000

  indicative build cost for the complete machine, USD, Aug 2026 — verify with a local supplier

  Quiet, standard-socket power draw, and a large memory pool — about 75% of installed unified memory is addressable by the GPU, which is where the 96 GB figure comes from on a 128 GB M4 Max. Memory bandwidth, not capacity, sets the ceiling: excellent for a pilot, a partner workstation or a single heavy user, weak for 40 people at once. Runs Ollama, llama.cpp and MLX; does not run CUDA builds of vLLM.

Dockerdocker

infrastructure

Linuxlinux

operations

LLM inferencellm-inference

ml

MLOpsmlops

operations

Networkingnetworking

networking

Security hardeningsecurity-hardening

security

Commands are copied from each project’s own documentation, and the page they came from is linked under the step. 0 of 6 steps currently open an evidence record. The rest are linked to their source; §10 says which of those documents were fetched and which were fetched without their anchor being found — two different states, named differently there.

1. 01

   Size the model against the GPU before installing anything

   Assessment

   Work out weights, KV cache and overhead for the model, context length and concurrency you expect. A 14B model at 4-bit needs roughly 7 GB of weights; the KV cache at 8k context for ten concurrent requests adds several more. Decide the quantisation now — it changes which card you need and how the answers read.

2. 02

   Start vLLM as an OpenAI-compatible serverversion-sensitive

   Assessment

   vLLM "can be deployed as a server that implements the OpenAI API protocol", starting on port 8000 by default. Run the official image with the GPU attached and the Hugging Face cache mounted so restarts do not re-download the weights.

   docker run --runtime nvidia --gpus all \
       -v ~/.cache/huggingface:/root/.cache/huggingface \
       --env "HF_TOKEN=$HF_TOKEN" \
       -p 8000:8000 \
       --ipc=host \
       vllm/vllm-openai:latest \
       --model Qwen/Qwen3-0.6B

   from the vLLM docs — substitute your model

   vllm serve Qwen/Qwen2.5-1.5B-Instruct

   the same thing without Docker

   Source documentation

3. 03

   Or start Ollama, if simplicity matters more than throughputversion-sensitive

   Assessment

   Ollama supports "Nvidia GPUs with compute capability 5.0+ and driver version 550 and newer". One container, a named volume, and `ollama run` to pull a model. Expect lower concurrency than vLLM on the same card.

   docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

   Source documentation

4. 04

   Put a key on the endpoint

   Assessment

   An unauthenticated inference server on an office network is a shared GPU for anyone who finds it. vLLM accepts `--api-key` or the `VLLM_API_KEY` environment variable and will accept any of several keys, which is what makes rotation possible without downtime.

   Source documentation

5. 05

   Add a gateway when more than one application calls the modelversion-sensitive

   Assessment

   LiteLLM brings up "the gateway on port 4000 and a Postgres database that stores your models, keys, and spend logs". Issue a virtual key per team or application so usage can be attributed, and so revoking one does not disturb the others.

   curl -sSL https://docs.litellm.ai/docker-compose.yml | docker compose -f - up -d

   Source documentation

6. 06

   Measure before you promise anything

   Assessment

   Record tokens per second at one, five and twenty concurrent requests, and watch GPU memory while you do it. That table is what tells you whether the model, the quantisation or the card has to change — and it is the evidence for the next hardware conversation.

   Source documentation

Structured issue-spotting to support your own review — not legal advice. Verify against the cited primary sources and your counsel.

• llama.cpp or LocalAI

  Run on CPU or mixed CPU/GPU where no suitable card exists, or on a workstation where the model has to share memory with everything else.

  • — Runs almost anywhere
  • — Much lower throughput; fine for one user, not for a team

  llama-cpplocalai

• A hosted OpenAI-compatible endpoint

  When the workload is spiky and the data allows it, a managed endpoint removes the GPU problem entirely — at the price of sending prompts to a provider.

  • — No hardware, instant capacity
  • — Prompts leave your network: DPA, region and retention terms apply

  azure-openaiaws-bedrockgoogle-vertex-ai

No alternative recipe is published yet.

Not yet in the ledger

1. vLLM — Quickstart (online serving)

   Docs

   not yet fetched

2. vLLM — Using Docker

   Docs

   not yet fetched

3. vLLM — Optimization and tuning

   Docs

   not yet fetched

4. Ollama — Docker

   Docs

   not yet fetched

5. Ollama — GPU support

   Docs

   not yet fetched

6. LiteLLM — Docker quick start (gateway)

   Docs

   not yet fetched

Deployed this stack, or hit something this page does not cover? Corrections, sources and implementation reports are what keep a recipe worth reading.

If you would rather not build it, we can introduce a forward-deployed engineer who has deployed this stack before. The enquiry form starts from this recipe.