AI coding assistant with private code

Architecture and data flow

Organisation size

5–1000 employees

Data classes

confidential

Constraints

a contractual, regulatory or policy reason the code cannot go to a hosted assistant; a 48 GB GPU for a 32B model shared by a team, or 24 GB for a smaller model; developers who will tell you honestly whether it is good enough — it may not be

Industries

Technology, Financial services, Public sector, Telecommunications

Jurisdictions

European union

• On-premise single 24 GB GPU server

  GPU

  NVIDIA RTX 4090 24 GB (or NVIDIA L4 24 GB for a rack-mounted, 72 W alternative)

  VRAM

  24 GB

  System RAM

  64 GB

  Storage

  2000 GB

  CPU

  16-core x86 server CPU (AMD EPYC 7003/9004 or Intel Xeon Scalable)

  Form factor

  Tower server

  Indicative costUS$4,000 – US$9,000

  indicative build cost for the complete machine, USD, Aug 2026 — verify with a local supplier

  The default box for a 20–60 person firm. Fits a 14B model at 4-bit with roughly 8 GB of KV cache left for concurrent chat, or an 8B model at fp16. NVMe storage sized for the model cache plus a document corpus and its embeddings. Add a UPS and an offsite backup target — this machine holds the whole knowledge base.

• On-premise single 48 GB GPU server

  GPU

  NVIDIA L40S 48 GB (rack) or NVIDIA RTX 6000 Ada 48 GB (office workstation)

  VRAM

  48 GB

  System RAM

  128 GB

  Storage

  4000 GB

  CPU

  24–32-core x86 server CPU (AMD EPYC or Intel Xeon Scalable)

  Form factor

  Rack server

  Indicative costUS$12,000 – US$22,000

  indicative build cost for the complete machine, USD, Aug 2026 — verify with a local supplier

  The step up when answer quality matters more than price: a 32B model at 4-bit with generous KV cache, or a 14B model at fp16 with a long context. ECC memory and a standard 2U chassis mean it belongs in a rack or comms room rather than under a desk.

• Cloud GPU instance — 1–2 × NVIDIA A100 80 GB

  GPU

  NVIDIA A100 80 GB (AWS P4d, Azure NDasr A100 v4, GCP A2)

  VRAM

  80 GB

  System RAM

  256 GB

  Storage

  2000 GB

  CPU

  24–48 vCPU

  Form factor

  Cloud instance

  Indicative costUS$2 – US$12

  indicative on-demand rental, USD per hour, Aug 2026 — verify against the provider price list for your region

  For 32B models with long context, or 70B-class models on two cards. The band covers one to two GPUs and the gap between committed-use and on-demand rates. Check that the GPU family you need exists in the region you are required to stay in before committing to a design.

Change managementchange-management

operations

Dockerdocker

infrastructure

Linuxlinux

operations

LLM inferencellm-inference

ml

Networkingnetworking

networking

Security hardeningsecurity-hardening

security

Commands are copied from each project’s own documentation, and the page they came from is linked under the step. 0 of 7 steps currently open an evidence record. The rest are linked to their source; §10 says which of those documents were fetched and which were fetched without their anchor being found — two different states, named differently there.

1. 01

   Write down the actual constraint

   Assessment

   Is it a client contract, a regulator, an internal policy, or a preference? The answer sets the bar: a contract that forbids third-party processing rules out hosted assistants entirely, while a policy might be satisfied by a vendor with a zero-retention commitment. Self-hosting costs quality, so be sure the constraint is real before paying for it.

2. 02

   Serve the chat model with vLLMversion-sensitive

   Assessment

   Start vLLM as an OpenAI-compatible endpoint on the shared GPU and set an API key. One 48 GB card comfortably serves in-editor chat for a small team; completion traffic is far heavier and is handled separately below.

   docker run --runtime nvidia --gpus all \
       -v ~/.cache/huggingface:/root/.cache/huggingface \
       --env "HF_TOKEN=$HF_TOKEN" \
       -p 8000:8000 \
       --ipc=host \
       vllm/vllm-openai:latest \
       --model Qwen/Qwen3-0.6B

   from the vLLM docs — substitute the model you sized for

   Source documentation

3. 03

   Run Tabby for inline completionversion-sensitive

   Assessment

   Tabby runs in Docker with the GPU attached, serving a completion model and a chat model together, and exposes its UI on port 8080. Completion latency is what developers judge: measure it at the editor, not at the server.

   docker run -d \
     --name tabby \
     --gpus all \
     -p 8080:8080 \
     -v $HOME/.tabby:/data \
     registry.tabbyml.com/tabbyml/tabby \
       serve \
       --model StarCoder-1B \
       --chat-model Qwen2-1.5B-Instruct \
       --device cuda

   from the Tabby docs — substitute the models you have chosen

   docker logs -f tabby

   Source documentation

4. 04

   Configure Continue against the local endpointversion-sensitive

   Assessment

   Continue reaches any OpenAI-compatible server: "either Continue will have a built-in provider or the API you use will be OpenAI-compatible, in which case you can use the \"openai\" provider and change the \"baseUrl\" to point to the server". Ship one `config.yaml` to the team so everybody gets the same models and rules.

   Source documentation

5. 05

   Turn off telemetry and install offline where required

   Assessment

   For an air-gapped or strictly controlled environment, Continue documents the offline path: download the extension file from the releases page and install it into VS Code, turn off "Allow Anonymous Telemetry" in the user settings, point the default model at a local one, and restart the editor. Verify with a network capture rather than trusting the setting.

   Source documentation

6. 06

   Issue per-developer keys through a gatewayversion-sensitive

   Assessment

   A gateway gives you one endpoint, a key per developer and a usage log. It is also how you find out, six weeks later, whether anyone is actually using the thing — which is the question that decides whether to buy a bigger GPU. LiteLLM, cited below, is the gateway these commands run; it is optional in the sense that you can point IDEs straight at vLLM and give up the per-developer keys and the spend log with it.

   curl -sSL https://docs.litellm.ai/docker-compose.yml | docker compose -f - up -d

   Source documentation

7. 07

   Trial with volunteers and ask a blunt question

   Assessment

   Give it to five willing developers for three weeks, then ask whether they would object to it being taken away. That answer is worth more than any benchmark. If it is lukewarm, the honest options are a bigger model, a bigger GPU, or revisiting the constraint.

Structured issue-spotting to support your own review — not legal advice. Verify against the cited primary sources and your counsel.

• A hosted assistant with an enterprise agreement

  Materially better completions today. Enterprise plans offer contractual commitments on training and retention — read them against your actual constraint before ruling them out.

  • — Best-in-class quality, nothing to run
  • — Source code is processed by a third party under contract
  • — Per-seat cost that scales with the team

  github-copilotcursor

• Local models on each developer machine

  Ollama on a workstation with a small model: no shared server, no gateway, nothing to operate. Good enough for completion on modern laptops.

  • — No infrastructure at all
  • — Model size limited by the laptop, and no central view of usage or configuration

  ollamacontinue

No alternative recipe is published yet.

Not yet in the ledger

1. Tabby — Docker installation

   Docs

   not yet fetched

2. Continue — How to run Continue without internet

   Docs

   not yet fetched

3. Continue — How to self-host a model

   Docs

   not yet fetched

4. vLLM — Using Docker

   Docs

   not yet fetched

5. vLLM — Quickstart (online serving)

   Docs

   not yet fetched

6. LiteLLM — Docker quick start (gateway)

   Docs

   not yet fetched

Deployed this stack, or hit something this page does not cover? Corrections, sources and implementation reports are what keep a recipe worth reading.

If you would rather not build it, we can introduce a forward-deployed engineer who has deployed this stack before. The enquiry form starts from this recipe.