Private company knowledge base (self-hosted RAG)

Architecture and data flow

Organisation size

20–200 employees

Data classes

confidential, personal, privileged

Constraints

documents must not leave the company network; somewhere to put a server: an office comms room, a rack, or colocation; someone who can run Linux and Docker, in-house or on retainer; a GPU with at least 24 GB of memory for a 14B model at 4-bit

Industries

Legal, Accounting, Professional services, Financial services

Jurisdictions

any

• On-premise single 24 GB GPU server

  GPU

  NVIDIA RTX 4090 24 GB (or NVIDIA L4 24 GB for a rack-mounted, 72 W alternative)

  VRAM

  24 GB

  System RAM

  64 GB

  Storage

  2000 GB

  CPU

  16-core x86 server CPU (AMD EPYC 7003/9004 or Intel Xeon Scalable)

  Form factor

  Tower server

  Indicative costUS$4,000 – US$9,000

  indicative build cost for the complete machine, USD, Aug 2026 — verify with a local supplier

  The default box for a 20–60 person firm. Fits a 14B model at 4-bit with roughly 8 GB of KV cache left for concurrent chat, or an 8B model at fp16. NVMe storage sized for the model cache plus a document corpus and its embeddings. Add a UPS and an offsite backup target — this machine holds the whole knowledge base.

• On-premise single 48 GB GPU server

  GPU

  NVIDIA L40S 48 GB (rack) or NVIDIA RTX 6000 Ada 48 GB (office workstation)

  VRAM

  48 GB

  System RAM

  128 GB

  Storage

  4000 GB

  CPU

  24–32-core x86 server CPU (AMD EPYC or Intel Xeon Scalable)

  Form factor

  Rack server

  Indicative costUS$12,000 – US$22,000

  indicative build cost for the complete machine, USD, Aug 2026 — verify with a local supplier

  The step up when answer quality matters more than price: a 32B model at 4-bit with generous KV cache, or a 14B model at fp16 with a long context. ECC memory and a standard 2U chassis mean it belongs in a rack or comms room rather than under a desk.

• Apple Silicon workstation (unified memory)

  GPU

  Apple M4 Max (up to 128 GB unified) or M3 Ultra (up to 256 GB unified)

  VRAM

  96 GB

  System RAM

  128 GB

  Storage

  2000 GB

  CPU

  Apple M4 Max / M3 Ultra

  Form factor

  Workstation

  Indicative costUS$4,500 – US$12,000

  indicative build cost for the complete machine, USD, Aug 2026 — verify with a local supplier

  Quiet, standard-socket power draw, and a large memory pool — about 75% of installed unified memory is addressable by the GPU, which is where the 96 GB figure comes from on a 128 GB M4 Max. Memory bandwidth, not capacity, sets the ceiling: excellent for a pilot, a partner workstation or a single heavy user, weak for 40 people at once. Runs Ollama, llama.cpp and MLX; does not run CUDA builds of vLLM.

Dockerdocker

infrastructure

Identity and SSOidentity-sso

security

Linuxlinux

operations

LLM evaluationllm-evaluation

ml

LLM inferencellm-inference

ml

Networkingnetworking

networking

PostgreSQLpostgresql

data

Retrieval-augmented generationrag

ml

Security hardeningsecurity-hardening

security

Commands are copied from each project’s own documentation, and the page they came from is linked under the step. 0 of 11 steps currently open an evidence record. The rest are linked to their source; §10 says which of those documents were fetched and which were fetched without their anchor being found — two different states, named differently there.

1. 01

   Prepare the server and GPU drivers

   Assessment

   Install a current Linux LTS on the machine, then the NVIDIA driver, Docker Engine and the NVIDIA Container Toolkit so containers can see the GPU. Configure the Docker runtime and restart the daemon. Verify with `nvidia-smi` inside a container before going further — every later step assumes the GPU is visible.

   sudo apt-get install -y nvidia-container-toolkit

   NVIDIA Container Toolkit

   sudo nvidia-ctk runtime configure --runtime=docker
   sudo systemctl restart docker

   Source documentation

2. 02

   Run PostgreSQL with the pgvector extensionversion-sensitive

   Assessment

   Use the pgvector image, which is the Postgres image with the extension added, and enable the extension once in the database you will use. Put the data directory on the machine’s own storage, not a network share, and set up a nightly dump to an offsite target now rather than later — this database ends up holding the whole knowledge base.

   docker pull pgvector/pgvector:pg18-trixie

   CREATE EXTENSION vector;

   run once in each database that stores embeddings

   Source documentation

3. 03

   Serve the model with vLLMversion-sensitive

   Assessment

   vLLM "can be deployed as a server that implements the OpenAI API protocol", which is what lets the rest of the stack treat it as a drop-in endpoint. Start the official image with the GPU attached and the Hugging Face cache mounted, substituting the model you sized for — Qwen/Qwen2.5-14B-Instruct at 4-bit on a 24 GB card. Set an API key so nothing else on the network can use the GPU: pass `--api-key` or set `VLLM_API_KEY`. How much GPU memory this needs is computed for your own headcount in the hardware section above — weights from the model’s published parameter count, plus a KV cache sized for the concurrency assumed there, plus runtime overhead — and it is an assessment, not a measurement. Size the card against the typical figure, watch queue depth in the first month, and re-check before adding users.

   docker run --runtime nvidia --gpus all \
       -v ~/.cache/huggingface:/root/.cache/huggingface \
       --env "HF_TOKEN=$HF_TOKEN" \
       -p 8000:8000 \
       --ipc=host \
       vllm/vllm-openai:latest \
       --model Qwen/Qwen3-0.6B

   from the vLLM docs — replace the --model value with the model you sized for

   curl http://localhost:8000/v1/models

   confirm the server is up

   Source documentation

4. 04

   Run Open WebUIversion-sensitive

   Assessment

   Start the container with a named volume so nothing is lost on restart, and pin a version tag rather than `:main` for a deployment people depend on — the docs are explicit that `:main` and `:latest` are rolling images. The first account created becomes the admin.

   docker run -d -p 3000:8080 -v open-webui:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:main

   Source documentation

5. 05

   Point Open WebUI at the vLLM server

   Assessment

   In Admin Settings → Connections → OpenAI → Manage, add a connection with the API URL `http://localhost:8000/v1` (or `http://host.docker.internal:8000/v1` when Open WebUI is itself in a container) and the API key you set on vLLM. The model then appears in the model selector.

   Source documentation

6. 06

   Move Open WebUI onto PostgreSQL and pgvector

   Assessment

   The default SQLite file and local Chroma index do not survive real concurrency. Set the database URL and switch the vector store to pgvector, then restart. Do this before loading documents: migrating a populated instance is far more work than starting on the right store.

   DATABASE_URL=postgresql://user:password@db-host:5432/openwebui
   VECTOR_DB=pgvector
   PGVECTOR_DB_URL=postgresql://user:password@db-host:5432/openwebui

   environment variables from the scaling guide

   Source documentation

7. 07

   Turn on single sign-on

   Assessment

   Create an OIDC client in Keycloak (or your existing identity provider), then set `WEBUI_URL`, `OAUTH_CLIENT_ID`, `OAUTH_CLIENT_SECRET` and `OPENID_PROVIDER_URL` in Open WebUI. `OPENID_PROVIDER_URL` is mandatory and the redirect URI must be exactly `<your-domain>/oauth/oidc/callback`. Map identity-provider groups to Open WebUI groups so that access to a document collection follows the same rules as access to the file share.

   Source documentation

8. 08

   Harden the deployment before anyone logs in

   Assessment

   Open WebUI "is built for private, trusted networks": put it behind your VPN or a reverse proxy with TLS and IP allowlisting, and give it no public route. Set `WEBUI_SECRET_KEY` explicitly if you will ever run more than one instance, keep new accounts pending until an admin approves them, and set the secure-cookie and security-header variables.

   DEFAULT_USER_ROLE=pending

   WEBUI_SESSION_COOKIE_SECURE=true
   WEBUI_SESSION_COOKIE_SAME_SITE=strict

   Source documentation

9. 09

   Load a first document collection

   Assessment

   Build one Knowledge collection for a single team and a single, well-understood set of documents — not the whole file server. Knowledge "uses retrieval (RAG) to find the relevant chunks on demand", so the quality of extraction sets the ceiling: switch the content-extraction engine to Docling if the corpus is scanned or table-heavy. Set the embedding model once, because changing it means re-indexing everything.

   Source documentation

10. 10

    Evaluate retrieval before rollout, and again after

    Assessment

    Write 30–50 real questions with the answer and the source document recorded by a person who knows the corpus. Measure two things separately: did retrieval surface the right document, and did the answer use it. Retrieval failures are fixed by chunking, the embedding model or metadata filters; answer failures by the prompt or a larger model. Keep the question set — it is how you will judge the next upgrade.

11. 11

    Set the operating routine

    Assessment

    Nightly database dump held offsite; monthly patch window for the containers and the host; a named owner for adding and removing document collections; quarterly review of who has access to what. Watch GPU memory and queue depth — the first sign of outgrowing a 24 GB card is queuing at peak, not an error.

Structured issue-spotting to support your own review — not legal advice. Verify against the cited primary sources and your counsel.

• AnythingLLM with Ollama

  A single container plus Ollama, installed in an afternoon on a workstation. The right choice for a team under about 20 people or for proving the idea before buying a server.

  • — Simpler to run and to explain
  • — Lower throughput: Ollama serves fewer concurrent users than vLLM on the same GPU
  • — Fewer administrative controls for groups and document permissions

  anythingllmollama

• Dify

  Use Dify when the requirement grows past search into workflows and agents — approvals, multi-step tools, applications built on the corpus rather than a chat box over it.

  • — Far more capable as a builder
  • — More moving parts to operate
  • — Chat-with-documents is one feature among many rather than the product

  dify

• Onyx

  Onyx leads with connectors: if most of the knowledge lives in Google Drive, SharePoint, Slack and Confluence rather than in a folder, start there instead of building ingestion.

  • — Connector coverage and permission syncing out of the box
  • — Heavier deployment
  • — Less direct control over chunking and retrieval than a purpose-built RAG engine

  onyx

• Private-cloud RAG in one region

  The same stack on a rented GPU instance in a region you name. Choose it when there is nowhere sensible to put a server, or when the workload is seasonal.

  • — No hardware to buy, rack or replace
  • — The cloud provider becomes a processor: DPA, region commitment and access review
  • — Costs more than owning the box if it runs continuously for a year

  vllmopen-webui

• Private cloudPrivate-cloud RAG in a single regionA GPU instance in one region running vLLM and Open WebUI, a managed PostgreSQL with pgvector for chats and embeddings, object storage for the original files, and your existing identity provider for sign-in. Same software as the on-premise recipe; different trust boundary.
• Self-hostedPrivate ChatGPT on your own serverOllama serving an open-weight model on one GPU, Open WebUI in front of it for accounts, chats and admin controls, both in Docker on a single machine. Deliberately the smallest thing that works: no vector database, no connectors, no cluster.

Not yet in the ledger

1. Open WebUI — Quick Start (Docker)

   Docs

   not yet fetched

2. Open WebUI — Scaling to multiple users and instances

   Docs

   not yet fetched

3. Open WebUI — SSO (OAuth, OIDC, Trusted Header)

   Docs

   not yet fetched

4. Open WebUI — Hardening

   Docs

   not yet fetched

5. Open WebUI — Knowledge (workspace)

   Docs

   not yet fetched

6. Open WebUI — Retrieval Augmented Generation

   Docs

   not yet fetched

7. Open WebUI — Connect a provider: vLLM

   Docs

   not yet fetched

8. Open WebUI — LICENSE (branding condition)

   Readme

   not yet fetched

9. vLLM — Quickstart (online serving)

   Docs

   not yet fetched

10. vLLM — Using Docker

    Docs

    not yet fetched

11. pgvector — README (installation and getting started)

    Readme

    not yet fetched

12. Keycloak — Getting started with Docker

    Docs

    not yet fetched

13. NVIDIA — Installing the NVIDIA Container Toolkit

    Docs

    not yet fetched

14. RAGFlow — README (start up the server)

    Readme

    not yet fetched

Deployed this stack, or hit something this page does not cover? Corrections, sources and implementation reports are what keep a recipe worth reading.

If you would rather not build it, we can introduce a forward-deployed engineer who has deployed this stack before. The enquiry form starts from this recipe.