Self-hosted

Private ChatGPT on your own server

Ollama serving an open-weight model on one GPU, Open WebUI in front of it for accounts, chats and admin controls, both in Docker on a single machine. Deliberately the smallest thing that works: no vector database, no connectors, no cluster.

Save to your workspace

Source: Editorial recipe — no step evidence has been fetched yet
Verified: Evidence not verified
Confidence: Low

01Objective

Give every employee a chat assistant that runs entirely on company hardware, so nothing they type is sent to an outside service.

Reference deployments using this recipe

A controlled AI learning environment for a Hong Kong secondary schoolSix hundred students who already use consumer AI tools, and a school with no way to see or shape that. A self-hosted assistant inside the school network, scoped per subject, with teacher visibility and no third party receiving a single student prompt.Reference deployment · R-03

Field notes about this recipe

02Recommended stack

5 components

Role	Component	Notes
Authenticationoptional	Existing identity provider via OIDC (Entra ID, Google Workspace, Keycloak)	Optional at ten users, necessary at fifty — it is also how leavers lose access.
Gateway	Reverse proxy with TLS on an internal hostname	No public route. A VPN or zero-trust proxy if people need it from outside the office.
Inference	Ollama	One command to pull and run a model. Simpler than vLLM and slower under load — swap to vLLM when more than about twenty people use it at once.
Inference	Qwen2.5-14B-Instruct (4-bit), or Llama-3.1-8B-Instruct on smaller hardware	Check the model card licence before commercial use; not every open-weight model is permissive.
Ui	Open WebUI	Accounts, groups, chat history and admin settings. The part employees actually see.

Architecture and data flow

External data transfer · NONE

No edge in this design crosses out of the company network.
Model weights are pulled once at setup; afterwards the machine needs no internet access to answer a question.
There is no document store in this design: only what people type is retained.

03Suitable for

Organisation size: 5–100 employees
Data classes: confidential, personal
Constraints: staff are already pasting company text into consumer chatbots and you want that stopped; one machine with a 24 GB GPU, or an Apple Silicon workstation for a small team; someone comfortable with Docker on Linux
Industries: Professional services, Legal, Accounting, Technology, Other
Jurisdictions: any

04Hardware

On-premise single 24 GB GPU server
GPU
NVIDIA RTX 4090 24 GB (or NVIDIA L4 24 GB for a rack-mounted, 72 W alternative)
VRAM
24 GB
System RAM
64 GB
Storage
2000 GB
CPU
16-core x86 server CPU (AMD EPYC 7003/9004 or Intel Xeon Scalable)
Form factor
Tower server
Indicative costUS$4,000 – US$9,000
indicative build cost for the complete machine, USD, Aug 2026 — verify with a local supplier
The default box for a 20–60 person firm. Fits a 14B model at 4-bit with roughly 8 GB of KV cache left for concurrent chat, or an 8B model at fp16. NVMe storage sized for the model cache plus a document corpus and its embeddings. Add a UPS and an offsite backup target — this machine holds the whole knowledge base.
Apple Silicon workstation (unified memory)
GPU
Apple M4 Max (up to 128 GB unified) or M3 Ultra (up to 256 GB unified)
VRAM
96 GB
System RAM
128 GB
Storage
2000 GB
CPU
Apple M4 Max / M3 Ultra
Form factor
Workstation
Indicative costUS$4,500 – US$12,000
indicative build cost for the complete machine, USD, Aug 2026 — verify with a local supplier
Quiet, standard-socket power draw, and a large memory pool — about 75% of installed unified memory is addressable by the GPU, which is where the 96 GB figure comes from on a 128 GB M4 Max. Memory bandwidth, not capacity, sets the ceiling: excellent for a pilot, a partner workstation or a single heavy user, weak for 40 people at once. Runs Ollama, llama.cpp and MLX; does not run CUDA builds of vLLM.
CPU-only server (small models and embeddings)
GPU
unknown
VRAM
unknown
System RAM
64 GB
Storage
1000 GB
CPU
16–32 core x86 server CPU with AVX-512
Form factor
Tower server
Indicative costUS$1,500 – US$4,000
indicative build cost for the complete machine, USD, Aug 2026 — verify with a local supplier
No GPU. Embedding models and document parsing run acceptably here, which is enough for a search-only pilot or a nightly batch pipeline. Chat generation with a 7–8B model at 4-bit works but reads at a few tokens per second — usable for one person testing, not for a team. The honest use of this profile is to prove the retrieval quality before buying a GPU.

Indicative costUSD · one-off plus monthly

Software licences: All components are open source and self-installed. Check each licence for use limits.; US$0
Model usage: Open weights served locally: no per-token charge. The GPU cost sits under infrastructure.; US$0
Machine with a 24 GB GPU (or Apple Silicon workstation): Hardware profile onprem-small-24gb — indicative build cost, Aug 2026, verify locally.; US$4,000 – US$9,000
Implementation (3–8 FDE-days): 3–8 FDE-days at US$760–1940 per day, converted from the HK$6,000–15,000 band at the HKMA Linked Exchange Rate band of HK$7.75–7.85 to one US dollar. One-off; excludes internal staff time.; US$2,280 – US$15,520

10–50 users, one site, no document corpus and no connectors.
Assumes an existing identity provider if SSO is wanted; standing one up adds two to four days.
Every figure is an assessment from the inputs listed, not a quotation.

05Difficulty

2 / 5

A few days, mostly configuration

06Skills

Dockerdocker: infrastructure
Identity and SSOidentity-sso: security
Linuxlinux: operations
LLM inferencellm-inference: ml
Networkingnetworking: networking
Security hardeningsecurity-hardening: security

07Deployment steps

6 steps

Commands are copied from each project’s own documentation, and the page they came from is linked under the step. 0 of 6 steps currently open an evidence record. The rest are linked to their source; §10 says which of those documents were fetched and which were fetched without their anchor being found — two different states, named differently there.

01
Run Ollama with the GPU attachedversion-sensitive
Assessment
Install the NVIDIA Container Toolkit, configure the Docker runtime, then start Ollama with `--gpus=all` and a named volume for the model cache. On Apple Silicon, install the native app instead — the Docker image has no GPU access on macOS.
```
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
```
Source documentation
02
Pull a model and check it answersversion-sensitive
Assessment
Pull the model you sized for and run it once from the command line before wiring up a UI. If it is slow here it will be slow for everyone; that is the moment to reconsider the model size or the quantisation rather than after rollout.
```
docker exec -it ollama ollama run llama3.2
```
substitute the model you chose
Source documentation
03
Run Open WebUI and connect it to Ollamaversion-sensitive
Assessment
Start Open WebUI with a persistent volume, then point it at the Ollama endpoint. When both run on the same host, `OLLAMA_BASE_URL` handles it; when Ollama is on another machine, set the URL to that host. The first account created is the administrator.
```
docker run -d -p 3000:8080 -e OLLAMA_BASE_URL=https://example.com -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main
```
replace the base URL with your Ollama host
Source documentation
04
Decide how people get accounts
Assessment
Keep new sign-ups pending until an admin approves them, or connect your identity provider over OIDC and let group membership decide access. `OPENID_PROVIDER_URL` is mandatory for the OIDC path and the redirect URI must be exactly `<your-domain>/oauth/oidc/callback`.
```
DEFAULT_USER_ROLE=pending
```
Source documentation
05
Keep it on the internal network
Assessment
Open WebUI "is built for private, trusted networks". Put it behind the VPN or a reverse proxy with TLS and an IP allowlist, set the secure-cookie variables, and do not expose it to the internet even temporarily "just for testing".
```
WEBUI_SESSION_COOKIE_SECURE=true
WEBUI_SESSION_COOKIE_SAME_SITE=strict
```
Source documentation
06
Write the one-page policy and tell people it exists
Assessment
The technical work is the easy half. Say what the tool is for, what must not be pasted into it, that chats are stored on the company server and for how long, and who to ask when it is wrong. Adoption comes from this page and a demo, not from the install.

08Compliance considerations

Structured issue-spotting to support your own review — not legal advice. Verify against the cited primary sources and your counsel.

Applies everywhere

Personal data · Retentionmedium
Chat history is a new store of personal data — what employees typed, when, and about whom. Set a retention period and a deletion routine before rollout, not after the first request.
Confidentiality · Securityhigh
The point of this deployment is that content stays inside. That only holds if the instance has no public route and no outbound model API is configured alongside the local one.
Open-source licensingmedium
Open WebUI's licence prohibits altering or removing its branding except where end users "does not exceed fifty (50) within any rolling thirty (30) day period", by written permission, or under an enterprise licence. Verify the current LICENSE text before rebranding. Model weights carry their own licences — check each model card.
Transparency · Human oversightmedium
Tell staff the assistant can be confidently wrong and that output must be checked before it reaches a client. Say plainly that conversations are stored on a company server.

09Alternatives

AnythingLLM
One container with chat, document workspaces and an embedded vector store. Fewer parts than Open WebUI plus Ollama, and it will also do basic document Q&A on day one.
- — Fastest path to something usable
- — Less granular group and permission control
- — Data lives in a single storage directory — the persistence flag is not optional
anythingllm ollama
LibreChat
Choose LibreChat when you want one interface across several model providers with per-user keys and a configuration file under version control.
- — Strong multi-provider and configuration story
- — MongoDB plus several services rather than one container
librechat
Add retrieval over your documents
When "chat" turns into "chat about our files", move to the knowledge-base recipe: vLLM for throughput, PostgreSQL with pgvector for embeddings, and permissions on collections.
- — Answers grounded in your documents
- — More components to run and to govern
open-webui vllm pgvector

Self-hostedPrivate company knowledge base (self-hosted RAG)Open WebUI as the employee interface, vLLM serving a Qwen2.5-14B-Instruct model on a single 24 GB GPU, PostgreSQL with pgvector for chats and embeddings, and OIDC single sign-on — all in Docker on one server in your office or colocation rack. Ollama replaces vLLM for teams under about 20 users; a 48 GB GPU lets you run a 32B model for better answers.

10Evidence

0 of 8 fetched

Not yet in the ledger

Open WebUI — Quick Start (Docker)
Docs
not yet fetched
Open WebUI — Connect a provider: Ollama
Docs
not yet fetched
Open WebUI — Hardening
Docs
not yet fetched
Open WebUI — SSO (OAuth, OIDC, Trusted Header)
Docs
not yet fetched
Open WebUI — LICENSE (branding condition)
Readme
not yet fetched
Ollama — Docker
Docs
not yet fetched
Ollama — GPU support
Docs
not yet fetched
AnythingLLM — Get started in Docker
Docs
not yet fetched

11Community

Deployed this stack, or hit something this page does not cover? Corrections, sources and implementation reports are what keep a recipe worth reading.

Improve this page

From the field

0 deployments · 0 questions

Nobody has reported deploying this here yet, and no question has been opened against this page. Both appear once a reviewer accepts them.

12Hire an FDE

If you would rather not build it, we can introduce a forward-deployed engineer who has deployed this stack before. The enquiry form starts from this recipe.

Hire an FDE for this stack

01Objective

Reference deployments using this recipe

Field notes about this recipe

02Recommended stack

Architecture and data flow

03Suitable for

04Hardware

On-premise single 24 GB GPU server

Apple Silicon workstation (unified memory)

CPU-only server (small models and embeddings)

Indicative cost·USD · one-off plus monthly

05Difficulty

06Skills

07Deployment steps

Run Ollama with the GPU attachedversion-sensitive

Pull a model and check it answersversion-sensitive

Run Open WebUI and connect it to Ollamaversion-sensitive

Decide how people get accounts

Keep it on the internal network

Write the one-page policy and tell people it exists