Field note · Technical
How to deploy a private company ChatGPT
The shortest honest path from "we want our own ChatGPT" to a system employees use: what the four moving parts are, which decisions are irreversible, and the two places every one of these deployments gets stuck.
- Source
- Written by us. Method and judgement — no statement on this page rests on a fetched document.
- Evidence
- none on this page — it links to the pages that hold it
01Four parts, and only four
A private company ChatGPT is a chat interface, an inference server, a model, and an identity provider. Everything else — retrieval over your documents, connectors, audit logging — is an addition to that base, and each addition should be justified separately.
Keeping the base this small is not minimalism for its own sake. Each part is replaceable, and the value of the deployment is that you can replace the model in year two without replacing the interface your staff have learned.
| Part | Typical choice | What the choice decides |
|---|---|---|
| Interface | Open WebUI, LibreChat, AnythingLLM | What staff see, and how permissions are expressed |
| Inference server | Ollama for a small team, vLLM for concurrency | How many people can ask at once |
| Model | An open-weights instruct model, sized to the GPU | Answer quality, languages, and VRAM |
| Identity | Your existing OIDC provider, or Keycloak | Who is in, who is out, and what leaving looks like |
02The order to build it in
- 01Get one model answering on the target hardwareBefore any interface. If the model cannot be served at acceptable latency on the GPU you actually have, nothing above it matters, and you have learned that in an afternoon rather than in week three.
- 02Put the interface in front of it, on the internal network onlyNo public ingress until identity is wired. A chat interface reachable from the internet with local accounts is the single most common avoidable mistake in these deployments.
- 03Wire identity before the first real userRetro-fitting single sign-on after fifty staff have created local accounts means migrating their history or losing it. Do it while the user table is empty.
- 04Add retrieval only once people are using itRetrieval is where deployments stall, because it exposes every inconsistency in how your documents are stored and permissioned. Let staff use the plain assistant first; the questions they ask will tell you which corpus is worth indexing.
- 05Turn on logging and an evaluation setTwenty real questions with expected answers, run before and after every model or prompt change. Without it, "the new model is better" is an opinion.
- ASSESSMENT
Open WebUI publishes a single-container quick start in its own documentation, which is enough to have the interface running against a local model before any identity work begins.
Label withdrawnWritten as a fact resting on a fetched document, and shown as an assessment here because no stored evidence record for it could be resolved when this page was rendered.
- ASSESSMENT
Open WebUI documents single sign-on against any provider that supports OIDC, so an existing corporate identity provider can be used rather than a second user directory.
Label withdrawnWritten as a fact resting on a fetched document, and shown as an assessment here because no stored evidence record for it could be resolved when this page was rendered.
- ASSESSMENT
Ollama documents the GPU compute capability and driver versions it supports, which is the check to run against your existing hardware before buying any.
Label withdrawnWritten as a fact resting on a fetched document, and shown as an assessment here because no stored evidence record for it could be resolved when this page was rendered.
03The two places these deployments get stuck
The first is concurrency. A single-user demonstration on a workstation says nothing about forty people asking at 09:15 on Monday. The inference server, not the model, decides this: a server built for batching serves many more concurrent sessions from the same GPU than one built for a single local session.
The second is permissions on retrieved documents. Once the assistant can quote from a shared drive, it inherits every mistake in that drive’s access control, and it makes those mistakes visible in a way that browsing never did. Treat the first retrieval corpus as a permissions project with a chat interface attached.
- ASSESSMENT
Open WebUI publishes hardening guidance for a deployment that will carry company documents; read it before the first non-pilot user, not after.
Label withdrawnWritten as a fact resting on a fetched document, and shown as an assessment here because no stored evidence record for it could be resolved when this page was rendered.
- ASSESSMENT
We assess concurrency, not answer quality, as the usual reason a working pilot fails at rollout, and we size the inference server from expected simultaneous users rather than from headcount.
BasisRests on the hardware model used across this site: VRAM is weights plus a KV cache that grows with concurrent sessions, so the same model and the same GPU support very different user counts depending on the server in front of them.
04One licence clause worth reading first
- ASSESSMENT
Open WebUI’s licence adds a branding condition that applies once the number of users exceeds fifty within any rolling thirty-day period; below that threshold the condition does not bite.
Label withdrawnWritten as a fact resting on a fetched document, and shown as an assessment here because no stored evidence record for it could be resolved when this page was rendered.
- RECOMMENDATION
We recommend reading the licence of the interface before the licence of the model, because the interface is the component an organisation is most likely to rebrand and least likely to check.
BasisDepends on the deployment having more than fifty users and on any intention to remove or replace project branding. Rests on the licence clause cited above.
05Related
Use cases
Stacks
Tools
Deployment reports
More field notes
Ask this as a question