Self-hosted
Local LLM on a workstation (LM Studio / Ollama)
LM Studio or Ollama on an Apple Silicon machine (unified memory holds large models) or an NVIDIA workstation (a 96 GB card serves them faster), running an open-weight model sized to the hardware, exposed as a local server when other tools need it. Single-machine: the strongest privacy story and the weakest governance story.
- Source
- Editorial recipe — no step evidence has been fetched yet
- Verified
- Evidence not verified
- Confidence
- Low
01Objective
Run a capable open-weight model on one workstation — as a chat assistant and as a local OpenAI-compatible endpoint your own scripts can call — with nothing leaving the machine.
- use caseLocal LLMRunning an open-weight model on hardware you own — a workstation, a GPU server, a laptop. The question is usually which model fits the memory you have and how many people can use it at once.
- use casePrivate company ChatGPTA chat assistant for staff that behaves like a consumer chatbot but runs under company control: your accounts, your logging, your retention, your choice of model, and optionally your own documents attached.
- use casePrivate LLMUsing a language model without your data leaving a boundary you control — a country, a cloud region, or your own network. The constraint is where processing happens and who can see the prompts, not which model runs.
02Recommended stack
| Role | Component |
|---|---|
| Inference | LM Studio |
| Inferenceoptional | Ollama |
| Inference | Open-weight model sized to the machine (Qwen3-8B, Gemma-3-12B or Phi-4) |
| Ui | Local chat interface |
Architecture and data flow
Components
- Person at the workstation (and local tools) — people
- LM Studio (chat + OpenAI-compatible server) — application
- llama.cpp / MLX runtime — inference server
- Open-weight model (Qwen / Gemma / Phi) — model
- Apple Silicon or NVIDIA workstation — hardware
Connections
- Person at the workstation (and local tools) to LM Studio (chat + OpenAI-compatible server) — HTTPS (confidential data)
- LM Studio (chat + OpenAI-compatible server) to llama.cpp / MLX runtime — prompt (confidential data)
- llama.cpp / MLX runtime to Open-weight model (Qwen / Gemma / Phi) — loaded weights
- Open-weight model (Qwen / Gemma / Phi) to Apple Silicon or NVIDIA workstation — GPU memory
- llama.cpp / MLX runtime to Apple Silicon or NVIDIA workstation — GPU memory
External data transfer · NONE
- No edge in this design crosses out of the company network.
- Everything runs on one machine you own; model weights are downloaded once from Hugging Face at setup and nothing else leaves it.
- The local endpoint is on the machine by default; sharing it on the network is a deliberate step, and one to secure.
03Suitable for
- Organisation size
- 1–10 employees
- Data classes
- confidential, personal, public
- Constraints
- a single workstation with enough unified memory or VRAM to hold the model; a person comfortable installing software and managing model files; acceptance that each machine is an island — no shared index, history or central deletion
- Industries
- Technology, Professional services, Legal, Financial services, Other
- Jurisdictions
- any
04Hardware
Apple Silicon workstation (unified memory)
- GPU
- Apple M4 Max (up to 128 GB unified) or M3 Ultra (up to 256 GB unified)
- VRAM
- 96 GB
- System RAM
- 128 GB
- Storage
- 2000 GB
- CPU
- Apple M4 Max / M3 Ultra
- Form factor
- Workstation
Indicative costUS$4,500 – US$12,000
indicative build cost for the complete machine, USD, Aug 2026 — verify with a local supplier
Quiet, standard-socket power draw, and a large memory pool — about 75% of installed unified memory is addressable by the GPU, which is where the 96 GB figure comes from on a 128 GB M4 Max. Memory bandwidth, not capacity, sets the ceiling: excellent for a pilot, a partner workstation or a single heavy user, weak for 40 people at once. Runs Ollama, llama.cpp and MLX; does not run CUDA builds of vLLM.
Indicative costUSD · one-off plus monthly
- Software licences
- All components are open source and self-installed. Check each licence for use limits.
- US$0
- Model usage
- Open weights served locally: no per-token charge. The GPU cost sits under infrastructure.
- US$0
- Workstation
- Not estimated: a one-off purchase priced per configuration, and no price was fetched. An Apple Silicon Mac Studio (unified memory from 48 GB) or an NVIDIA RTX PRO 6000 workstation (96 GB) — see the vendor pages for current pricing in your region. Many teams already own a suitable machine, in which case this is nil.
- Not estimated
- Implementation (1–4 FDE-days)
- 1–4 FDE-days at US$760–1940 per day, converted from the HK$6,000–15,000 band at the HKMA Linked Exchange Rate band of HK$7.75–7.85 to one US dollar. One-off; excludes internal staff time.
- US$760 – US$7,760
- No software or model cost: the runner and the open-weight model are free; the only outlay is the workstation, which many people already have.
- Implementation is an afternoon: install, download a model, confirm it runs, and secure the endpoint. It is deliberately small because that is the point of this option.
- Sized for one person or a few technical users, not a team.
05Difficulty
1 / 5
An afternoon for one competent engineer
06Skills
- Linuxlinux
- operations
- LLM inferencellm-inference
- ml
- Security hardeningsecurity-hardening
- security
07Deployment steps
Commands are copied from each project’s own documentation, and the page they came from is linked under the step. 0 of 7 steps currently open an evidence record. The rest are linked to their source; §10 says which of those documents were fetched and which were fetched without their anchor being found — two different states, named differently there.
- 01
Size the model to the memory you have
AssessmentThe memory holds the model, so the machine sets the ceiling. A Mac Studio offers "48GB, 64GB, or 128GB unified memory (M5 Max with 18-core CPU and 40-core GPU)", shared between the model and everything else; an NVIDIA RTX PRO 6000 workstation is "equipped with 96 GB of ultra-fast GDDR7 memory" dedicated to the GPU. Work out the model plus its context at 4-bit before downloading tens of gigabytes.
- 02
Choose the machine for how you will use it
AssessmentApple Silicon trades speed for capacity and quiet: the M5 Ultra reaches "512GB unified memory (M5 Ultra with 36-core CPU and 80-core GPU)", enough to hold very large models, but bandwidth bounds throughput. An NVIDIA workstation card — the RTX PRO 6000 is "built on the groundbreaking NVIDIA Blackwell architecture" — is faster for one heavy user. For a single person, either is fine; choose on noise, speed and what else the machine does.
- 03
Install LM Studio and download a modelversion-sensitive
AssessmentLM Studio runs on Mac, Windows and Linux, and "on Apple Silicon Macs, LM Studio also supports running LLMs using Apple’s" MLX for speed. Install it, search and download a quantised model from within the app, and chat to it to confirm the machine handles the size you chose before wiring anything else to it.
- 04
Turn on the local endpoint for your own tools
AssessmentBeyond the chat window, LM Studio can "serve local models on OpenAI-like endpoints, locally and on the network" — which is what makes this an inference platform as well as an assistant. Point a script, an editor plugin or a notebook at the local endpoint and it uses the same OpenAI client, with the base URL changed to your machine.
- 05
Or use Ollama, especially on NVIDIAversion-sensitive
AssessmentOllama is the command-line alternative and pairs well with an NVIDIA workstation: it "supports Nvidia GPUs with compute capability 5.0+ and driver version 550 and newer". One command pulls and runs a model, and it exposes its own local OpenAI-compatible endpoint.
ollama run qwen3:8bpulls the model on first run, then chats
- 06
Keep the endpoint on the machine unless you mean otherwise
AssessmentServing "locally and on the network" is a choice with consequences: an endpoint bound beyond the loopback interface is a model anyone on the network can use, usually with no authentication. Keep it on localhost unless you deliberately want to share it, and if you do, put it behind something that asks for a key.
- 07
Write down the one rule the setup cannot enforce
AssessmentThere is no administrative log here and no central deletion, so governance is a written rule rather than a control: what may be pasted in, and that each machine is a separate island with its own history. That is the honest trade — the same locality that makes this the strongest privacy story makes it the weakest governance story.
08Compliance considerations
Structured issue-spotting to support your own review — not legal advice. Verify against the cited primary sources and your counsel.
Applies everywhere
Security · Model accesshigh
The risk is an exposed local server. LM Studio can serve "locally and on the network"; keep it on the loopback interface unless you intend to share it, and put a key in front of it if you do. An unauthenticated model on the office network is a shared GPU for anyone who finds it.
Logging · Prompt leakagemedium
There is no administrative audit log and no central record of what was asked. For anything beyond one person, that absence is the reason this does not scale — write the acceptable-use rule down, because nothing in the setup will enforce it.
Open-source licensinglow
Two licences apply: the runner’s and the model’s. Ollama is MIT; check LM Studio’s own terms for business use, and read each model card for field-of-use or user-count conditions before relying on it for work.
Confidentiality · Data residencylow
Nothing leaves the machine, which is the strongest residency and confidentiality story available — no processor, no transfer, no region question. The cost is that the same locality gives you no central control, so it is right for a person and wrong for a company.
09Alternatives
A shared inference server for a team
When more than a few people need the same model, one GPU server with vLLM and a gateway serves them all with per-team keys and a spend log — the shared version of this setup.
- — One model, many users, with keys and usage attribution
- — A server to run and secure, rather than an app on a laptop
A private knowledge base with a UI and retrieval
When the workstation model needs to answer over shared company documents, the on-premise knowledge base adds a web UI, retrieval and identity — still entirely on your own machines.
- — Shared documents, accounts and a real interface
- — A server and a stack to operate, not a single app
- Self-hostedLocal LLM inference server (Ollama / vLLM)vLLM for throughput or Ollama for simplicity, one GPU, and LiteLLM in front as the gateway that issues per-team keys and records spend. The endpoint other recipes point at.
- Self-hostedPrivate company knowledge base (self-hosted RAG)Open WebUI as the employee interface, vLLM serving a Qwen2.5-14B-Instruct model on a single 24 GB GPU, PostgreSQL with pgvector for chats and embeddings, and OIDC single sign-on — all in Docker on one server in your office or colocation rack. Ollama replaces vLLM for teams under about 20 users; a 48 GB GPU lets you run a 32B model for better answers.
10Evidence
Not yet in the ledger
not yet fetched
not yet fetched
not yet fetched
- Apple — Mac Studio unified memory (M5 Max)
Vendor page
not yet fetched
- Apple — Mac Studio unified memory (M5 Ultra)
Vendor page
not yet fetched
- NVIDIA — RTX PRO 6000 Blackwell 96 GB
Vendor page
not yet fetched
11Community
Deployed this stack, or hit something this page does not cover? Corrections, sources and implementation reports are what keep a recipe worth reading.
Improve this page
Sign in to contribute
From the field
0 deployments · 0 questions
Nobody has reported deploying this here yet, and no question has been opened against this page. Both appear once a reviewer accepts them.
12Hire an FDE
If you would rather not build it, we can introduce a forward-deployed engineer who has deployed this stack before. The enquiry form starts from this recipe.