Use case
Local LLM
Running an open-weight model on hardware you own — a workstation, a GPU server, a laptop. The question is usually which model fits the memory you have and how many people can use it at once.
- Source
- Editorial ontology entry — no fetched document behind this page
- Evidence
- none on this page — it links to the pages that hold it
- Category
- Infrastructure
- Typical risk
- medium
- Entry
- editorial · reviewed 25 Aug 2026
01What this is
A local LLM deployment downloads open-weight models and serves them from your own machine through an inference server. Practical model choice is set by memory: weights in the chosen quantisation, plus the KV cache for the context length and the number of concurrent users.
A good deployment sizes hardware from the workload rather than the model name, picks a quantisation deliberately, pins model versions, measures tokens per second at the intended concurrency, and keeps a fallback path for peaks. Model licences are checked before commercial use — open weights are not automatically open licences.
Pitfalls: buying a GPU before measuring; a 70B model chosen for a task a 14B handles; and an inference server run as a single point of failure with no monitoring. Weights are downloaded once from a registry over the network — that is normally the only external traffic, and it is worth stating explicitly.
- Typical data
- internal documents, source code, client documents
- Solution classes it admits
- Self-hosted
02Deployment options
- ASSESSMENT
On a neutral reading of this use case, Self-hosted is a strong alternative, Private cloud is a strong alternative and Enterprise SaaS is conditional.
- ASSESSMENT
Read as a generic reading of this page, not a recommendation: no organisation, size, jurisdiction, budget or technical capability has been supplied, so wherever an option depends on one of those, it says "unknown". Ask your own question to get a verdict that accounts for them.
Private cloud
STRONG ALTERNATIVEManaged model API or private model deployment in your cloud account
- ASSESSMENT
Two patterns fit inside one account in a region you name: call a managed foundation-model API such as Amazon Bedrock, Microsoft Foundry / Azure OpenAI, or Vertex AI; or deploy an open-weight model on GPU compute you control. Your application, retrieval layer, storage, identity and logs remain in your cloud boundary in both patterns.
- ASSESSMENT
The managed-API pattern can use closed-source frontier models without buying or operating GPUs. It is usually the fastest way to build a custom workflow, but prompts and retrieved context are processed by the managed service, so model availability, retention, abuse monitoring and regional routing must be checked for the exact feature and endpoint.
- ASSESSMENT
The private-model pattern gives more control over weights, serving and network paths, and can use managed endpoints or your own containers. It also makes your team responsible for capacity, patches, model upgrades, evaluation and failover.
- ASSESSMENT
The cloud provider becomes a data processor in either pattern: you need a DPA, a documented region, and an answer on cross-region routing and where support staff can access the environment from.
- RECOMMENDATION
Start with the managed-API pattern when the workflow is custom but model operations are not the source of competitive advantage; move to private model serving only if evaluation, volume, portability or the data boundary justifies the extra operations. Your stated technical capability is "unknown".
Self-hosted
STRONG ALTERNATIVEOpen-weight models on infrastructure you operate
- ASSESSMENT
Documents, queries and embeddings stay on machines you own, using an open-weight model whose licence you review. For a brief that involves confidential documents, that removes a model-API vendor from the data path rather than governing that transfer by contract.
- ASSESSMENT
It costs you the operational work instead: a GPU server, Docker, Linux, backups and a patching routine. Your stated technical capability is "unknown", which is the attribute this option most depends on.
- ASSESSMENT
No processor agreement, subprocessor list or cross-border transfer assessment is needed for the model itself, because no third party processes the content.
- RECOMMENDATION
Recommended where local processing is preferred (you did not say so) and the content is sensitive (confidential documents).
Enterprise SaaS
CONSIDER IFFinished closed-source cloud product with enterprise controls
- ASSESSMENT
This is a complete vendor application, not a model API: examples include an enterprise assistant, coding copilot or document product with the workflow, interface, connectors and administration already built. It can use closed-source cloud models while requiring no model hosting or application engineering from your team.
- RECOMMENDATION
Choose it when the product already performs the actual workflow and its controls meet your requirements. Do not choose it only because its underlying model is strong: a finished SaaS product is less flexible than building against a managed API when your process, integrations or review steps are organisation-specific.
- ASSESSMENT
Vendor commitments are treated as unverified until we have fetched the page that makes them. Until then this option carries questions to ask, not assurances: a signed data processing agreement covering the data you will actually put in; a documented data residency commitment naming the region, in the contract rather than a blog post; a written no-training commitment for your content, including uploads and connected sources; stated retention periods and a deletion path you can exercise; an administrative audit log you can export, and SSO with group-based access control.
- RECOMMENDATION
No jurisdiction was named, so this is the check rather than the conclusion: compare the vendor's stated processing locations and subprocessor list against the cross-border transfer rules wherever you operate before uploading anything.
03Deployment stacks
- Self-hostedLocal LLM inference server (Ollama / vLLM)vLLM for throughput or Ollama for simplicity, one GPU, and LiteLLM in front as the gateway that issues per-team keys and records spend. The endpoint other recipes point at.
- Self-hostedPrivate ChatGPT on your own serverOllama serving an open-weight model on one GPU, Open WebUI in front of it for accounts, chats and admin controls, both in Docker on a single machine. Deliberately the smallest thing that works: no vector database, no connectors, no cluster.
04Tools by hosting option
Self-hosted11
- HybridAnythingLLMDesktop and server application that turns a document set into a chat workspace, with per-workspace embeddings, multiple model back ends and a built-in vector store.
- HybridContinueOpen-source IDE extension for VS Code and JetBrains that connects completion and chat to any model back end, including a local server, under a checked-in configuration file.
- PlatformLiteLLMGateway that presents one OpenAI-compatible API in front of many providers and local servers, with per-team keys, budgets, rate limits, fallbacks and request logging.
- Libraryllama.cppC++ inference engine for quantised models on CPU, Apple Silicon and GPUs, with a bundled HTTP server. The engine underneath many desktop runtimes and the GGUF quantisation format.
- HybridLM StudioDesktop application for downloading and running open-weight models locally, with a chat interface and a local OpenAI-compatible server. Windows, macOS and Linux.
- Model serverLocalAIDrop-in OpenAI-compatible API server that runs text, embedding, image and audio models locally across several back ends, including CPU-only deployments.
- Model serverOllamaLocal model runtime with a one-command install, a model library and an OpenAI-compatible API. The usual starting point for running open-weight models on a workstation or small server.
- Self-hostedOpen WebUISelf-hosted chat interface for local and hosted models, with user accounts, groups, document upload and built-in retrieval. Runs in Docker against Ollama, vLLM or any OpenAI-compatible endpoint.
- Model serverSGLangServing framework for large models with prefix caching and structured-output support, aimed at high-throughput deployments and multi-GPU serving.
- Self-hostedTabbySelf-hosted coding assistant with its own inference server, repository-aware context and IDE extensions. Runs entirely inside the network, including on a single GPU.
- Model servervLLMHigh-throughput GPU inference server using paged attention and continuous batching, with an OpenAI-compatible API. Built for many concurrent users rather than single-session use.
Private cloud9
- HybridAnythingLLMDesktop and server application that turns a document set into a chat workspace, with per-workspace embeddings, multiple model back ends and a built-in vector store.
- HybridContinueOpen-source IDE extension for VS Code and JetBrains that connects completion and chat to any model back end, including a local server, under a checked-in configuration file.
- PlatformLiteLLMGateway that presents one OpenAI-compatible API in front of many providers and local servers, with per-team keys, budgets, rate limits, fallbacks and request logging.
- Model serverLocalAIDrop-in OpenAI-compatible API server that runs text, embedding, image and audio models locally across several back ends, including CPU-only deployments.
- Model serverOllamaLocal model runtime with a one-command install, a model library and an OpenAI-compatible API. The usual starting point for running open-weight models on a workstation or small server.
- Self-hostedOpen WebUISelf-hosted chat interface for local and hosted models, with user accounts, groups, document upload and built-in retrieval. Runs in Docker against Ollama, vLLM or any OpenAI-compatible endpoint.
- Model serverSGLangServing framework for large models with prefix caching and structured-output support, aimed at high-throughput deployments and multi-GPU serving.
- Self-hostedTabbySelf-hosted coding assistant with its own inference server, repository-aware context and IDE extensions. Runs entirely inside the network, including on a single GPU.
- Model servervLLMHigh-throughput GPU inference server using paged attention and continuous batching, with an OpenAI-compatible API. Built for many concurrent users rather than single-session use.
Vendor cloud2
- HybridContinueOpen-source IDE extension for VS Code and JetBrains that connects completion and chat to any model back end, including a local server, under a checked-in configuration file.
- PlatformLiteLLMGateway that presents one OpenAI-compatible API in front of many providers and local servers, with per-team keys, budgets, rate limits, fallbacks and request logging.
On-premise (enterprise plan)4
- Self-hostedOpen WebUISelf-hosted chat interface for local and hosted models, with user accounts, groups, document upload and built-in retrieval. Runs in Docker against Ollama, vLLM or any OpenAI-compatible endpoint.
- Model serverSGLangServing framework for large models with prefix caching and structured-output support, aimed at high-throughput deployments and multi-GPU serving.
- Self-hostedTabbySelf-hosted coding assistant with its own inference server, repository-aware context and IDE extensions. Runs entirely inside the network, including on a single GPU.
- Model servervLLMHigh-throughput GPU inference server using paged attention and continuous batching, with an OpenAI-compatible API. Built for many concurrent users rather than single-session use.
05Compliance hot spots
This use case usually raises open-source licensing, model access, security, intellectual property, confidentiality, logging, personal data, prompt leakage, retention, transparency.
- Open-source licensing
- No published jurisdiction page names this topic yet
- Model access
- No published jurisdiction page names this topic yet
- Security
- China (mainland)
- Intellectual property
- No published jurisdiction page names this topic yet
- Logging
- European Union
- Retention
- Hong Kong
- Transparency
- United KingdomJapanSouth Korea
06Example questions
Each of these opens the question box with the text already in it. The answer is researched for your organisation, not for this page.
07Related use cases
- InfrastructurePrivate LLMUsing a language model without your data leaving a boundary you control — a country, a cloud region, or your own network. The constraint is where processing happens and who can see the prompts, not which model runs.
- InfrastructurePrivate company ChatGPTA chat assistant for staff that behaves like a consumer chatbot but runs under company control: your accounts, your logging, your retention, your choice of model, and optionally your own documents attached.
Improve this page
Sign in to contribute
From the field
0 deployments · 0 questions
Nobody has reported deploying this here yet, and no question has been opened against this page. Both appear once a reviewer accepts them.