Assistant on a model vendor’s API

Architecture and data flow

Organisation size

10–5000 employees

Data classes

confidential, personal

Constraints

a team that can build and run a small application, or an implementation partner who will; a paid business account with the model vendor, not a personal key; agreement that prompts and retrieved passages may leave the network under contract

Industries

Technology, Professional services, Financial services, Retail, Other

Jurisdictions

any

No hardware profile was sized for this answer.

API integrationapi-integration

development

Identity and SSOidentity-sso

security

LLM inferencellm-inference

ml

PostgreSQLpostgresql

data

Prompt engineeringprompt-engineering

ml

Retrieval-augmented generationrag

ml

Security hardeningsecurity-hardening

security

Commands are copied from each project’s own documentation, and the page they came from is linked under the step. 0 of 7 steps currently open an evidence record. The rest are linked to their source; §10 says which of those documents were fetched and which were fetched without their anchor being found — two different states, named differently there.

1. 01

   Stand up the application shell

   Assessment

   Start from Open WebUI rather than a blank repository: it brings the chat interface, role-based access control and the conversation store, and points at any OpenAI-compatible endpoint. Run it in a container on your own infrastructure so the interface, the logs and the history are inside your boundary even though the model is not.

   docker run -d -p 3000:8080 -v open-webui:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:main

   Source documentation

2. 02

   Put a LiteLLM gateway in front of the vendor keyversion-sensitive

   Assessment

   Do not let the application hold the raw vendor key. LiteLLM "brings up the gateway on port 4000 and a Postgres database that stores your models, keys, and spend logs"; point Open WebUI at the gateway, put the vendor key only in the gateway, and issue a virtual key per team so usage is attributable and one revocation does not disturb the others.

   curl -sSL https://docs.litellm.ai/docker-compose.yml | docker compose -f - up -d

   Source documentation

3. 03

   Connect the closed model, and read its default data terms firstversion-sensitive

   Assessment

   Choosing the vendor is choosing the model. Before real content flows, read the commercial data terms: Anthropic states that "by default, we will not use your inputs or outputs from our commercial products" to train, and OpenAI’s enterprise page describes commitments that give you "ownership and control over your business data". A published page is where the contract starts, not a substitute for it — record which plan and endpoint the wording applies to.

   Source documentation

4. 04

   Add retrieval with pgvector, and treat it as the sensitive part

   Assessment

   Enable pgvector in the database and index the corpus behind the assistant. pgvector’s own instruction is to "enable the extension (do this once in each database where you want to use it)". Remember what retrieval does to the request: it attaches internal passages to the prompt, so the most confidential content in a call is usually the part your own system added, not the user’s question.

   CREATE EXTENSION vector;

   run once in the database that stores embeddings

   Source documentation

5. 05

   Wire sign-in to your identity provider

   Assessment

   Configure OIDC before the pilot so nobody has a local password and leavers lose access with their directory account. Open WebUI states that "any authentication provider that supports OIDC can be configured", so the assistant joins the estate rather than becoming a second directory to maintain.

   Source documentation

6. 06

   Verify the data-handling questions against the vendor’s own pagesversion-sensitive

   Assessment

   For the vendor and endpoint you will actually call, confirm: the data processing agreement covers the API and not only the consumer product; whether a zero-data-retention or no-logging mode exists on your account; the default retention period and what triggers human review; the region the request is served from; and the supported-countries page. The no-training defaults differ in wording by vendor — Azure states prompts and completions "are NOT used to train any generative AI foundation models without your permission or instruction" — so verify the exact commitment rather than assuming parity.

   Source documentation

7. 07

   Set a spend cap, a logging decision and an evaluation set before launch

   Assessment

   Put a budget alert and a per-team spend limit on the gateway from the first week — a metered model bill grows with every user and every retrieved passage. Decide deliberately whether the gateway logs full prompts and completions, because those logs then hold the same confidential content as the source documents. Assemble a set of real questions with agreed answers and run it before launch and after every model or prompt change.

   Source documentation

Structured issue-spotting to support your own review — not legal advice. Verify against the cited primary sources and your counsel.

• The same model inside your own cloud account

  Where the same closed model is offered by your hyperscaler, calling it there keeps the request on your own network and under one existing agreement. The application does not change; only the tenancy of the model does.

  • — The call stays inside your cloud boundary and existing DPA
  • — You are limited to the models that cloud actually offers in your region
  • — A cloud engineering job — private endpoints, quotas — replaces a single API key

  aws-bedrockazure-openaigoogle-vertex-ai

• Self-hosted, if prompts may not leave the network

  Serve an open-weight model on your own GPU and no prompt reaches a vendor. The interface and retrieval layer are the same; the model is weaker and the operations are yours.

  • — No vendor sees the prompt or the retrieved passages
  • — A weaker model and a GPU to size, patch and pay for

  open-webuivllmpgvector

• Self-hostedPrivate company knowledge base (self-hosted RAG)Open WebUI as the employee interface, vLLM serving a Qwen2.5-14B-Instruct model on a single 24 GB GPU, PostgreSQL with pgvector for chats and embeddings, and OIDC single sign-on — all in Docker on one server in your office or colocation rack. Ollama replaces vLLM for teams under about 20 users; a 48 GB GPU lets you run a 32B model for better answers.
• Private cloudAssistant on Amazon BedrockA model served through Amazon Bedrock, Bedrock Knowledge Bases for retrieval over your content (with OpenSearch Serverless if you manage the vector store yourself), a web application you build, IAM Identity Center for sign-in, and a VPC with PrivateLink so the call stays on your network.

Not yet in the ledger

1. Open WebUI — Quick Start (Docker)

   Docs

   not yet fetched

2. LiteLLM — Docker quick start (gateway)

   Docs

   not yet fetched

3. Anthropic — Is my data used for model training? (commercial products)

   Vendor page

   not yet fetched

4. OpenAI — Enterprise privacy

   Vendor page

   not yet fetched

5. Azure — Data, privacy and security for models sold by Azure

   Vendor page

   not yet fetched

6. pgvector — README (installation and getting started)

   Readme

   not yet fetched

7. Open WebUI — SSO (OAuth, OIDC, Trusted Header)

   Docs

   not yet fetched

Deployed this stack, or hit something this page does not cover? Corrections, sources and implementation reports are what keep a recipe worth reading.

If you would rather not build it, we can introduce a forward-deployed engineer who has deployed this stack before. The enquiry form starts from this recipe.