Agent workflow on a model vendor’s API

Architecture and data flow

Organisation size

20–10000 employees

Data classes

confidential, personal

Constraints

a genuine multi-step task — planning, tool use, looping — not a single question; a team that can build and operate an application, or an implementation partner; agreement that prompts and tool inputs may leave the network to the model vendor

Industries

Technology, Professional services, Financial services, Recruitment hr, Other

Jurisdictions

any

No hardware profile was sized for this answer.

API integrationapi-integration

development

LLM evaluationllm-evaluation

ml

Prompt engineeringprompt-engineering

ml

Security hardeningsecurity-hardening

security

TypeScripttypescript

development

Workflow automationworkflow-automation

operations

Commands are copied from each project’s own documentation, and the page they came from is linked under the step. 0 of 7 steps currently open an evidence record. The rest are linked to their source; §10 says which of those documents were fetched and which were fetched without their anchor being found — two different states, named differently there.

1. 01

   Design the graph before the promptsversion-sensitive

   Assessment

   LangGraph is "a low-level orchestration framework and runtime for building, managing, and deploying long-running, stateful agents". Its point is that you can "mix deterministic, hand-coded steps with LLM-driven agentic steps in the same graph" — so make the steps that must be reliable deterministic, and reserve the model for the parts that genuinely need judgement. Draw the graph first; the prompts come after.

   pip install -U langgraph

   Source documentation

2. 02

   Connect the model, and read its data terms firstversion-sensitive

   Assessment

   The model does the planning and tool selection, so the vendor sees the prompt and the tool inputs. Before real data flows, confirm the commercial terms: Anthropic states that "by default, we will not use your inputs or outputs from our commercial products" to train, and OpenAI’s enterprise page describes commitments giving you "ownership and control over your business data". Record which plan and endpoint the wording applies to.

   Source documentation

3. 03

   Define narrow tools and scope their permissions

   Assessment

   Each tool is a capability you hand the agent. Give each one the least access it needs, run it under the identity of the user who triggered the agent rather than a superuser service account, and validate its inputs in code — a model that can call a tool with any argument is a model that can call it with the wrong one. The orchestration graph is where these calls are wired.

   Source documentation

4. 04

   Put a human gate on the consequential actions

   Assessment

   Not every step should run unattended. LangGraph lets you "incorporate human oversight by inspecting and modifying agent state at any point" — use it: pause before an action that spends money, emails a customer or changes a record, and require an approval. The cost of the gate is latency; the cost of not having it is an autonomous mistake.

   Source documentation

5. 05

   Trace every run with Langfuse

   Assessment

   Langfuse is "an open-source AI engineering platform" whose "traces include all LLM and non-LLM calls, including retrieval, embedding, API calls, and more", and where "LLM agents can be visualized as a graph to illustrate the flow of complex agentic workflows". Wire it in from the first prototype: without the trace, an agent that did the wrong thing is a black box, and the trace is also the audit record.

   Source documentation

6. 06

   Evaluate on real tasks, and cap the loop

   Assessment

   Build a set of real tasks with agreed outcomes and run them through Langfuse’s evaluation before launch and after every prompt or model change — an agent’s failure modes are compounding, not one-shot. Cap the number of steps and the spend per run so a reasoning loop that goes wrong stops rather than bills, and alert on runs that hit the cap.

   Source documentation

7. 07

   Verify the vendor’s data-handling before productionversion-sensitive

   Assessment

   For the model endpoint you will call, confirm the DPA covers the API, whether a zero-data-retention mode is available, the retention period and what triggers human review, and the region the request is served from. The agent sends more to the vendor than a single-shot assistant does — every tool result it reasons over — so the data question is larger here, not smaller.

   Source documentation

Structured issue-spotting to support your own review — not legal advice. Verify against the cited primary sources and your counsel.

• A managed agent service in your own cloud

  Where you are already on a hyperscaler, its managed agent runtime keeps the orchestration, the model and the traces inside your own account rather than split between a framework you host and a vendor API.

  • — Orchestration, model and traces in one cloud boundary and agreement
  • — Less control over the graph than a framework you own, and tied to that cloud

  bedrock-agentsfoundry-agent-servicevertex-ai-agent-engine

• A single retrieval-grounded answer, if that is all it needs

  Many tasks framed as "agents" are one retrieval and one answer. If yours is, the assistant recipe is simpler, cheaper and far easier to make reliable.

  • — Much simpler and more predictable
  • — Cannot plan, use tools or loop — one question, one answer

  open-webuilitellmpgvector

• Model APIAssistant on a model vendor’s APIOpen WebUI as the interface, a LiteLLM gateway holding the vendor key and the spend log, a closed model (GPT-5, Claude Sonnet or Gemini) reached over its API, and pgvector for retrieval over your own documents. The application is identical to a self-hosted one; only the model moves to a vendor endpoint.
• Private cloudAgent workflow on a managed cloud runtimeA managed agent runtime that handles orchestration, memory, tool invocation and observability; a model from the same cloud; your APIs exposed as tools; private networking; and human approval gates on consequential actions. Same shape as the framework-hosted agent, one tenancy instead of two.

Not yet in the ledger

1. LangGraph — orchestration framework for stateful agents

   Docs

   not yet fetched

2. LangGraph — mixing deterministic and agentic steps

   Docs

   not yet fetched

3. LangGraph — human-in-the-loop oversight

   Docs

   not yet fetched

4. Langfuse — open-source AI engineering platform

   Docs

   not yet fetched

5. Langfuse — traces of LLM and non-LLM calls

   Docs

   not yet fetched

6. Langfuse — agents visualized as a graph

   Docs

   not yet fetched

7. Anthropic — Is my data used for model training? (commercial products)

   Vendor page

   not yet fetched

8. OpenAI — Enterprise privacy

   Vendor page

   not yet fetched

Deployed this stack, or hit something this page does not cover? Corrections, sources and implementation reports are what keep a recipe worth reading.

If you would rather not build it, we can introduce a forward-deployed engineer who has deployed this stack before. The enquiry form starts from this recipe.