NVIDIA TensorRT-LLM in Japan

Self-hosting moves the security obligation to you

Required checks

• Name the person responsible for patching each component, and the cadence.
• Confirm backups exist, are encrypted, and have been restored at least once.
• Confirm there is an incident response path that includes this system.

Technical controls

• Encrypt at rest and in transit, including between the application and the inference server.
• Centralise authentication through the existing identity provider rather than local accounts.
• Keep an audit log of who queried what, and protect it from the people it records.
• Subscribe to security advisories for each component and track upgrade lag.

Being able to reconstruct a decision months later

Required checks

• Decide what is recorded per interaction: model and version, prompt template version, retrieved document ids, output, reviewer and outcome.
• Set how long those records are kept, balanced against the retention duties that also apply to them.

Vendor questions

• Does the vendor pin model versions, and how much notice is given before a model is retired or changed?

Technical controls

• Version prompt templates in source control and log the version used.
• Log the model identifier and version returned by the provider, not the one you requested.

Confidentiality duties bind independently of data protection law

Required checks

• Review the confidentiality clauses in the contracts covering the material going into the system.
• Identify any contract requiring notice or consent before a subcontractor processes the material.
• Decide whether the deployment needs a confidentiality carve-out negotiated into new contracts.

Vendor questions

• Will the vendor accept a confidentiality undertaking beyond its standard terms?
• Which staff at the vendor can access customer content, under what controls?

Technical controls

• Segregate the most sensitive corpora into an index that the general assistant cannot reach.

Human review required — take this to your counsel

We were not told whether a person reviews the output

Required checks

• Name the role that reviews the output and what they see when they do.
• Decide what evidence is retained about each review, so the practice can be shown to exist.
• Set a threshold below which the system must not act without review.

Vendor questions

• Does the product expose the retrieved context and the confidence behind a suggestion, or only the answer?

Technical controls

• Show the reviewer the retrieved sources next to the suggestion, not the suggestion alone.
• Record the reviewer’s decision, including overrides, as part of the audit trail.

Human review required — take this to your counsel

An AI deployment creates new copies of the data

Required checks

• List every store the deployment creates and add each to the retention schedule.
• Establish how a deletion request propagates to the index, the caches and the logs.
• Establish how long backups keep material that has been deleted from the live system.

Vendor questions

• What does the vendor retain, where, and for how long after we delete our copy?

Technical controls

• Store the source document id with every embedding so deletion can cascade.
• Set time-to-live on prompt and completion logs rather than relying on manual cleanup.

Who can reach the model, the index and the weights

Required checks

• Confirm the inference endpoint is not reachable from outside the network and requires authentication.
• Confirm retrieval filters by the requesting user’s permissions, not only by relevance.
• Confirm who can read the model files and the index volume at the operating-system level.

Technical controls

• Bind the inference server to a private interface and put an authenticating proxy in front of it.
• Carry document-level access control into the index and enforce it at query time.
• Encrypt the volume holding the weights and the index, and restrict it to the service account.
• Rotate API keys and keep them out of client-side code and container images.

What ends up in a prompt, and where it goes next

Required checks

• Write down which categories of information may be entered into a prompt, and tell people.
• Establish what the system prompt contains and whether disclosing it would matter.
• Establish which shadow tools staff are already using; the policy has to name the permitted ones.

Vendor questions

• Are prompts and completions retained, for how long, and can retention be set to zero?
• Are prompts used for abuse monitoring, and if so who can read them and for how long?

Technical controls

• Redact or block high-risk patterns before the prompt leaves the application.
• Keep prompt and completion logs out of general-purpose observability tools.
• Set an explicit retention period on prompt logs and enforce it.

The APPI is mid-amendment, with no commencement date yet

Required checks

• Note the amendment in the deployment’s review schedule rather than treating today’s text as final.