Skip to content
Is there an AI for this?

Field note · Customer education

Internal AI knowledge assistant: a company-document pilot

Test an internal AI assistant using company documents, source ownership, permissions, citations and outdated policies. Includes a measurable pilot plan.

Source
Editorial guidance with 1 retrieved source documents for context. Recommendations are our judgement.
Evidence
none on this page — it links to the pages that hold it

By Is There an AI for This? · Editorial guidance · Reviewed · How we use evidence

01Define what a useful answer must contain

Choose one collection of approved material and one team. Ask the owner for questions colleagues already ask: which form to use, where a procedure lives, which policy applies and who can approve an exception. A useful answer should point to the relevant source, identify the applicable version and say when the material does not answer the question.

We recommend beginning with a document assistant that reads selected sources and proposes answers. Keep updates to official records outside the first pilot. If the actual problem is that nobody knows which document is current, appoint the source owner and resolve that first. Putting competing policy versions into a chat interface does not settle the underlying disagreement.

02Prepare a small source register

  • For every included document, record its title, owner, approval status, effective date and access group. Exclude drafts unless the answer clearly identifies them as drafts.
  • Keep the original file available from the answer. Ask reviewers to open the cited passage rather than accepting a document title as proof.
  • Define how updates and deletions reach the assistant. Test a withdrawn policy, a replacement version and a file whose access has been removed.
  • Identify restricted material by project, department or client. Test an ordinary user, a privileged reviewer and a recently transferred employee separately.

For the pilot, use a question bank with answerable questions, questions the sources cannot answer and questions that require access the tester does not have. Write the expected behaviour for each group. “I cannot find an approved answer” can be the correct result. Do not score every refusal as a failure.

03Follow one question from source to reviewer

  1. 01Ask in ordinary workplace languageHave users type the question they would send to a colleague, including abbreviations and incomplete context. Note which details the assistant should ask for before answering.
  2. 02Check the answer against the passageThe reviewer records whether the cited text supports every material instruction, whether its version applies and whether a missing condition changes the meaning.
  3. 03Resolve the gap in the correct placeA bad source should go to the source owner. A wrong retrieval should go to the implementation team. A confusing question may need a clarification prompt. Keep these categories separate.

04Measure finding and checking time together

Add a second reviewer for disputed answers and save the reason for disagreement. If experienced staff cannot agree on the correct response, mark the question unresolved rather than turning one person’s preference into a benchmark. These disagreements can reveal a documentation job that is more valuable than another model change.

05Treat source access and versioning as buying requirements

  • RECOMMENDATION

    For a knowledge assistant over confidential company files, we recommend testing permissions and withdrawn documents before measuring how fluent the answers sound.

    BasisThis depends on mixed access groups and changing internal procedures. Microsoft’s primary documentation is relevant reading about permission-aware retrieval; your selected product and configuration still need their own tests.

Illustrative acceptance criteria are at least 90% of answerable test questions meeting the agreed rubric, all tested restricted questions denied, all tested withdrawn versions excluded, and a 20% improvement in median find-and-check time. Treat access failures as a stop condition rather than averaging them into a quality score. A passing sample cannot prove that all future permissions are correct.

Stop if the assistant exposes restricted passages, invents a policy condition or continues citing a removed document after the agreed refresh window. Resume only after the responsible owner fixes the cause and the failed examples pass. At handover, give the business owner a simple way to report an answer, update a source and see which unresolved questions recur.

06Related

Use cases

Stacks

Deployment reports

More field notes

Ask this as a question

  1. 01Our 60-person company keeps asking the same questions about internal documents. Can an AI assistant help without showing confidential files to the wrong staff?

07Evidence

1 record
  1. 01
    Tier 2official vendor legal documentationVendor legal and technical documentation

    Data, Privacy, and Security for Microsoft Copilot

    https://learn.microsoft.com/en-us/microsoft-365/copilot/microsoft-365-copilot-privacy

    FreshRetrieved 19 Sep 2026sha256:9e77981ebc11

    1 record

08Primary-source reading

Consult these primary documents when checking a proposed product. Reading links remain available when evidence has not been verified; they do not establish that your planned use is approved or that a recommendation has been proven.