Retrieval is the product

When a retrieval deployment gives a bad answer, the model is rarely the cause. Nine times in ten the passage that contained the answer was never retrieved, or the document was never ingested, or the ingestion turned a table into unreadable text. The model faithfully answered the question it was given, which was the wrong question.

That should change where you spend your effort. Swapping in a larger model is the expensive intervention with the smallest effect; fixing chunking, adding keyword search alongside vectors, and re-ranking the candidates is cheap and usually decisive.

Ingestion converts a file into text and structure. Scanned PDFs, tables and multi-column layouts are where this fails, and it fails silently — the pipeline reports success and the text is nonsense.

Chunking splits that text into retrievable units. Too small and a passage loses the context that made it meaningful; too large and the embedding averages several topics into a vector that matches nothing well. Respect the document’s own structure — sections, clauses, headings — rather than counting characters.

Search finds candidates. Vector similarity alone misses exact terms: a case number, a product code, a person’s name. Combining it with keyword search covers both, and the combination is worth more than a better embedding model.

Re-ranking puts the best candidate first with a model that reads the query and the passage together, which is slower and much more accurate than a vector comparison. Retrieve widely, re-rank, then send a few passages to the model.

Write twenty real questions with the document that should answer each. Run them. Count how often the right document appears at all, and how often it is first. That number is your baseline, and every change you make afterwards is measured against it.

It is also the only honest thing to show a customer before go-live. “Retrieval finds the right document first for seventeen of twenty questions your team wrote” is a statement they can act on. “It works well” is not.