Question answering over 180,000 pages of procedures, circulars and contracts, where every answer cites the exact source page and respects per-department access control.
Operations staff spent an average of 18 minutes on each procedure lookup: open the intranet, guess at keywords, download a few hundred-page PDFs and skim. Legal fielded roughly 400 repeat questions a month, most of them already answered somewhere in the documents.
Two constraints were non-negotiable from day one: no answer ships without a verifiable citation, and nobody sees a document outside their clearance. Those two conditions shaped the whole architecture.
Re-OCR'd 40,000 scanned pages, separated tables from prose, and merged circular amendments into a single timeline.
BM25 for coarse filtering, embeddings for reranking, with access control applied inside the query rather than after results.
200 questions with reference answers written by the legal team, run automatically on every prompt or model change.
vLLM on two on-prem GPUs, a dashboard tracking citation error rate, and runbooks for the client's infra team.
The first version filtered results post-retrieval, so users saw result counts shift with their clearance and could infer that confidential documents existed. We turned department labels into predicates inside the pgvector query.
Fee schedules and limits drive most real questions, but flattened into prose the model read across the wrong columns. Tables are extracted separately, stored as markdown and returned whole.
Operators need to open the exact page to verify. We carry page coordinates through the entire chunking pipeline, which is more expensive to store, and it bought trust.
We built the eval set after the prototype and paid for it with six weeks of arguing about answer quality on gut feel. Next time, 200 questions with reference answers get written first, before an embedding model is even chosen.
Need something similar on your own data? Send a few lines of context.