RAG will not save dirty data

Starting out, I assumed the hard part would be picking an embedding model and tuning prompts. Three months later my work log said otherwise: nearly all of it went into making 180,000 pages consistently machine-readable.

This post records what broke, how we found it, and the changes that stuck.

01Symptom: fluent answers, wrong facts

The first demo handled 40 sample questions well. Opened to the operations team, the flagged-answer rate jumped to 31%. The pattern was consistent: the system retrieved the right document but the wrong revision, or the right revision with the conditions table on the next page cut off.

• 34% of errors came from duplicate revisions with no effective-date field.
• 28% from tables shattered during PDF extraction into numbers with no column headers.
• 18% from chunking that split a clause in half.
• The rest were genuine model errors.

02Normalize first, chunk second

Instead of chunking by character count, we added an intermediate step: every document is normalized into a shared structure with title, effective date, owning department, and a list of typed content blocks. Tables stay tables; a clause stays one block.

@dataclass
class Block:
    doc_id: str
    kind: Literal["heading", "clause", "table", "para"]
    text: str
    effective_from: date
    department: str

def normalize(doc: RawDoc) -> list[Block]:
    meta = extract_header(doc)          # effective date, department
    blocks = layout_parse(doc)          # keep tables intact
    return [b.with_meta(meta) for b in blocks if b.text.strip()]

Each block keeps the parent document’s metadata, so department and date filters run at the retrieval layer.

03Measured results

Same model, same prompt, different data layer. Measured on 320 questions written by the operations team.

“A model is only as good as the worst chunk you hand it.”

04What I would do differently

1. Write the eval set in week one. Without it, every quality debate is a matter of taste.
2. Audit the data before promising a timeline. Sample 200 documents and read them.
3. Ship citations in the UI from day one. Users find data problems faster than any dashboard.