Contamination: when the model has seen the test before it is graded

Contamination: when the model has seen the test before it is graded A data-flow diagram generated by Archify. 01 / Benchmark 02 / Leaks to the web 03 / Into the corpus 04 / Checking 05 / Measured result Benchmark Q&A · MMLU · ARC · HumanEval · 01 / Benchmark · what we score with Benchmark Q&A MMLU · ARC · HumanEval what we score with Original eval set · kept for comparison · 01 / Benchmark Original eval set kept for comparison Public reposts · blogs · GitHub · forums · 02 / Leaks to the web · leak path Public reposts blogs · GitHub · forums leak path Training corpus · crawl picks up solutions · 03 / Into the corpus Training corpus crawl picks up solutions n-gram lookup table · hash set · Bloom filter · 03 / Into the corpus n-gram lookup table hash set · Bloom filter Overlap check · n-gram · paragraph · near-dup · 04 / Checking · decontamination Overlap check n-gram · paragraph · near-dup decontamination Score reflects ability · still comparable · 05 / Measured result Score reflects ability still comparable Inflated score · recall counted as skill · 05 / Measured result Inflated score recall counted as skill reposted crawled in before training keep if no overlap build lookup structure compare each paragraph what slips through Legend primary data policy / PII data store data flow