What a benchmark number is made of

What a benchmark number is made of A workflow diagram generated by Archify. 01 / Real capability 02 / Already seen: contamination 03 / Sensitivity to how you ask 04 / Statistical noise Capability · what we want to measure · Real capability Capability what we want to measure Signal · holds on new questions · Real capability Signal holds on new questions How to measure · several benchmarks agree · Real capability How to measure several benchmarks agree How to protect · task-specific evals · Real capability How to protect task-specific evals Already seen · test items in training data · Already seen: contamination Already seen test items in training data Signal · drops on rewritten tests · Already seen: contamination Signal drops on rewritten tests How to measure · order, Min-K%, by date · Already seen: contamination How to measure order, Min-K%, by date How to remove · live benchmarks, private sets · Already seen: contamination How to remove live benchmarks, private sets How you ask · format, order, scoring · Sensitivity to how you ask How you ask format, order, scoring Signal · new format, new score · Sensitivity to how you ask Signal new format, new score How to measure · run many formats · Sensitivity to how you ask How to measure run many formats How to remove · report a range, state scoring · Sensitivity to how you ask How to remove report a range, state scoring Noise · a finite question set · Statistical noise Noise a finite question set Signal · a 1–2 point gap · Statistical noise Signal a 1–2 point gap How to measure · SE, CI, paired tests · Statistical noise How to measure SE, CI, paired tests How to reduce · more questions, resampling · Statistical noise How to reduce more questions, resampling Legend part of the number how to measure how to reduce or remove how it shows up