Quality filtering: what to keep, and what you lose by keeping too little

Quality filtering: what to keep, and what you lose by keeping too little A workflow diagram generated by Archify. 01 / Keep path — heuristics first, classifier second EX / Drop path — and the price that comes with it Document · has text ≠ worth learning · Keep path — heuristics first, classifier second Document has text ≠ worth learning Measure features · length · letter ratio · Keep path — heuristics first, classifier second Measure features length · letter ratio Heuristic thresholds · cheap at any scale · Keep path — heuristics first, classifier second Heuristic thresholds cheap at any scale Quality classifier · learned from labels · Keep path — heuristics first, classifier second Quality classifier learned from labels Quality score · threshold you choose · Keep path — heuristics first, classifier second Quality score threshold you choose Into training mixture · tokens actually learned · Keep path — heuristics first, classifier second · unique tokens left Into training mixture tokens actually learned unique tokens left Dropped by rule · spam · repeated lines · Drop path — and the price that comes with it Dropped by rule spam · repeated lines Dropped by score · below threshold · Drop path — and the price that comes with it Dropped by score below threshold False drops · code · poetry · tables · Drop path — and the price that comes with it False drops code · poetry · tables raw text features passed rules model score above cutoff breaks a rule below threshold good data too Legend Agent logic Policy Tool action Context / trace External system