Deduplication: hashes catch copies, MinHash catches near-duplicates

Deduplication: hashes catch copies, MinHash catches near-duplicates A workflow diagram generated by Archify. 01 / Near-duplicates — MinHash and LSH, usable at web scale 02 / Exact duplicates — full-text hash, the cheapest EX / The hash blind spot Document · one article · Near-duplicates — MinHash and LSH, usable at web scale Document one article n-gram shingles · runs of n words · Near-duplicates — MinHash and LSH, usable at web scale n-gram shingles runs of n words MinHash signature · short fixed vector · Near-duplicates — MinHash and LSH, usable at web scale MinHash signature short fixed vector LSH buckets · compare within bucket · Near-duplicates — MinHash and LSH, usable at web scale LSH buckets compare within bucket Candidate pairs · far fewer pairs · Near-duplicates — MinHash and LSH, usable at web scale Candidate pairs far fewer pairs Jaccard check · confirm, then drop · Near-duplicates — MinHash and LSH, usable at web scale Jaccard check confirm, then drop Content hash · normalize, then hash · Exact duplicates — full-text hash, the cheapest Content hash normalize, then hash Hash seen? · look up a hash set · Exact duplicates — full-text hash, the cheapest Hash seen? look up a hash set Drop the copy · byte-identical · Exact duplicates — full-text hash, the cheapest Drop the copy byte-identical Treated as new · even one character off · The hash blind spot Treated as new even one character off cheap path hash string seen not seen to n-grams minimum hash hash by band same bucket compare shingles Legend Agent logic Policy Tool action Context / trace External system