Splitting data and the leakage paths

Splitting data and the leakage paths A workflow diagram generated by Archify. 01 / Data preparation 02 / Role of each split EX / Data leakage Before splitting Split After splitting Raw dataset · text + labels · Data preparation › Before splitting Raw dataset text + labels Deduplication · by content and by group · Data preparation › Before splitting · do before splitting Deduplication by content and by group do before splitting Split · train / validation / test · Data preparation › Split Split train / validation / test Train · forward → backward → update · Role of each split › Split · learned from Train forward → backward → update learned from Validation · pick checkpoint, LR · Role of each split › After splitting · measures only Validation pick checkpoint, LR measures only Test · run once, at the end · Role of each split › After splitting · stays unbiased Test run once, at the end stays unbiased Fit on everything · scaler, vocab, statistics · Data leakage › Before splitting Fit on everything scaler, vocab, statistics Duplicate samples · same text in two splits · Data leakage › Split Duplicate samples same text in two splits Peeking at test · test turns into validation · Data leakage › After splitting Peeking at test test turns into validation what the model learns evaluate after each epoch after tuning is done fit before splitting dedup skipped tuned on test score Legend Agent logic Policy Tool action Context / trace Cloud service External system