From a web page to a clean document

From a web page to a clean document A data-flow diagram generated by Archify. 01 / Web page 02 / Crawl archive 03 / Extraction 04 / Normalization 05 / Corpus Web page · nav · article · ads · footer · 01 / Web page · content mixed with boilerplate Web page nav · article · ads · footer content mixed with boilerplate Crawler · URL · headers · timestamp · 01 / Web page Crawler URL · headers · timestamp WARC · raw response as-is · 02 / Crawl archive WARC raw response as-is WAT and WET · metadata · ready-made plaintext · 02 / Crawl archive WAT and WET metadata · ready-made plaintext Main-content extraction · keep code, tables, formulas · 03 / Extraction · a design decision Main-content extraction keep code, tables, formulas a design decision Discarded parts · nav · cookie · related posts · 03 / Extraction Discarded parts nav · cookie · related posts Normalization · Unicode · whitespace · control · 04 / Normalization · too aggressive breaks structure Normalization Unicode · whitespace · control too aggressive breaks structure Language ID · en 0.99 · vi 0.97 · mixed? · 04 / Normalization Language ID en 0.99 · vi 0.97 · mixed? Plain-text document · with source metadata · 05 / Corpus · still far from an ideal dataset Plain-text document with source metadata still far from an ideal dataset HTTP response full HTML main text normalized text recorded as-is plaintext shortcut removed from the document language label Legend primary data async batch data store data flow