GB, documents or tokens: three counts that are not interchangeable

GB, documents or tokens: three counts that are not interchangeable An architecture diagram generated by Archify. 500 GB · varies with encoding and markup · Architecture component 500 GB varies with encoding and markup 2 trillion tokens · a comparable unit · Architecture component · closest to compute 2 trillion tokens a comparable unit closest to compute Training steps · tokens ÷ (batch × T) · Architecture component Training steps tokens ÷ (batch × T) 100 million documents · 20 or 100,000 tokens both count as 1 · Architecture component 100 million documents 20 or 100,000 tokens both count as 1 Compute budget · real GPU-hours · Architecture component Compute budget real GPU-hours Model quality · what we actually care about · Architecture component Model quality what we actually care about Parameter count · model capacity · Architecture component Parameter count model capacity Undertrained · many parameters, few tokens · Architecture component Undertrained many parameters, few tokens tokenize to count very different lengths divide by batch × T number of updates budget for data budget for model usable capacity tokens not increased Legend Frontend Backend Database Cloud Message bus External