The contract between tokenizer, embeddings, and weights

The contract between tokenizer, embeddings, and weights An architecture diagram generated by Archify. Pretraining tokenizer · "cat" → 4281 · Three parts co-adapted throughout pretraining · produces IDs Pretraining tokenizer "cat" → 4281 produces IDs Embedding matrix E · row 4281 learned for "cat" · Three parts co-adapted throughout pretraining · |V| × d parameters Embedding matrix E row 4281 learned for "cat" |V| × d parameters Transformer + LM head · logits over exactly |V| dims · Three parts co-adapted throughout pretraining · adapted to this representation Transformer + LM head logits over exactly |V| dims adapted to this representation Switch to another tokenizer · "house" → 4281 · Architecture component Switch to another tokenizer "house" → 4281 Semantic mismatch · old vector, new token · Architecture component Semantic mismatch old vector, new token Add a new token · no row in E yet · Architecture component Add a new token no row in E yet Initialize, then keep training · random · old-token mix · transfer · Architecture component Initialize, then keep training random · old-token mix · transfer ID = row address input vectors still fetches row 4281 returns the old token's vector needs a new row append a row to E Three parts co-adapted throughout pretraining Legend Frontend Backend Database Cloud Security External