Padding, packing and document boundaries in a sequence

Padding, packing and document boundaries in a sequence An architecture diagram generated by Archify. Doc A and Doc B · 4 tokens and 3 tokens · Architecture component Doc A and Doc B 4 tokens and 3 tokens Pad each document · [A A A EOS PAD PAD PAD] · Architecture component Pad each document [A A A EOS PAD PAD PAD] PAD positions · GPU computes, learns nothing · Architecture component PAD positions GPU computes, learns nothing Concatenate, then chunk · [A A A EOS B B EOS C] · Architecture component · higher utilization Concatenate, then chunk [A A A EOS B B EOS C] higher utilization Fixed-length sequence · one sample, many documents · Architecture component Fixed-length sequence one sample, many documents Cross-boundary attention · B1 sees A1 A2 A3 · Two options when a sequence holds several documents Cross-boundary attention B1 sees A1 A2 A3 Document-boundary mask · each block sees only itself · Two options when a sequence holds several documents Document-boundary mask each block sees only itself pad to full length the leftover space concatenate cut by length let attention run through block at boundary Two options when a sequence holds several documents Legend Backend Database Security Message bus External