Three layers for debugging an LLM system

Three layers for debugging an LLM system A workflow diagram generated by Archify. 01 / Layer 1 · Representation 02 / Layer 2 · Neural computation 03 / Layer 3 · Inference system Triage Check within the layer Fix and re-measure Tokenizer · decode input_ids · Layer 1 · Representation › Check within the layer Tokenizer decode input_ids Padding + mask · 1 real / 0 pad · Layer 1 · Representation › Check within the layer Padding + mask 1 real / 0 pad Symptom · wrong / slow / OOM · Layer 2 · Neural computation › Triage Symptom wrong / slow / OOM Which layer? · ask before fixing · Layer 2 · Neural computation › Triage · don't swap model yet Which layer? ask before fixing don't swap model yet Task head · right AutoModelFor? · Layer 2 · Neural computation › Check within the layer Task head right AutoModelFor? Logits + labels · shape and id2label · Layer 2 · Neural computation › Check within the layer Logits + labels shape and id2label Fix the right layer · one change at a time · Layer 2 · Neural computation › Fix and re-measure Fix the right layer one change at a time Re-measure · same input and batch · Layer 2 · Neural computation › Fix and re-measure Re-measure same input and batch Memory · dtype, KV cache · Layer 3 · Inference system › Check within the layer Memory dtype, KV cache Batching + kernel · TTFT, tokens/s · Layer 3 · Inference system › Check within the layer Batching + kernel TTFT, tokens/s garbage output slow or OOM batch changes result measure both phases fix preprocessing fix serving config wrong label/shape read output right fix the head Legend User UI Agent logic Policy Tool action Context / trace Cloud service External system