Constrained decoding: compile the schema once, mask the logits at every step

Constrained decoding: compile the schema once, mask the logits at every step A workflow diagram generated by Archify. 01 / Once, before generation 02 / At every generation step JSON Schema · or regex, CFG · Once, before generation JSON Schema or regex, CFG Compile · into an FSM or PDA · Once, before generation Compile into an FSM or PDA Token index · state → valid tokens · Once, before generation Token index state → valid tokens Current state · e.g. expecting a key · At every generation step Current state e.g. expecting a key Bitmask · 1 bit per token · At every generation step Bitmask 1 bit per token Mask logits · then sample as usual · At every generation step Mask logits then sample as usual New token · always valid syntax · At every generation step New token always valid syntax advance one FSM step initial state look up Legend input supplied by the user compute step hard mask over the vocabulary state that changes with every token precomputed lookup table