Unconstrained sampling preserves every candidate.
A high-probability closing bracket can be selected even when the parser expects a property value. Once emitted, downstream repair must guess what the model intended.
A transparent structured-generation experiment
Only valid tokens survive.
Constrained decoding does not repair malformed JSON after generation. It changes the next-token distribution during generation, removing choices that cannot lead to a syntax- or schema-valid result.
candidate token distributionschema state: property value
"Paris""Tokyo"]"kelvin"null
0.61 -> 1.00 mass
After invalid tokens are masked, the remaining probability mass is renormalized before a token is selected.
Build one valid object, one token at a time.
The vocabulary and completion space are finite so every candidate, mask, and probability remains inspectable. Ajv verifies the full completion against the visible schema.
raw probability retained after mask masked candidate
Choose the next token
Percentages show the post-mask distributionraw mass retained--
valid next tokens--
renormalization--
schema completions left--
The same logits produce different reachable outputs.
A constraint engine changes which paths remain possible. It does not change the model's learned logits or guarantee that a valid value is factually correct.
Syntax is not schema.
{"unit":"kelvin"} may be valid JSON while violating an enum that permits only "C" or "F".
Validation happens during search.
Each prefix is checked against the set of possible valid continuations. Tokens with no accepting path receive zero probability.
Inspect the active schema.
Interpret the guarantee precisely.
A complete result can be guaranteed to match the represented grammar and schema vocabulary.
A schema-valid tool call can still request the wrong city, contact, or order status.
Finite vocabularies make this lab exact; production engines compile broader grammars into efficient token-level recognizers.
From schema to next-token mask.
The production implementation may use automata, parsers, tries, or specialized kernels. The logical loop remains the same.
Compile the accepted language.
Turn a grammar or JSON Schema into a recognizer that can answer which token extensions still have a path to an accepting state.
Mask logits at every decoding step.
Set the logits of tokens that cannot extend the current prefix to negative infinity before softmax or sampling.
Renormalize and continue.
Redistribute probability over the surviving tokens, select one, advance the recognizer state, and repeat until an accepting completion is reached.
Know what each constraint can and cannot promise.
Primary sources and standards.
Follow the original methods and specifications behind constrained structured generation.
Grammar-Constrained Decoding for Structured NLP TasksGeng et al., EMNLP 2023
PICARD: Parsing Incrementally for Constrained Auto-Regressive DecodingScholak et al., EMNLP 2021
LMQL: A Programming Language for Large Language ModelsBeurer-Kellner et al., 2022
JSON Schema Core, Draft 2020-12JSON Schema specification