Convert probability to information.
A common token carries less surprisal. A rare token carries more. The horizontal chart position is exactly negative log probability in bits.
An information-theoretic decoding experiment
Decode near expected surprise.
The best next token is not always the most probable one. Locally typical sampling keeps tokens whose information content is close to the model's expected information at this step.
entropy H
thetoo expected
quietlynear H
throughnear H
xylophonetoo surprising
Typicality ranks distance from expected surprisal, then retains enough probability mass to sample.
Decode around the expected surprise.
This transparent synthetic distribution is not a production model benchmark. It isolates the exact selection rule so probability, surprisal, entropy, and retained mass remain inspectable.
Current context
The lantern flickered as Mira stepped into
retained filtered conditional entropy
Generated continuationLocally typical
waiting
--mean distance from H
Conditional entropy3.82 bits
Retained tokens6 / 12
Actual retained mass82%
Mean typicality distance0.41 bits
The rule in three calculations.
Typical sampling changes the ranking criterion. It does not rank tokens by probability alone: it ranks their absolute distance from the distribution's expected information content.
I(x) = -log2 p(x)
H(p) = sum p(x) I(x)
typicality(x) = |I(x) - H(p)|
H(p) = sum p(x) I(x)
typicality(x) = |I(x) - H(p)|
Compute expected information.
Conditional entropy is the probability-weighted average surprisal for this next-token distribution. It moves when temperature reshapes the distribution.
Fill the set by typicality.
Sort tokens by distance from entropy and add them until their cumulative probability reaches the configured mass. Renormalize only over that retained set, then sample.
Same probabilities, different ideas of a credible token.
Top-k imposes a fixed count. Nucleus sampling follows cumulative probability. Locally typical sampling follows information content relative to entropy.
Locally typical
Can filter the single most probable token when it is much less surprising than the distribution expects. The retained set need not be a probability prefix.
6 tokensTop-p nucleus
Sorts by probability and keeps the smallest prefix whose cumulative mass reaches p.
5 tokensTop-k
Keeps a fixed number regardless of whether the distribution is sharp or flat.
4 tokensInterpret the output carefully.
A smaller typicality distance means information content is closer to the model's expectation.
Typical sampling is a decoding rule, not proof that a sampled token is factually correct.
Temperature changes both token probabilities and the entropy target, so it can move both dots and the red line.
Match the sampler to the failure mode you are studying.
No truncation rule guarantees quality. Compare distributions, tasks, and human judgments instead of treating a single parameter as universal.
Read the primary work.
Locally Typical SamplingMeister, Pimentel, Wiher, and Cotterell - TACL 2023
The Curious Case of Neural Text DegenerationHoltzman et al. - nucleus sampling and degeneration
On the Probability-Quality Paradox in Language GenerationMeister et al. - an information-theoretic account
Truncation Sampling as Language Model DesmoothingHewitt, Manning, and Liang - entropy-dependent truncation
Inspect the distribution before choosing the decoder.
Use this lab to understand the selection mechanics, then evaluate real generations with task-specific quality, safety, factuality, and diversity measurements.