Rollouts
Chapter 2 of 6
Glossary
Contents
Contents   /   Chapter 2

02

A distribution of answers

Separate what could happen from what happened in a few samples.

Possibilities are not counts

Suppose repeated responses to our prompt include 5, The answer is 5., and 6. They are three possible answers, not three steps in one answer.

A stochastic policy assigns probabilities; a sampled group records what was actually drawn. These are different objects. A probable answer can fail to appear in a small group. A less probable answer can appear more than once. Sampling permits variety, but does not promise it. Spinning Up: stochastic policies

A deliberately small world

To see that distinction without downloading a model, our lab uses only three whole-completion choices. These choices and logits are invented teaching inputs, not measurements of an LLM.

Toy whole-completion distribution at temperature 1
IndexCompletionLogitProbability
051.258.96%
1The answer is 5.0.426.49%
26−0.214.54%

A logit here is an unnormalized score. Softmax turns the three scores into probabilities:

pi(T)=exp⁡(zi/T)∑j=02exp⁡(zj/T)p_i(T)=\frac{\exp(z_i/T)}{\sum_{j=0}^{2}\exp(z_j/T)}

The probabilities sum to one before display rounding. The table's percentages sum to 99.99% only because each is rounded separately.

Figure 003Six samples. Three distinct answers.
One prompt fans into six separately sampled rolloutsDefault seeded toy group: draw 1: 5; draw 2: 5; draw 3: The answer is 5.; draw 4: The answer is 5.; draw 5: 5; draw 6: 6. Repeated text does not merge samples.SAME PROMPT1 · “5”2 · “5”3 · “The answer is 5.”4 · “The answer is 5.”5 · “5”6 · “6”one group · K = 6One prompt fans into six separately sampled rolloutsDefault seeded toy group: draw 1: 5; draw 2: 5; draw 3: The answer is 5.; draw 4: The answer is 5.; draw 5: 5; draw 6: 6. Repeated text does not merge samples.SAME PROMPT1 · “5”2 · “5”3 · “The answer is 5.”4 · “The answer is 5.”5 · “5”6 · “6”
Each line is one separately sampled whole completion, not a token-decoding tree. Equal line weights count draws, not probabilities. This is the fixed-seed toy group at T = 1, K = 6—not a measurement of an LLM.

Temperature changes the distribution

At temperature T=1T=1, 5 has probability about 0.5896. Lowering TT to 0.5 increases it to about 0.7919; raising TT to 2 lowers it to about 0.4615. Those numbers follow from the formula and the fixed toy logits.

This does not say that higher temperature is generally better or worse. It says what division by TT does to this particular distribution. Nor does changing a sampling setting train a model.

The lab draws an entire completion as one category. A real autoregressive language model selects tokens from changing, context-dependent distributions. The preceding trajectory picture explains that mechanism; this simplified sampler lets us inspect probabilities, sampled groups, and scores.

Check your understanding

If 5 has probability about 59%, must exactly 59% of six rollouts be 5?

Reveal answer

No. Six draws can only produce counts from zero to six. A finite sample is not the probability distribution itself.

Once we have sampled an answer, how do we decide what feedback it receives?