What was the rollout?

You give a language model a small task:

A box has 3 red labels and 2 blue labels. How many labels? Reply with one digit.

It replies 5. What, exactly, was the rollout?

Not the digit alone. In this article, one rollout is one sampled trajectory: the path from the starting prompt through generated actions to a stopping condition. The completed text is the part you can read. The path explains how that text was produced.

RL introductions commonly use trajectory, episode, and rollout for this idea, although individual systems may use “rollout” for a shorter segment. Spinning Up: trajectories

A longer context, one action at a time

For a simple text-only, single-turn interaction, picture the prompt as the beginning of a line. The model selects a token, appends it, and uses that longer context for the next selection. Eventually generation stops.

We can name the parts without guessing how a particular tokenizer would split the words:

Here $x$ is the prompt, $s_t$ is the prompt plus the generated prefix, and $a_t$ is the next token. The policy $\pi_{\mathrm{beh}}$ supplies the action probabilities. Repeating these steps gives one trajectory, $\tau=(x,a_1,\ldots,a_N)$, where $N$ is the number of generated tokens.

The subscript “beh” means this policy generated the behavior; chapter 5 will make that distinction useful. This token/state mapping is a synthesis for the stated text-only case, not a model of every tool-using agent. DeepSeekMath §4.1.1

First collect, then evaluate

The rollout exists before a reward function scores it. A score can be attached afterward; an update may happen later still. Neither a correct answer nor a completed update is required to identify the trajectory.

If you independently generate six responses to the same prompt, you have six rollouts—even if every response reads 5. You do not have one rollout with six steps.

One answer tells you what happened once. What tells you what could happen?

Possibilities are not counts

Suppose repeated responses to our prompt include 5, The answer is 5., and 6. They are three possible answers, not three steps in one answer.

A stochastic policy assigns probabilities; a sampled group records what was actually drawn. These are different objects. A probable answer can fail to appear in a small group. A less probable answer can appear more than once. Sampling permits variety, but does not promise it. Spinning Up: stochastic policies

A deliberately small world

To see that distinction without downloading a model, our lab uses only three whole-completion choices. These choices and logits are invented teaching inputs, not measurements of an LLM.

A logit here is an unnormalized score. Softmax turns the three scores into probabilities:

The probabilities sum to one before display rounding. The table's percentages sum to 99.99% only because each is rounded separately.

Temperature changes the distribution

At temperature $T=1$, 5 has probability about 0.5896. Lowering $T$ to 0.5 increases it to about 0.7919; raising $T$ to 2 lowers it to about 0.4615. Those numbers follow from the formula and the fixed toy logits.

This does not say that higher temperature is generally better or worse. It says what division by $T$ does to this particular distribution. Nor does changing a sampling setting train a model.

The lab draws an entire completion as one category. A real autoregressive language model selects tokens from changing, context-dependent distributions. The preceding trajectory picture explains that mechanism; this simplified sampler lets us inspect probabilities, sampled groups, and scores.

Once we have sampled an answer, how do we decide what feedback it receives?

Start with a scoring rule

5 follows both parts of our instruction: it is correct, and it is one digit. The answer is 5. is correct but misses the format. 6 is incorrect.

Our toy verifier encodes those judgments as scores:

The numbers are not calibrated measures of quality, and they do not mean short answers are always preferable. Real systems may instead use a learned reward model or a task-specific rule. InstructGPT §3.1 and §3.5 · DeepSeekMath §4.1.2

A score is not yet a comparison

Take a hand-authored group with rewards $[1,0.75,0,1]$. This is an arithmetic example, not a seeded sample or a claim about frequencies. Its mean is

Now ask a different question: how far above or below the group average is each reward? Our teaching signal is

The results are $[+0.3125,+0.0625,-0.6875,+0.3125]$. The reward 0.75 gives a positive signal because it lies above 0.6875. If the other rollouts raised the mean above 0.75, that same reward would give a negative signal. The score did not change; the comparison did.

This is a centered group-relative teaching signal, not full GRPO. Published outcome-supervision GRPO additionally scales by the group's standard deviation and places the result inside a richer optimization objective. DeepSeekMath §4.1.1–§4.1.2

Three terms worth separating

Reward is feedback. Return accumulates rewards under a declared rule. Advantage compares action value with a baseline; it is not another name for reward. Spinning Up: reward, return, and advantage

Our toy assumes zero intermediate task rewards, one terminal score, and no discounting or additional shaping. Only under those assumptions does the task return reduce to that final score:

Keep one rollout's reward fixed and watch its comparison change.

Follow draw 3

The most useful thing to track is not the entire table. Track draw 3.

At $T=1$, that draw is The answer is 5. and always receives reward 0.75 from our fixed verifier. Before changing anything, predict: if we give it different companions, must its relative signal keep the same sign?

state = (1664525 * state + 1013904223) mod 2^32
u = state / 2^32

rewards = [1, 1, 0.75, 0.75, 1, 0]
mean = sum(rewards) / len(rewards)
centered = [reward - mean for reward in rewards]
# mean: 0.75
# centered: [0.25, 0.25, 0, 0, 0.25, -0.75]

Change the group, not the reward rule

Leave temperature at 1. Try group sizes $K=3$, $K=6$, and $K=8$. Here $K$ counts separately drawn whole completions; it does not count tokens.

These values follow from the frozen sampler and reward rule. The same 0.75 score is below one mean, equal to another, and above the third. The baseline is part of the answer whenever you discuss a relative signal.

At the default $T=1$, $K=6$, seed 20260903, the recorded local toy run drew candidate indices $[0,0,1,1,0,2]$. Their rewards are $[1,1,0.75,0.75,1,0]$, their mean is 0.75, and their centered signals are $[0.25,0.25,0,0,0.25,-0.75]$. This is output of the disclosed seeded toy process, not samples from a language model. Recorded lab output (local copy)

Now change temperature

Predict whether increasing $T$ from 1 to 2 makes 5 more or less probable. Then try it. Follow the chain: temperature changes probabilities; probabilities and the fixed random draws determine the new group; the reward rule determines its scores; those scores determine the mean.

Every setting starts a prospective draw from the same initial seed. Reset restores $T=1$, $K=6$, and seed 20260903. Nothing edits the verifier's reward for a recorded completion, and nothing updates model weights.

This remains a whole-completion toy and centered teaching signal—not full GRPO.

A useful score still does not tell us which policy produced the data.

The answer did not change its author

Suppose model version 12 generated a rollout. The trainable model has since advanced to version 14. The response still says 5, and its reward is still 1. Has it become version-14 data?

No. A useful answer can be old behavior. The text cannot tell you the policy that generated it.

Three names, three jobs

The behavior policy is the distribution that generated the recorded actions. For this example, $\pi_{\mathrm{beh}}=\pi_{v12}$.

The current policy is the trainable policy under consideration now. Here, $\pi_{\mathrm{cur}}=\pi_{v14}$. In ratio-based methods such as PPO and GRPO, comparisons use the current policy's probabilities and those of the behavior or old policy that generated the actions. PPO §3 · DeepSeekMath §4.1.1

An optional reference policy supplies a different comparison—for example, a KL penalty that discourages moving too far from a reference distribution. It is not automatically the behavior policy; it is not the reward model or the group mean either. Our notation could name it $\pi_{\mathrm{ref-v1}}$, separately identified from both versions above. InstructGPT §3.5 · DeepSeekMath equation 3

These are functional roles, not necessarily three unrelated architectures or three permanent model copies. At fresh collection, current and behavior weights may coincide. A method may omit a reference role altogether.

On-policy is about provenance

An introductory on-policy distinction asks whether the data comes from the policy used for the update, rather than an older or different behavior policy. It is not a label for whether an answer is correct. Spinning Up: kinds of RL algorithms

There is a practical qualification: original PPO can reuse a freshly collected batch for several minibatch epochs. “On-policy” therefore does not mean collecting a new rollout before every individual gradient step. It also does not grant unlimited reuse of old data. PPO abstract and Algorithm 1

We can finally put collection, feedback, and learning on one timeline.

Find the step that changes weights

Imagine requesting eight answers instead of one. You now have more sampled behavior. You have not, just by generating it, changed the model's parameters.

That boundary is the reason to separate the stages of a learning loop:

Collection produces trajectories under a recorded behavior policy. Scoring supplies feedback. An estimator constructs the quantities an algorithm needs. Optimization changes the trainable policy; later collection can then use a newer snapshot.

PPO and DeepSeekMath describe this alternation between collecting behavior and optimizing a policy. The diagram is a conceptual order, not a demand that production systems run every stage serially. PPO Algorithm 1 · DeepSeekMath Algorithm 1

The same generation can have a different job

Ordinary serving inference and rollout collection can use the same basic autoregressive generation mechanics. In serving, the immediate job is to return an answer. In an RL workflow, generated behavior and the metadata required by the method feed a learning path. This is a distinction in purpose and downstream use, not a claim that every serving and training stack is identical.

The browser lab stops well before optimization. It has no token-level likelihoods, learned reward model, critic, KL calculation, policy-ratio loss, clipping, or optimizer. Its changing plot is not a training curve.

Take the idea somewhere new

Try a different task:

A shelf has 4 green cards and 3 black cards. Return JSON with one key named n.

A new stipulated toy rule gives {"n": 7} reward 1, plain 7 reward 0.6, and {"n": 8} reward 0. A four-rollout group scores $[1,0.6,0,1]$. Behavior version 12 generated it; the proposed update targets version 14.

You now have the complete small model: a trajectory records what happened; reward scores it; a relative signal compares it; provenance identifies who produced it; optimization is the separate step that changes the policy.