One rollout

What was the rollout?

You give a language model a small task:

A box has 3 red labels and 2 blue labels. How many labels? Reply with one digit.

It replies 5. What, exactly, was the rollout?

Not the digit alone. In this article, one rollout is one sampled trajectory: the path from the starting prompt through generated actions to a stopping condition. The completed text is the part you can read. The path explains how that text was produced.

RL introductions commonly use trajectory, episode, and rollout for this idea, although individual systems may use “rollout” for a shorter segment. Spinning Up: trajectories

A longer context, one action at a time

For a simple text-only, single-turn interaction, picture the prompt as the beginning of a line. The model selects a token, appends it, and uses that longer context for the next selection. Eventually generation stops.

We can name the parts without guessing how a particular tokenizer would split the words:

Here $x$ is the prompt, $s_t$ is the prompt plus the generated prefix, and $a_t$ is the next token. The policy $\pi_{\mathrm{beh}}$ supplies the action probabilities. Repeating these steps gives one trajectory, $\tau=(x,a_1,\ldots,a_N)$, where $N$ is the number of generated tokens.

The subscript “beh” means this policy generated the behavior; chapter 5 will make that distinction useful. This token/state mapping is a synthesis for the stated text-only case, not a model of every tool-using agent. DeepSeekMath §4.1.1

First collect, then evaluate

The rollout exists before a reward function scores it. A score can be attached afterward; an update may happen later still. Neither a correct answer nor a completed update is required to identify the trajectory.

If you independently generate six responses to the same prompt, you have six rollouts—even if every response reads 5. You do not have one rollout with six steps.

One answer tells you what happened once. What tells you what could happen?