Rollouts
Chapter 6 of 6
Glossary
Contents
Contents   /   Chapter 6

06

Collection is not learning

Generating behavior and changing a policy are separate operations. Close the loop.

Find the step that changes weights

Imagine requesting eight answers instead of one. You now have more sampled behavior. You have not, just by generating it, changed the model's parameters.

That boundary is the reason to separate the stages of a learning loop:

Figure 002Generation creates data. Optimization changes weights.
The collection, feedback and update loopGenerate a rollout. Serving can return the text here. A learning path scores the rollout, estimates a signal and updates the current policy. Only optimization changes weights. Later collection can start another round.1. Generate2. Score3. Estimate4. Updateweights change here5. Collect againor return textThe collection, feedback and update loopGenerate a rollout. Serving can return the text here. A learning path scores the rollout, estimates a signal and updates the current policy. Only optimization changes weights. Later collection can start another round.1. Generate2. Score3. Estimate4. Updateweights change here5. Collect againor return text
Serving can exit after generation. A learning path continues to feedback, estimation, and an update. This is a representative conceptual order, not a required production schedule.

Collection produces trajectories under a recorded behavior policy. Scoring supplies feedback. An estimator constructs the quantities an algorithm needs. Optimization changes the trainable policy; later collection can then use a newer snapshot.

PPO and DeepSeekMath describe this alternation between collecting behavior and optimizing a policy. The diagram is a conceptual order, not a demand that production systems run every stage serially. PPO Algorithm 1 · DeepSeekMath Algorithm 1

The same generation can have a different job

Ordinary serving inference and rollout collection can use the same basic autoregressive generation mechanics. In serving, the immediate job is to return an answer. In an RL workflow, generated behavior and the metadata required by the method feed a learning path. This is a distinction in purpose and downstream use, not a claim that every serving and training stack is identical.

The browser lab stops well before optimization. It has no token-level likelihoods, learned reward model, critic, KL calculation, policy-ratio loss, clipping, or optimizer. Its changing plot is not a training curve.

Take the idea somewhere new

Try a different task:

A shelf has 4 green cards and 3 black cards. Return JSON with one key named n.

A new stipulated toy rule gives {"n": 7} reward 1, plain 7 reward 0.6, and {"n": 8} reward 0. A four-rollout group scores [1,0.6,0,1][1,0.6,0,1]. Behavior version 12 generated it; the proposed update targets version 14.

Your turn

How many rollouts are there? Calculate their mean and centered signals. Is a positive reward necessarily above the baseline? Has collection changed version 14?

Reveal the worked answer

Four rollouts. The mean is rˉ=(1+0.6+0+1)/4=0.65\bar r=(1+0.6+0+1)/4=0.65, giving centered signals [+0.35,−0.05,−0.65,+0.35][+0.35,-0.05,-0.65,+0.35].

Reward 0.6 is positive but below the group average. Collection alone changes no weights. And the data still belongs to behavior version 12; an algorithm contract must decide its use for version 14.

You now have the complete small model: a trajectory records what happened; reward scores it; a relative signal compares it; provenance identifies who produced it; optimization is the separate step that changes the policy.