Which policy made this?

The answer did not change its author

Suppose model version 12 generated a rollout. The trainable model has since advanced to version 14. The response still says 5, and its reward is still 1. Has it become version-14 data?

No. A useful answer can be old behavior. The text cannot tell you the policy that generated it.

Three names, three jobs

The behavior policy is the distribution that generated the recorded actions. For this example, $\pi_{\mathrm{beh}}=\pi_{v12}$.

The current policy is the trainable policy under consideration now. Here, $\pi_{\mathrm{cur}}=\pi_{v14}$. In ratio-based methods such as PPO and GRPO, comparisons use the current policy's probabilities and those of the behavior or old policy that generated the actions. PPO §3 · DeepSeekMath §4.1.1

An optional reference policy supplies a different comparison—for example, a KL penalty that discourages moving too far from a reference distribution. It is not automatically the behavior policy; it is not the reward model or the group mean either. Our notation could name it $\pi_{\mathrm{ref-v1}}$, separately identified from both versions above. InstructGPT §3.5 · DeepSeekMath equation 3

These are functional roles, not necessarily three unrelated architectures or three permanent model copies. At fresh collection, current and behavior weights may coincide. A method may omit a reference role altogether.

On-policy is about provenance

An introductory on-policy distinction asks whether the data comes from the policy used for the update, rather than an older or different behavior policy. It is not a label for whether an answer is correct. Spinning Up: kinds of RL algorithms

There is a practical qualification: original PPO can reuse a freshly collected batch for several minibatch epochs. “On-policy” therefore does not mean collecting a new rollout before every individual gradient step. It also does not grant unlimited reuse of old data. PPO abstract and Algorithm 1

We can finally put collection, feedback, and learning on one timeline.