A visual introduction to reinforcement learning for LLMs
Rollouts, explained. From a generated answer to a learning signal. Six short chapters, one running example, and a lab you can reason through.
Start with prompts, tokens, and an average. No prior reinforcement learning needed. The lab uses an explicitly simplified toy—not a real model or trainer.
01 One rolloutAn answer is the visible end of a path. Start by seeing the path. ↗ 02 A distribution of answersSeparate what could happen from what happened in a few samples. ↗ 03 From reward to a relative signalA score tells you how an answer did. A baseline tells you what to compare it with. ↗ 04 The rollout labKeep one answer fixed. Give it different companions. Watch its relative signal change. ↗ 05 Which policy made this?Good answers can be old behavior. The response text cannot tell you its provenance. ↗ 06 Collection is not learningGenerating behavior and changing a policy are separate operations. Close the loop. ↗