Reading companion
Glossary & sources
The short definitions, exact assumptions, and evidence behind the book.
A small vocabulary
- Rollout
- A sampled trajectory; here, one complete generation from a prompt to a stopping condition.
- Policy
- A rule for selecting actions. A stochastic policy assigns probabilities to actions.
- State
- In this single-turn text-only mapping, the prompt and tokens generated so far.
- Action
- The next sampled token in the trajectory picture—not an entire completion in a real autoregressive model.
- Reward
- Feedback from a declared scoring rule, verifier, or reward model.
- Return
- Rewards accumulated under a stated convention, potentially including discounting or shaping.
- Advantage
- A relative value comparison. It is not another name for raw reward.
- Baseline
- The comparison value subtracted to form a relative signal. Our local toy uses the group mean.
- Behavior policy
- The policy distribution that generated the stored actions.
- Current policy
- The trainable policy considered for an update now.
- Reference policy
- An optional, separately identified comparison policy, for example a KL anchor.
- Temperature
- A sampling parameter. Here it divides fixed whole-completion logits before softmax. Changing it does not update model weights.
Primary sources
- OpenAI Spinning Up: Key Concepts in RL — policies, trajectories, reward, return, and advantage.
- Spinning Up: Kinds of RL Algorithms — on-policy and off-policy distinction.
- Schulman et al., Proximal Policy Optimization Algorithms (2017) — policy ratios and collection/update loop.
- Ouyang et al., InstructGPT (2022) — reward feedback and reference-policy comparison.
- Shao et al., DeepSeekMath (2024) — GRPO group-relative estimation and policy roles.
- Transformers generation documentation and TRL GRPO documentation — implementation examples, not universal defaults. The original ledger checked these on September 3, 2026; options can change.
What the toy leaves out
The lab is a whole-completion categorical sampler. It does not implement token-level probabilities, a tokenizer, a learned reward model, a critic, standard-deviation-scaled GRPO, policy ratios, clipping, KL evaluation, gradients, an optimizer, or production orchestration. Its numerical behavior is not evidence of model quality, convergence, safety, or learner outcomes.
The local mean-centered signal is illustrative. A canonical advantage definition is:
The baseline in the lab is instead the mean of a particular sampled group. Do not substitute it into another trainer without checking that trainer's estimator.
Evidence and reproducibility
Source-backed facts have adjacent source links. Invented inputs say “toy”; arithmetic follows the displayed equations; the recorded Python run is a seeded toy-process capture, not measured LLM behavior. Implementation-dependent questions stay qualified.
Full claim ledger (local copy) · Canonical Python source (local copy) · Captured run (local copy) · Lab manifest (local copy)
Traceability index (for reviewers)
Stable concepts: CON-001, CON-002, CON-003, CON-004, CON-005, CON-006, CON-007, CON-008, CON-009, CON-010, CON-011. The complete ledger and original blueprint retain each definition; chapter sources map the active reading path.
Additional arithmetic preserved from the first edition: one-of-each rewards [1, 0.75, 0] have mean 7/12 and centered values [5/12, 1/6, −7/12] (CL-037–038). The separate CL-044 group [1, 0.75, 0, 1] has mean 0.6875 and centered values [+0.3125, +0.0625, −0.6875, +0.3125].
About this edition
This chaptered edition replaces the original single-page presentation. It does not inherit that version's browser audit or its directional comparison scores. Its improvement is a design and implementation result, not a controlled test of learning gains.
The restrained chapter-and-figure format was informed by Learn Inference, Learn UI, and Learn Kernels. The prose and diagrams here are independently authored for this topic. Mathematics is typeset with KaTeX, bundled locally.