From reward to a relative signal

Start with a scoring rule

5 follows both parts of our instruction: it is correct, and it is one digit. The answer is 5. is correct but misses the format. 6 is incorrect.

Our toy verifier encodes those judgments as scores:

Invented rewards for this lesson only
Completion	Reward	Reason under this rule
`5`	1.00	Correct and correctly formatted
`The answer is 5.`	0.75	Correct, wrong format
`6`	0.00	Incorrect

The numbers are not calibrated measures of quality, and they do not mean short answers are always preferable. Real systems may instead use a learned reward model or a task-specific rule. InstructGPT §3.1 and §3.5 · DeepSeekMath §4.1.2

A score is not yet a comparison

Take a hand-authored group with rewards $[1,0.75,0,1]$ . This is an arithmetic example, not a seeded sample or a claim about frequencies. Its mean is

\bar r=\frac{1+0.75+0+1}{4}=0.6875

Now ask a different question: how far above or below the group average is each reward? Our teaching signal is

A_i^{\mathrm{toy}}=r_i-\bar r

Figure 004A score is a position. A signal is a difference.

Subtract the dashed mean from each reward. Right is positive; left is negative. This is the hand-authored [1, 0.75, 0, 1] arithmetic example (mean 0.6875), not the seeded sample. Centering alone is not full GRPO.

Check the arithmetic without the diagram

Hand-authored example, not a seeded sample; mean = 0.6875
Draw	Reward	Centered signal
1	1.0000	+0.3125
2	0.7500	+0.0625
3	0.0000	−0.6875
4	1.0000	+0.3125

A smaller hand-authored comparison makes the same point: [1, 0.75] has mean 0.8750, giving the 0.75 reward a centered value of −0.1250. In [0.75, 0], the mean is 0.3750 and that reward has centered value +0.3750. The reward is unchanged; its companions differ.

The results are $[+0.3125,+0.0625,-0.6875,+0.3125]$ . The reward 0.75 gives a positive signal because it lies above 0.6875. If the other rollouts raised the mean above 0.75, that same reward would give a negative signal. The score did not change; the comparison did.

This is a centered group-relative teaching signal, not full GRPO. Published outcome-supervision GRPO additionally scales by the group's standard deviation and places the result inside a richer optimization objective. DeepSeekMath §4.1.1–§4.1.2

Three terms worth separating

Reward is feedback. Return accumulates rewards under a declared rule. Advantage compares action value with a baseline; it is not another name for reward. Spinning Up: reward, return, and advantage

Our toy assumes zero intermediate task rewards, one terminal score, and no discounting or additional shaping. Only under those assumptions does the task return reduce to that final score:

G_t=\sum_{u=t}^{N}r_u=r_{\mathrm{terminal}}

How this relates to standard advantage

A standard definition is $A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s)$ , where $Q$ is the action value and $V$ is the state-value baseline. Our sampled group-mean calculation illustrates a relative comparison; it is not that general definition or a complete advantage estimator for every trainer.

A standardized estimator also needs an explicit convention when every reward is equal, because the group standard deviation is zero. This toy only mean-centers and never divides by the standard deviation.

Check your understanding

Two rollouts both receive reward 1. What are their mean-centered signals?

Reveal answer

Both are zero: their mean is 1. A high absolute score does not guarantee a positive relative signal. More generally, centered values sum to zero: $\sum_i(r_i-\bar r)=0$ .

Keep one rollout's reward fixed and watch its comparison change.