↗SSL
Chapter 2 of 5
Notes & sources
Contents ≡
Contents / Chapter 2

02

Predicting the missing part

The original data can supply the answer without giving that answer to the predictor.

Keep the answer on a separate path

Our invented image has 8×8=648\times8=64 binary pixels: 1 for an arrow pixel and 0 for the background. We hide the top-right 4×44\times4 region. The predictor would receive the remaining context and the positions of the missing pixels. A separate target path keeps the 16 original values for scoring.

FIG-002Keep the answer out of the input.
Context is not the answer keyThe upper-right sixteen pixels are absent from the context. The original target contains ten ones and six zeros. A prescribed half-valued prediction gives mean squared error one quarter.CONTEXT16 values hiddenPREDICTION0.5 at every pixelTARGET10 ones · 6 zeros−MSE0.25The loss sees the target. The predictor does not.Context is not the answer keySixteen masked pixels are kept separate from their original target. A fixed half-valued prediction has masked mean squared error 0.25.CONTEXT16 values hiddenPREDICTIONTARGET−all 0.510 ones / 16Masked MSE = 0.25

Crossed cells are unavailable input values, not zeros. Gray prediction cells are prescribed 0.5 values—not output from a trained decoder. Only the 16 hidden positions are scored.

The distinction is about roles, not separate datasets. We can inspect the complete original to construct the exercise. The hypothetical predictor must not receive those hidden values as its input answer key. Otherwise, a low reconstruction error could come from copying the supplied answer rather than using the visible context.

Score the sixteen hidden pixels

Let MM be the set of masked positions, xix_i the original value at position ii, and x^i\hat x_i its prediction. Mean squared error over the hidden region is:

Lmask=1∣M∣∑i∈M(x^i−xi)2L_{\mathrm{mask}}=\frac{1}{|M|}\sum_{i\in M}(\hat x_i-x_i)^2

The summation adds squared prediction errors; ∣M∣|M| counts the masked pixels. We divide by 16, not 64. This borrows the masked-region reconstruction pattern from MAE's objective, but our fixed block mask, binary pixels and constant prediction are not its full architecture or training recipe.

For a hand-picked prediction of 0.5 at every missing pixel, each error is either 0.5−10.5-1 or 0.5−00.5-0. Squaring gives 0.25 in both cases, so their mean is exactly 0.25. This is a local derivation, not a trained decoder's output.

The target contains ten ones and six zeros. Predicting zero everywhere instead costs 10/16=0.62510/16=0.625; predicting one everywhere costs 6/16=0.3756/16=0.375. These different constants reveal the target's pixel balance, not evidence that the model understands an arrow.

What does a perfect score prove?

Check your understanding: why not feed the original patch to the predictor?

It could copy the 16 target values and obtain zero error without inferring anything missing. Keeping the scoring target separate from the input removes that direct shortcut in this toy. Zero error still would not, on its own, establish a useful downstream representation.

The strictly hidden mask here is a teaching simplification. Original BERT does not hide every selected target token: its corruption rule also includes random replacements and unchanged selected tokens. See BERT §3.1 for that distinction.

So far, the target has been missing content. Another route does not ask the model to reconstruct pixels. It asks which two views came from the same source.