One image, two answer keys

Imagine handing a model the small arrow below. In a supervised direction-classification task, a person could supply the label “up.” The image is the input; “up” is the target, the answer against which a prediction is judged. Now cover part of the same image. The original pixels behind the cover provide another answer key, without anyone naming the arrow.

That change in the source of the answer is the central idea of self-supervised learning. In the approaches we will study, a training task constructs its targets or relationships from the data itself. BERT predicts selected text tokens; masked autoencoders reconstruct missing image content. We will start with a tiny image example so that the target is something you can inspect.

“Self” does not mean “no design”

A prediction is the model's proposed answer. A loss is a number that measures how well that answer satisfies the task. During training, an optimizer uses the loss to update the model's parameters. Self-supervision changes how the task gets its answer key; it does not remove the need to specify what counts as a good answer.

People still choose the data, transformations, model and objective. Text may be human-authored, and an image collection may be curated. “No category labels needed for this pretraining task” is therefore much more precise than “no human supervision anywhere.” A later application may use labeled examples to train or evaluate a classifier. The SimCLR evaluation protocol, for example, separates representation pretraining from labeled evaluation.

Learn something we can reuse

An encoder maps an input into a vector of features, called a representation. Think of the vector as an intermediate description that another task can use. Pretraining gives the encoder a task before that later, or downstream, use. A representation is useful only relative to what the downstream task needs to recover from it.

We will compare two ways to construct a pretraining task: hide content and predict it, or construct two related views and compare their representations. These are two important families, not an exhaustive definition of every self-supervised method.

Our arrow, pixel predictions and later vectors are invented teaching values. This guide computes losses; it does not train an encoder or measure feature quality. With basic arithmetic, you can follow where a target comes from, how one loss is calculated, and when a proposed transformation conflicts with a later task. First, let's make the hidden-pixel answer key explicit.

Keep the answer on a separate path

Our invented image has $8\times8=64$ binary pixels: 1 for an arrow pixel and 0 for the background. We hide the top-right $4\times4$ region. The predictor would receive the remaining context and the positions of the missing pixels. A separate target path keeps the 16 original values for scoring.

The distinction is about roles, not separate datasets. We can inspect the complete original to construct the exercise. The hypothetical predictor must not receive those hidden values as its input answer key. Otherwise, a low reconstruction error could come from copying the supplied answer rather than using the visible context.

Score the sixteen hidden pixels

Let $M$ be the set of masked positions, $x_i$ the original value at position $i$, and $\hat x_i$ its prediction. Mean squared error over the hidden region is:

The summation adds squared prediction errors; $|M|$ counts the masked pixels. We divide by 16, not 64. This borrows the masked-region reconstruction pattern from MAE's objective, but our fixed block mask, binary pixels and constant prediction are not its full architecture or training recipe.

For a hand-picked prediction of 0.5 at every missing pixel, each error is either $0.5-1$ or $0.5-0$. Squaring gives 0.25 in both cases, so their mean is exactly 0.25. This is a local derivation, not a trained decoder's output.

The target contains ten ones and six zeros. Predicting zero everywhere instead costs $10/16=0.625$; predicting one everywhere costs $6/16=0.375$. These different constants reveal the target's pixel balance, not evidence that the model understands an arrow.

What does a perfect score prove?

The strictly hidden mask here is a teaching simplification. Original BERT does not hide every selected target token: its corruption rule also includes random replacements and unchanged selected tokens. See BERT §3.1 for that distinction.

So far, the target has been missing content. Another route does not ask the model to reconstruct pixels. It asks which two views came from the same source.

Change the view, keep the source

Make two intensity variants of our arrow. Their pixels differ, but both were constructed from the same source image. We call this known relationship a positive pair. The pairing requires a record of common origin, not the semantic label “up.” Our intensity-only change is an illustrative augmentation, not SimCLR's complete transformation recipe.

In SimCLR's architecture, the two views pass through the same encoder. “Same” means shared parameters: changing an encoder weight changes the function used on both branches. A small network called a projection head then maps the encoder's features into the space used by the contrastive objective.

Here $v$ is a view, $f_\theta$ the encoder with parameters $\theta$, and $g_\phi$ the projector with parameters $\phi$. The encoder representation $h$ and projected representation $z$ are different objects. SimCLR applies its contrastive loss to $z$ and retains the encoder representation $h$ for downstream use.

Positive is a construction rule

Take one view as an anchor. Its paired view is the positive candidate. In the contrastive setup we will illustrate, views from other source images supply negative candidates. The objective compares the positive's similarity to the anchor with the competing similarities; matching the positive is not judged in isolation.

A negative here does not mean “known to belong to a different human class.” Two different images could both show upward arrows. Treating them as negatives can create a semantic false negative: the construction rule separates sources that a later task might want to group. Source identity and semantic category are not interchangeable.

Matching is not enough

Suppose every input receives exactly the same representation. Positive pairs match perfectly, but so does everything else. This is representation collapse: that representation preserves no distinction between inputs. The next chapter's three-candidate loss makes the problem concrete—equal scores give the positive only one third of the probability, not a perfect score.

Let's now isolate candidate competition. We will prescribe four tiny vectors directly, leaving the encoder out of the calculation so that every score is visible.

Read the geometry

The following vectors are hand-positioned toy representations, not encodings of our arrow. Fix the anchor at $a=(1,0)$. Place the positive at angle $\alpha$ on a unit circle, negative A at $(0,1)$, and negative B at $(-1,0)$. Every vector has length one.

Cosine similarity measures alignment. For unit vectors it is simply their dot product, so the anchor reads each candidate's horizontal coordinate. The positive score is $s_+=\cos\alpha$; the two negative scores are $s_A=0$ and $s_B=-1$. Moving the positive toward the anchor increases its score. This uses SimCLR's cosine-similarity convention.

Turn three scores into one loss

Divide each score by a positive temperature $\tau$, exponentiate, then normalize the three weights so they sum to one. This normalization is called softmax. Let $Z$ be the sum of those weights:

The positive is included in the denominator; the anchor is not. Our single-anchor InfoNCE-style loss is the negative natural logarithm of the positive's probability, measured in nats. A larger $p_+$ means a smaller loss. This is a teaching subset, not the complete SimCLR minibatch objective. See CPC's candidate-classification formulation and SimCLR Eq. 1.

At $\alpha=60^\circ$ and $\tau=0.5$, the scores are $(0.5,0,-1)$. Their scaled values are $(1,0,-2)$. The resulting positive probability is about 0.705385 and the loss is 0.349012 nats. These are deterministic calculations from the stated toy values.

Try a sharper comparison

Predict first: at $\alpha=120^\circ$, would reducing temperature from 1 to 0.1 lower the positive's loss? Set that angle, change only temperature, and watch the probability bars and loss. The vectors and their raw similarities should stay still.

Lower temperature sharpens score differences, favoring the highest-scoring candidate. At 120°, negative A scores 0 while the positive scores −0.5. In this specific comparison, sharpening raises the positive's loss from about 1.180270 to 5.006760 nats. Lower temperature does not universally lower positive loss; the effect depends on the scores and temperature.

These probabilities are relative weights among these three candidates—not calibrated confidence that an image belongs to a semantic category. Nor does moving a slider train a model: angle directly changes the prescribed geometry, while temperature changes only the score-to-probability calculation.

What if every score ties?

This arithmetic explains what the objective scores, not a guarantee about where training will converge. Even a low loss leaves another question unanswered: what information will the representation retain for a later task?

A transformation makes a promise

Constructing two views as a positive pair asks the objective to treat them as related despite their differences. The transformation therefore expresses a choice about which changes should matter less. Invariance means a representation remains unchanged under a transformation. Whether that is useful depends on the later task—not just whether the transformed image looks plausible. SimCLR's augmentation discussion makes these task-defining choices explicit.

Suppose our downstream task is to distinguish upward from downward arrows. A brightness change keeps the direction. A 180° rotation reverses it. These transformations cannot carry the same promise for that task.

Follow the information

This is an exact logical counterexample, not a measured failure rate or a universal ban on rotations. It assumes identical frozen encoder representations. Approximate similarity needs empirical analysis. And remember the two spaces in SimCLR: an invariance in projected $z$ does not automatically prove that encoder features $h$ have lost the same information.

Test the representation you will use

One evaluation route is a linear probe: freeze the encoder, compute $h$, and train a linear classifier on labeled examples. Only the classifier's parameters change. This tests how accessible the task information is to that linear head, not everything any possible decoder could recover. It follows the linear evaluation protocol described in SimCLR.

A different route is fine-tuning: update some or all pretrained encoder parameters for the downstream task, usually along with a task-specific head. BERT's downstream procedure illustrates this distinction. Fine-tuning can change the representation; a frozen-encoder information argument does not describe those updated features.

For either route, specify the target, training data and held-out evaluation data. Check the representation actually used downstream rather than treating low pretraining loss as a substitute for that evaluation. No probe, fine-tuning run or downstream benchmark has been performed by this guide.

The core idea is now compact: construct a learning signal from the data, optimize for that signal, then test what transfers. When you meet a new self-supervised method, ask three questions: Where does its target come from? What distinctions does its objective preserve or suppress? How will the downstream task check that choice?