05
What the representation keeps
A useful invariance for one task can erase the answer to another.
A transformation makes a promise
Constructing two views as a positive pair asks the objective to treat them as related despite their differences. The transformation therefore expresses a choice about which changes should matter less. Invariance means a representation remains unchanged under a transformation. Whether that is useful depends on the later task—not just whether the transformed image looks plausible. SimCLR's augmentation discussion makes these task-defining choices explicit.
Suppose our downstream task is to distinguish upward from downward arrows. A brightness change keeps the direction. A 180° rotation reverses it. These transformations cannot carry the same promise for that task.
Invented direction task. Brightness preserves its target; a half-turn changes it. Rotation can be sensible for other tasks. The question is which distinctions the downstream representation must keep.
Follow the information
Transfer: should up and down be forced into the same frozen representation?
No, if a later head must distinguish their directions using only that representation. Suppose the frozen encoder produces exactly the same vector for an upward arrow and its downward rotation. A deterministic head receives identical inputs and must return identical outputs. Since the two targets differ, it cannot classify both correctly. Brightness invariance does not create this particular conflict because the direction target stays the same.
This is an exact logical counterexample, not a measured failure rate or a universal ban on rotations. It assumes identical frozen encoder representations. Approximate similarity needs empirical analysis. And remember the two spaces in SimCLR: an invariance in projected does not automatically prove that encoder features have lost the same information.
Test the representation you will use
One evaluation route is a linear probe: freeze the encoder, compute , and train a linear classifier on labeled examples. Only the classifier's parameters change. This tests how accessible the task information is to that linear head, not everything any possible decoder could recover. It follows the linear evaluation protocol described in SimCLR.
A different route is fine-tuning: update some or all pretrained encoder parameters for the downstream task, usually along with a task-specific head. BERT's downstream procedure illustrates this distinction. Fine-tuning can change the representation; a frozen-encoder information argument does not describe those updated features.
For either route, specify the target, training data and held-out evaluation data. Check the representation actually used downstream rather than treating low pretraining loss as a substitute for that evaluation. No probe, fine-tuning run or downstream benchmark has been performed by this guide.
The core idea is now compact: construct a learning signal from the data, optimize for that signal, then test what transfers. When you meet a new self-supervised method, ask three questions: Where does its target come from? What distinctions does its objective preserve or suppress? How will the downstream task check that choice?