↗SSL
Chapter 1 of 5
Notes & sources
Contents ≡
Contents / Chapter 1

01

Where targets come from

A learning signal does not have to arrive as a human-written category.

One image, two answer keys

Imagine handing a model the small arrow below. In a supervised direction-classification task, a person could supply the label “up.” The image is the input; “up” is the target, the answer against which a prediction is judged. Now cover part of the same image. The original pixels behind the cover provide another answer key, without anyone naming the arrow.

FIG-001The target has a source.
Two sources of supervisionA human can supply an up label. Or the original image can supply a hidden patch as a reconstruction target. These are different sources of targets.Same raw dataHuman supplies a category“up”Data supplies held-out pixels16 target valuesTwo sources of supervisionThe same invented arrow supports either an external human category label or a data-derived pixel target.Same raw dataHuman label“up”Held-out pixelsDifferent ways to define a target.

The pixels and the illustrative “up” label are invented. The right-hand target is extracted from the original, then withheld from the predictor’s input.

That change in the source of the answer is the central idea of self-supervised learning. In the approaches we will study, a training task constructs its targets or relationships from the data itself. BERT predicts selected text tokens; masked autoencoders reconstruct missing image content. We will start with a tiny image example so that the target is something you can inspect.

“Self” does not mean “no design”

A prediction is the model's proposed answer. A loss is a number that measures how well that answer satisfies the task. During training, an optimizer uses the loss to update the model's parameters. Self-supervision changes how the task gets its answer key; it does not remove the need to specify what counts as a good answer.

People still choose the data, transformations, model and objective. Text may be human-authored, and an image collection may be curated. “No category labels needed for this pretraining task” is therefore much more precise than “no human supervision anywhere.” A later application may use labeled examples to train or evaluate a classifier. The SimCLR evaluation protocol, for example, separates representation pretraining from labeled evaluation.

Learn something we can reuse

An encoder maps an input into a vector of features, called a representation. Think of the vector as an intermediate description that another task can use. Pretraining gives the encoder a task before that later, or downstream, use. A representation is useful only relative to what the downstream task needs to recover from it.

We will compare two ways to construct a pretraining task: hide content and predict it, or construct two related views and compare their representations. These are two important families, not an exhaustive definition of every self-supervised method.

Our arrow, pixel predictions and later vectors are invented teaching values. This guide computes losses; it does not train an encoder or measure feature quality. With basic arithmetic, you can follow where a target comes from, how one loss is calculated, and when a proposed transformation conflicts with a later task. First, let's make the hidden-pixel answer key explicit.