Keep the definitions close.

Notation, primary sources, and the exact boundary around our experiment.

A compact glossary

Target

The answer or relationship used to score a prediction. It need not be a human category label.

Self-supervised pretraining

Learning with targets or relationships constructed from the observed data before a downstream use.

Encoder / representation

A function mapping an input to features; the resulting feature vector. Here $h=f_\theta(v)$.

Projection head

A network mapping encoder features $h$ to $z$. SimCLR applies its contrastive objective in this projected space.

Augmentation / view

A transformation of an example, and the resulting input. Which changes preserve useful information is task-dependent.

Anchor, positive, negative

The reference view and its candidates. Here the positive shares a source with the anchor; negatives come from other sources, not necessarily other semantic classes.

Cosine similarity

Alignment between vectors: $(a\cdot z)/(\|a\|\|z\|)$. For unit vectors, it is the dot product.

Softmax / temperature

Exponentiation and normalization of scores into weights that sum to one. Temperature $\tau>0$ divides scores before exponentiation.

Nat

The loss unit here because the objective uses the natural logarithm.

Collapse

The loss of distinctions when inputs receive identical representations. A tie between two candidates alone is not full collapse.

Invariance

A representation remaining unchanged under a transformation.

Linear probe / fine-tuning

A probe freezes the encoder and trains a linear head. Fine-tuning updates pretrained parameters for the downstream task.

Sources behind the explanation

Five primary papers inspected on 4 September 2026. These support the concepts; our invented examples are separate derivations, not paper results or measured learning outcomes.

1. BERT — Devlin et al., NAACL 2019

   Sections 3.1–3.2: selected-token prediction and labeled downstream fine-tuning. Original BERT does not mask every selected target. Claims CL-001, CL-002, CL-010.

2. Masked Autoencoders — He et al., CVPR 2022

   Section 3: masking and reconstruction loss over masked patches; section 4: evaluation. Our fixed binary block-mask example is not a reproduction. Claims CL-001–CL-003, CL-012.

3. SimCLR — Chen et al., ICML 2020

   Section 2.1, Figure 2 and Eq. 1: views, shared encoder, projector and contrastive loss. Sections 2.3 and 3: evaluation and augmentations. Claims CL-001, CL-004–CL-006, CL-010–CL-011.

4. Contrastive Predictive Coding — van den Oord et al., 2018

   Section 2.3, Eq. 4: the InfoNCE candidate-classification formulation. Inspected version 2, 2019. The toy does not establish CPC sampling assumptions or a mutual-information guarantee. Claim CL-005.

5. BYOL — Grill et al., NeurIPS 2020

   Section 3.1: a negative-free objective with online predictor, moving-average target network and stop-gradient target. Not a guarantee that arbitrary positive-only objectives work. Claims CL-008–CL-009, CL-011.

Reproduce the arithmetic

The canonical JavaScript module (local copy) defines the binary arrow, hidden patch, unit vectors, candidate probabilities and loss. Run node model.mjs from the source folder for baseline, comparisons and transfer values. No framework, randomness, downloaded model or learned weights are involved.

Numerical tests (local copy) check the hand cases, invariants, invalid inputs and all 130 angle/temperature configurations. Calculations use full precision; the display rounds to three decimals.

The arrow and hand-positioned vectors are different teaching constructions: the vectors are not computed by encoding the pixels. The masked example borrows a reconstruction-loss pattern; the contrastive example scores one anchor against three candidates. Neither recreates a complete published training pipeline.