↗Loss
Chapter 1 of 5
Notes & sources
Contents ≡
Contents / Chapter 1

01

Errors into a number

A prediction can miss in many ways. A loss function decides how those misses count.

Three points. One imperfect line.

A model predicts 2 when the target is −1. On another example, it predicts 2.5 instead of 1. On a third, it gets 3 exactly right. Before we can improve the model, we need a consistent answer to a deceptively simple question: how bad is this set of predictions?

Our running example uses three invented, noiseless data points: A is (−1,−1)(-1,-1), B is (0,1)(0,1), and C is (1,3)(1,3). In each pair, the first number is the input xx and the second is the target yy. We predict with a line:

y^=wx+b\hat y=wx+b

The hat marks a prediction. The parameter ww controls the line's slope; bb, its intercept. Start with w=0.5w=0.5 and b=2.5b=2.5. These choices produce the three predictions above. They are hand-picked teaching values, not the result of training.

FIG-001The error is a vertical gap.
Targets, predictions and signed errorsThree invented points. At weight0.5,bias2.5, targets are -1,1,3 and predictions2,2.5,3. Errors are3,1.5,0. Filled dots are targets; hollow dots are predictions.-101234-101target / predictioninput xAe=3Be=1.5Ce=0Targets, predictions and signed errorsThree invented points. At weight0.5,bias2.5, targets are -1,1,3 and predictions2,2.5,3. Errors are3,1.5,0. Filled dots are targets; hollow dots are predictions.-101234-101target / predictioninput xAe=3Be=1.5Ce=0

Invented data, exact calculation (CL-005). Filled dots are targets; hollow dots are predictions. The line is ŷ = 0.5x + 2.5. Error means prediction minus target, so these gaps are nonnegative.

Measure the miss, then assign a penalty

Define the residual as prediction minus target:

ei=y^i−yie_i=\hat y_i-y_i

The subscript ii identifies an example. A positive residual means overprediction under this sign convention; a negative one means underprediction. The residuals here are 3, 1.5, and 0. Simply adding signed residuals could let opposite mistakes cancel.

A per-example loss assigns a penalty to a prediction and its target. Squared loss uses ℓi=ei2\ell_i=e_i^2, making either sign of error count positively. Our three penalties are therefore 9, 2.25, and 0.

Combine them with a mean to obtain mean squared error, or MSE. With NN scalar-output examples:

L=1N∑i=1Nei2L=\frac{1}{N}\sum_{i=1}^{N}e_i^2
L=9+2.25+03=3.75L=\frac{9+2.25+0}{3}=3.75

This is the squared-loss mean convention used throughout this guide: no extra factor of one half and no regularization term. The summation symbol says to add the penalties before dividing by the number of examples.

The reduction is part of the definition

A reduction is the rule that combines individual penalties. The sum here is 11.25; the mean is 3.75. Neither number is wrong, but they are not interchangeable. Duplicate every unchanged example and the sum doubles, while the mean stays 3.75.

We will use ℓ\ell for an individual penalty and LL for the combined objective. In practice, “loss” often names either, so check the definition before comparing reported values. You now know where one loss number comes from. Next: why square the errors at all?