Three points. One imperfect line.

A model predicts 2 when the target is −1. On another example, it predicts 2.5 instead of 1. On a third, it gets 3 exactly right. Before we can improve the model, we need a consistent answer to a deceptively simple question: how bad is this set of predictions?

Our running example uses three invented, noiseless data points: A is $(-1,-1)$, B is $(0,1)$, and C is $(1,3)$. In each pair, the first number is the input $x$ and the second is the target $y$. We predict with a line:

The hat marks a prediction. The parameter $w$ controls the line's slope; $b$, its intercept. Start with $w=0.5$ and $b=2.5$. These choices produce the three predictions above. They are hand-picked teaching values, not the result of training.

Measure the miss, then assign a penalty

Define the residual as prediction minus target:

The subscript $i$ identifies an example. A positive residual means overprediction under this sign convention; a negative one means underprediction. The residuals here are 3, 1.5, and 0. Simply adding signed residuals could let opposite mistakes cancel.

A per-example loss assigns a penalty to a prediction and its target. Squared loss uses $\ell_i=e_i^2$, making either sign of error count positively. Our three penalties are therefore 9, 2.25, and 0.

Combine them with a mean to obtain mean squared error, or MSE. With $N$ scalar-output examples:

This is the squared-loss mean convention used throughout this guide: no extra factor of one half and no regularization term. The summation symbol says to add the penalties before dividing by the number of examples.

The reduction is part of the definition

A reduction is the rule that combines individual penalties. The sum here is 11.25; the mean is 3.75. Neither number is wrong, but they are not interchangeable. Duplicate every unchanged example and the sum doubles, while the mean stays 3.75.

We will use $\ell$ for an individual penalty and $L$ for the combined objective. In practice, “loss” often names either, so check the definition before comparing reported values. You now know where one loss number comes from. Next: why square the errors at all?

How fast should a mistake grow?

An error of magnitude 3 receives squared loss 9. An error of magnitude 1 receives squared loss 1. Tripling the miss makes its penalty nine times as large. Squared loss makes large residuals contribute disproportionately to the total.

A different choice is absolute loss, $\ell(e)=|e|$. The vertical bars mean magnitude, ignoring sign. Errors of magnitude 3 and 1 receive penalties 3 and 1. Averaging absolute penalties gives mean absolute error, or MAE. See the absolute-loss definition and reductions.

Both losses treat +3 and −3 equally. Their difference is how the penalty grows with magnitude. If every residual doubles, MAE doubles but MSE quadruples. That is a property of the formulas, not evidence that one model suddenly became four times less useful.

A quadratic center, linear tails

Huber loss joins two behaviors: a quadratic penalty close to zero and a linear penalty farther away. Its threshold $\delta$ marks the change. Set $\delta=1$ for this comparison:

The one-half factor is intentional; our MSE does not have it. At either join, both branches give 0.5, and their slopes agree. Beyond the threshold, the penalty grows linearly instead of quadratically, making Huber less sensitive to large residuals than squared loss. This is the documented Huber definition at a specific threshold.

Absolute loss has a sharp corner at zero: its ordinary derivative is undefined there. Huber has a smooth join in its first derivative. A corner does not mean a loss cannot be optimized; it means the optimizer must handle that boundary appropriately.

Smaller numbers are not a quality ranking

At residual 3, the plotted penalties are 9, 3, and 2.5. Huber's lower number does not make the unchanged prediction better. We have changed the measuring rule.

Before comparing loss values, match the definition, reduction, target scale, and evaluated examples. If a target carries physical units, absolute loss has those units and squared loss has their square. Choosing a penalty means choosing which error patterns to emphasize; no curve here establishes a universally best choice.

The horizontal axis so far has been the residual. We have asked what penalty an error earns. A model has a different question: which parameter choices produce smaller errors? That change of axes creates a loss landscape.

Change the axes, keep the data

Return to our three fixed points and squared loss. Every pair $(w,b)$ defines a different line, three predictions, and one mean loss. Place $w$ on one horizontal axis and $b$ on the other. Put the resulting $L(w,b)$ above that location. The collection of heights is a loss surface.

Expanding the three residuals and averaging their squares gives this exact expression for our invented example:

The lowest point is $(w,b)=(2,1)$, where both squared terms vanish. This particular line passes through all three targets. The surface is an analytic calculation—not a measured neural-network landscape or a history of training.

Move the model, not just the camera

The 3D view makes both parameter directions visible. The linked 2D plot cuts through the same surface at the current $b$, showing loss as $w$ varies. Use that flat slice and the numerical table for precise comparisons; depth and viewing angle can make heights harder to judge.

Changing a parameter recomputes the predictions and losses. Changing the view angle only rotates the projection: it cannot improve the model. The mesh samples the exact formula above, with a fixed height scale. These controls do not train a network.

A slice has its own floor

At the defaults, $w=0.5$ and $b=2.5$, loss is 3.75. Change only $w$ to 2. Loss falls to 2.25, but stops short of zero: the remaining term is $(2.5-1)^2$. Then change $b$ to 1 and both terms disappear.

More generally, a fixed bias leaves a floor of $(b-1)^2$, reached at $w=2$. Try a changed case: if you fix $b=0$, what is the lowest loss achievable by varying weight?

Our bowl has one minimum. Do not generalize its shape to neural networks: their objectives can be nonconvex, as discussed in Deep Learning, §8.2.2. Even on this simple surface, finding a direction and choosing how far to move are separate decisions.

Read the local slope

A derivative describes how quickly a quantity changes near the current input. With two parameters, a partial derivative varies one while holding the other fixed. The gradient collects these partial derivatives into a vector.

Differentiate our MSE expression with respect to $w$ and $b$:

At the starting point $(0.5,2.5)$, the gradient is $(-2,3)$. A small increase in $w$ lowers loss locally; a small increase in $b$ raises it. The negative gradient therefore points toward increasing $w$ and decreasing $b$.

Direction is not distance

Gradient descent proposes a step opposite the gradient. Write all parameters together as $\theta=(w,b)$ and choose a positive learning rate $\eta$:

The gradient describes local slope, not the entire path ahead. A finite step can go too far and raise loss. See the update and curvature discussion in Deep Learning, §4.3.

Try two independent single steps from the same starting point. With $\eta=0.25$:

The new loss is $59/48\approx1.229$, down from 3.75. But $\eta=1.2$ produces $(2.9,-1.1)$, with loss 4.95—higher than the starting value.

The horizontal axis is the candidate step size. Every plotted point starts from the same parameters and takes one step; these are not successive updates or epochs. The second bias, −1.1, is outside the interactive surface's viewing window. The mathematical calculation is not clamped to that window.

Reproduce one step

The canonical JavaScript module computes losses directly from the three data rows. With Node.js and the local companion module in the same folder, this focused excerpt reproduces the smaller step:

import { gradientStep } from './model.mjs';

const next = gradientStep({
  weight: 0.5, bias: 2.5, rate: 0.25
});
console.log(next.loss.toFixed(3)); // 1.229

The displayed value is deterministic toy arithmetic, rounded to three decimals. There is no network, random initialization, or training dataset download. The rate is a demonstration choice, not a recommendation for another model.

A training-loss curve would instead place update number or epoch on its horizontal axis and record the losses encountered during optimization. Always read the axes before interpreting a “loss curve.” And even a well-behaved downward curve leaves another question: is its loss measuring the task we care about?

A different kind of prediction

Leave the line-fitting example. In binary classification, the target $y$ is either 0 or 1. Suppose a model predicts a probability $p$ for class 1. This is not an unrestricted real-valued prediction or a raw logit: $p$ lies between 0 and 1.

Binary cross-entropy, also called binary log loss, evaluates the probability assigned to the observed label. With natural logarithm $\ln$, for $0\lt p\lt1$:

If $y=1$, this becomes $-\ln p$. If $y=0$, it becomes $-\ln(1-p)$. In either case, assigning more probability to the observed label lowers the penalty. The binary formula and natural-log convention make the meaning precise.

For a positive target, probabilities 0.1, 0.5, and 0.9 produce losses about 2.303, 0.693, and 0.105. The theoretical loss approaches infinity as $p$ approaches zero, and is zero at $p=1$. The finite plot begins at 0.01; it does not turn that infinite boundary into a finite penalty.

Same loss, different decisions

Try a new, invented case. The true labels are $[1,0]$. Model A predicts positive-class probabilities $[0.9,0.6]$; model B predicts $[0.6,0.4]$. Predict class 1 whenever $p\geq0.5$, otherwise class 0.

Which model has lower mean binary cross-entropy? Which has higher accuracy—the fraction of labels predicted correctly? Work out both before revealing the answer.

The objective is not the whole task

A low training loss describes fit to training examples; it does not by itself establish performance on unseen data. Validation data can inform model choices, while a separate test set supports evaluation without tuning those choices to it. The cross-validation guide explains why reusing training or test data for the wrong role can mislead.

Our line achieved zero loss on three invented points. We supplied no held-out examples, so that result says nothing about generalization. Even a real holdout gives evidence under its sampling conditions, not a guarantee for every deployment setting.

Keep three questions separate: what does each error cost, how are those costs combined, and what evidence shows the resulting model serves the task? A loss function answers the first two. Good evaluation is still needed for the third.