↗Loss
Chapter 2 of 5
Notes & sources
Contents ≡
Contents / Chapter 2

02

Different penalties

The same residual can receive very different penalties. That choice changes what improvement means.

How fast should a mistake grow?

An error of magnitude 3 receives squared loss 9. An error of magnitude 1 receives squared loss 1. Tripling the miss makes its penalty nine times as large. Squared loss makes large residuals contribute disproportionately to the total.

A different choice is absolute loss, ℓ(e)=∣e∣\ell(e)=|e|. The vertical bars mean magnitude, ignoring sign. Errors of magnitude 3 and 1 receive penalties 3 and 1. Averaging absolute penalties gives mean absolute error, or MAE. See the absolute-loss definition and reductions.

Both losses treat +3 and −3 equally. Their difference is how the penalty grows with magnitude. If every residual doubles, MAE doubles but MSE quadruples. That is a property of the formulas, not evidence that one model suddenly became four times less useful.

A quadratic center, linear tails

Huber loss joins two behaviors: a quadratic penalty close to zero and a linear penalty farther away. Its threshold δ\delta marks the change. Set δ=1\delta=1 for this comparison:

ℓHuber(e)={12e2,∣e∣≤1∣e∣−12,∣e∣>1\ell_{\text{Huber}}(e)=\begin{cases}\tfrac12 e^2,&|e|\leq1\\|e|-\tfrac12,&|e|>1\end{cases}

The one-half factor is intentional; our MSE does not have it. At either join, both branches give 0.5, and their slopes agree. Beyond the threshold, the penalty grows linearly instead of quadratically, making Huber less sensitive to large residuals than squared loss. This is the documented Huber definition at a specific threshold.

FIG-002A mistake can be penalized in different ways.
Squared, absolute and Huber penaltiesAll use the same signed error axis. At error plus or minus3, squared penalty is9, absolute3, Huber delta1 is2.5. Squared is solid, absolute dashed, Huber dotted. Huber uses half squared error inside plus or minus1.0369-303per-example penaltysigned prediction error eSquared, absolute and Huber penaltiesAll use the same signed error axis. At error plus or minus3, squared penalty is9, absolute3, Huber delta1 is2.5. Squared is solid, absolute dashed, Huber dotted. Huber uses half squared error inside plus or minus1.0369-303per-example penaltysigned prediction error e

Solid: squared (at e=3: 9). Dashed: absolute (3). Dotted: Huber, δ=1 (2.5). These are analytic penalty shapes, not measured model performance. Different scales are part of the definitions; a lower line is not a better model.

Absolute loss has a sharp corner at zero: its ordinary derivative is undefined there. Huber has a smooth join in its first derivative. A corner does not mean a loss cannot be optimized; it means the optimizer must handle that boundary appropriately.

Smaller numbers are not a quality ranking

At residual 3, the plotted penalties are 9, 3, and 2.5. Huber's lower number does not make the unchanged prediction better. We have changed the measuring rule.

Before comparing loss values, match the definition, reduction, target scale, and evaluated examples. If a target carries physical units, absolute loss has those units and squared loss has their square. Choosing a penalty means choosing which error patterns to emphasize; no curve here establishes a universally best choice.

The horizontal axis so far has been the residual. We have asked what penalty an error earns. A model has a different question: which parameter choices produce smaller errors? That change of axes creates a loss landscape.