A landscape of choices

Two parameters choose the predictions. Their mean loss becomes the height of a surface.

Change the axes, keep the data

Return to our three fixed points and squared loss. Every pair $(w,b)$ defines a different line, three predictions, and one mean loss. Place $w$ on one horizontal axis and $b$ on the other. Put the resulting $L(w,b)$ above that location. The collection of heights is a loss surface.

Expanding the three residuals and averaging their squares gives this exact expression for our invented example:

L(w,b)=\frac{2}{3}(w-2)^2+(b-1)^2

The lowest point is $(w,b)=(2,1)$ , where both squared terms vanish. This particular line passes through all three targets. The surface is an analytic calculation—not a measured neural-network landscape or a history of training.

Move the model, not just the camera

The 3D view makes both parameter directions visible. The linked 2D plot cuts through the same surface at the current $b$ , showing loss as $w$ varies. Use that flat slice and the numerical table for precise comparisons; depth and viewing angle can make heights harder to judge.

Changing a parameter recomputes the predictions and losses. Changing the view angle only rotates the projection: it cannot improve the model. The mesh samples the exact formula above, with a fixed height scale. These controls do not train a network.

EXP-001 · EXPLORE THE LANDSCAPE

Two parameters, one height.

Three invented points. Exact mean squared error. A sampled 3D projection—not a measured neural-network landscape.

Make a prediction first (optional)

If bias is fixed at 2, can weight alone make the loss zero? Yes No

Weight w 0.5Bias b 2.5View angle 45°

FIG-003 · Height is loss; the two ground axes are model parameters.

Mean squared error 3.750

THE SAME LANDSCAPE, SLICED FLAT

Hold bias fixed. Vary only weight.

Weight 0.5, bias 2.5: mean loss 3.750. With this bias held fixed, the lowest possible loss is 2.250, at weight 2. View angle 45° changes only the projection.

Three invented examples · values rounded to three decimals
Point / x	Target y	Prediction	Error e	Squared e²
A / -1	-1	2.000	3.000	9.000
B / 0	1	2.500	1.500	2.250
C / 1	3	3.000	0.000	0.000
Mean of the three squared errors				3.750

The mesh samples a smooth mathematical surface. Perspective can obscure heights; the flat slice and numbers provide the exact reading. Heights use one fixed scale, 0 to 6.667, throughout this control window. No optimizer runs when you move a slider.

Compare three fixed states

Invented-data arithmetic; the camera cannot alter these values.
Weight	Bias	Mean loss	What changed?
0.5	2.5	3.750	Starting model
2	2.5	2.250	Best weight at this bias
2	1	0.000	Both parameters fit the toy

The first move removes the weight error. The remaining bias error still costs 2.25. Only the final state fits all three points.

A slice has its own floor

At the defaults, $w=0.5$ and $b=2.5$ , loss is 3.75. Change only $w$ to 2. Loss falls to 2.25, but stops short of zero: the remaining term is $(2.5-1)^2$ . Then change $b$ to 1 and both terms disappear.

More generally, a fixed bias leaves a floor of $(b-1)^2$ , reached at $w=2$ . Try a changed case: if you fix $b=0$ , what is the lowest loss achievable by varying weight?

Check the changed-bias case

The best weight is still $w=2$ , but the remaining bias term is $(0-1)^2=1$ . The surface's global minimum and a slice's minimum are different ideas.

Where did the simple landscape formula come from?

Let $a=w-2$ and $d=b-1$ . The three residuals become $-a+d$ , $d$ , and $a+d$ . Their cross terms cancel:

\begin{aligned}L&=\frac{(-a+d)^2+d^2+(a+d)^2}{3}\\&=\frac{2a^2+3d^2}{3}\end{aligned}

This clean separation follows from these particular inputs and targets. It is not a formula for every regression dataset.

Our bowl has one minimum. Do not generalize its shape to neural networks: their objectives can be nonconvex, as discussed in Deep Learning, §8.2.2. Even on this simple surface, finding a direction and choosing how far to move are separate decisions.