↗Loss
Chapter 4 of 5
Notes & sources
Contents ≡
Contents / Chapter 4

04

Following the slope

A gradient describes local change. A learning rate decides how far to trust that direction.

Read the local slope

A derivative describes how quickly a quantity changes near the current input. With two parameters, a partial derivative varies one while holding the other fixed. The gradient collects these partial derivatives into a vector.

Differentiate our MSE expression with respect to ww and bb:

∂L∂w=43(w−2)∂L∂b=2(b−1)\begin{aligned}\frac{\partial L}{\partial w}&=\frac{4}{3}(w-2)\\\frac{\partial L}{\partial b}&=2(b-1)\end{aligned}

At the starting point (0.5,2.5)(0.5,2.5), the gradient is (−2,3)(-2,3). A small increase in ww lowers loss locally; a small increase in bb raises it. The negative gradient therefore points toward increasing ww and decreasing bb.

Direction is not distance

Gradient descent proposes a step opposite the gradient. Write all parameters together as θ=(w,b)\theta=(w,b) and choose a positive learning rate η\eta:

θnext=θ−η∇L(θ)\theta_{\text{next}}=\theta-\eta\nabla L(\theta)

The gradient describes local slope, not the entire path ahead. A finite step can go too far and raise loss. See the update and curvature discussion in Deep Learning, §4.3.

Try two independent single steps from the same starting point. With η=0.25\eta=0.25:

wnext=0.5−0.25(−2)=1bnext=2.5−0.25(3)=1.75\begin{aligned}w_{\text{next}}&=0.5-0.25(-2)=1\\b_{\text{next}}&=2.5-0.25(3)=1.75\end{aligned}

The new loss is 59/48≈1.22959/48\approx1.229, down from 3.75. But η=1.2\eta=1.2 produces (2.9,−1.1)(2.9,-1.1), with loss 4.95—higher than the starting value.

FIG-004A bigger step is not always a better step.
Loss after one negative-gradient step for different ratesEvery candidate starts at weight0.5,bias2.5,loss3.75. Rate0.25 gives1.2291667; rate1.2 gives4.95, higher than the initial loss. The horizontal axis is a step-size choice, not epoch or training time.024600.250.751.25loss after that one stepone-step learning rate ηstart: 3.750η .25: 1.229η 1.2: 4.950Loss after one negative-gradient step for different ratesEvery candidate starts at weight0.5,bias2.5,loss3.75. Rate0.25 gives1.2291667; rate1.2 gives4.95, higher than the initial loss. The horizontal axis is a step-size choice, not epoch or training time.024600.250.751.25loss after that one stepone-step learning rate ηstart: 3.750η .25: 1.229η 1.2: 4.950

Derived from the same toy, always starting at w=0.5, b=2.5 (CL-009). Each point is a separate one-step experiment, not the next step in a training history. The larger-rate example is evaluated without clipping parameters to the lab window.

The horizontal axis is the candidate step size. Every plotted point starts from the same parameters and takes one step; these are not successive updates or epochs. The second bias, −1.1, is outside the interactive surface's viewing window. The mathematical calculation is not clamped to that window.

Reproduce one step

The canonical JavaScript module computes losses directly from the three data rows. With Node.js and the local companion module in the same folder, this focused excerpt reproduces the smaller step:

import { gradientStep } from './model.mjs';

const next = gradientStep({
  weight: 0.5, bias: 2.5, rate: 0.25
});
console.log(next.loss.toFixed(3)); // 1.229

The displayed value is deterministic toy arithmetic, rounded to three decimals. There is no network, random initialization, or training dataset download. The rate is a demonstration choice, not a recommendation for another model.

A training-loss curve would instead place update number or epoch on its horizontal axis and record the losses encountered during optimization. Always read the axes before interpreting a “loss curve.” And even a well-behaved downward curve leaves another question: is its loss measuring the task we care about?