∂ Backprop
Chapter 1 of 5
Notes & sources
Contents ≡
Contents / Chapter 1

01

One mistake, five parameters

A forward pass turns inputs into a loss. Keep its intermediate values: the backward pass will need them.

Who should change?

A small network predicts 1.5 when the target is 2. The prediction is too low. That tells us the direction of the output error, but it does not yet tell us how to change an earlier weight. That weight’s effect passes through other operations—and may reach the output by more than one route.

Backpropagation computes how sensitive one scalar loss is to each parameter. An optimizer uses those sensitivities to propose new parameters. We will do both operations by hand before letting the lab repeat them.

You need arithmetic and the idea that a model has adjustable numbers. We introduce derivatives and vector shapes when they become useful. If you already know losses, treat the half-squared error below as our fixed measuring rule.

A network you can hold in your head

Imagine a normalized electricity-demand feature xx and a target yy. Our numbers are invented and dimensionless. There is no timestamp, real grid data, fitted forecast or physical unit behind them. The demand story motivates inspecting a learning mechanism.

The five adjustable parameters are w,b,v,u,cw,b,v,u,c. Every variable here is a scalar: one number. First form an affine value zz (multiply, then add); apply a rectified linear unit, or ReLU, to get the hidden activation hh:

z=wx+b,h=max⁡(0,z)z=wx+b,\qquad h=\max(0,z)

ReLU passes a positive input unchanged and outputs zero for a negative input. The same hh is used in two branches, aa and dd, which rejoin at prediction y^\hat y:

a=vh,d=uha=vh,\qquad d=uh
y^=a+d+c\hat y=a+d+c

This deliberately redundant readout could be collapsed to (v+u)h+c(v+u)h+c. Keeping its branches separate makes one shared value’s two uses visible. These are not two different hidden neurons.

BP-F01One shared activation forks, then rejoins.
One hidden value, two forward pathsInput two yields affine value one and hidden value one. The shared h splits into branches one and 0.5, rejoins at prediction 1.5, then gives loss 0.125.x = 2 → z = 1z = w×x + bh = 1ReLU(z)a = 1v×h = 1×1d = 0.5u×h = 0.5×1prediction = 1.5a + d + closs = 0.125½ × (1.5 − 2)²One hidden value, two forward pathsInput two yields affine value one and hidden value one. The shared h splits into branches one and 0.5, rejoins at prediction 1.5, then gives loss 0.125.x = 2 → z = 1z = w×x + bh = 1ReLU(z)a = 1v×h = 1×1d = 0.5u×h = 0.5×1prediction = 1.5a + d + closs = 0.125½ × (1.5 − 2)²

Forward arrows show dependency. Both branches use the same h. Invented values; BP-C09–10.

Start with x=2, y=2x=2,\ y=2 and w=0.5, b=0, v=1, u=0.5, c=0w=0.5,\ b=0,\ v=1,\ u=0.5,\ c=0. Forward values are z=h=1, a=1, d=0.5, y^=1.5z=h=1,\ a=1,\ d=0.5,\ \hat y=1.5. Arrows record dependencies. PyTorch’s autograd mechanics describes how recorded operations form the graph used for differentiation. BP-C01, BP-C09–10

End at one number

Define residual e=y^−ye=\hat y-y and loss LL:

L=12e2L=\frac12 e^2

Here e=−0.5e=-0.5, so L=0.125L=0.125. The factor one-half is a convenience: differentiating cancels the square’s factor two. This is a loss for one example, not a batch mean.

Record intermediate values rather than only the final answer. To differentiate a=vha=vh, we will need the values of both vv and hh from this forward pass. Backpropagation travels backward through dependencies; it does not run the prediction formula in reverse to recover the input.