∂ Backprop
Chapter 3 of 5
Notes & sources
Contents ≡
Contents / Chapter 3

03

When paths meet, add

Multiply along a path. Add across paths. Reusing a value creates another contribution to its sensitivity.

One value, two uses

Through d=uhd=uh, the contribution to the hidden value is dˉ u=−0.5(0.5)=−0.25\bar d\,u=-0.5(0.5)=-0.25. Changing hh affects both branches at once:

hˉ=aˉ v+dˉ u\bar h=\bar a\,v+\bar d\,u
hˉ=−0.5−0.25=−0.75\bar h=-0.5-0.25=-0.75
BP-F03Add where the branches meet.
Two paths add before moving upstreamThe same loss sensitivity contributes minus 0.5 via a and minus 0.25 via d. Sum minus 0.75 at h. Multiply by ReLU slope one and input two to obtain w gradient minus 1.5.Loss sensitivity at prediction: −0.5via a: −0.5−0.5 × vvia d: −0.25−0.5 × uh sensitivity = −0.75ADD the two contributionsz sensitivity = −0.75× ReLU slope 1w gradient = −1.5× input x = 2Two paths add before moving upstreamThe same loss sensitivity contributes minus 0.5 via a and minus 0.25 via d. Sum minus 0.75 at h. Multiply by ReLU slope one and input two to obtain w gradient minus 1.5.Loss sensitivity at prediction: −0.5via a: −0.5−0.5 × vvia d: −0.25−0.5 × uh sensitivity = −0.75ADD the two contributionsz sensitivity = −0.75× ReLU slope 1w gradient = −1.5× input x = 2

These arrows follow the backward calculation. The branch contributions add before propagating through the activation. BP-C02, BP-C11.

This is the multivariable chain rule. If a value qq feeds several direct consumers rjr_j, accumulate every contribution before propagating farther upstream:

qˉ=∑jrˉj∂rj∂q\bar q=\sum_j\bar r_j\frac{\partial r_j}{\partial q}

Conceptually, initialize a node’s sensitivity to zero and add contributions. Overwriting −0.5 with −0.25 discards a real dependence. Add first, then continue in reverse dependency order. BP-C02

Through the activation

Our z=1z=1 is in ReLU’s positive region, whose local slope is one. Hence zˉ=hˉ(1)=−0.75\bar z=\bar h(1)=-0.75. For z=wx+bz=wx+b, the local derivatives are xx with respect to ww and one with respect to bb:

wˉ=zˉ x=−0.75(2)=−1.5\bar w=\bar z\,x=-0.75(2)=-1.5
bˉ=zˉ=−0.75\bar b=\bar z=-0.75
All gradients at the default forward pass
ParameterCalculationGradient
wzˉx\bar z x-1.5
bzˉ\bar z-0.75
vaˉh\bar a h-0.5
udˉh\bar d h-0.5
cy^ˉ\bar{\hat y}-0.5

The gradient collects all five partial derivatives in the declared order (w,b,v,u,c)(w,b,v,u,c). A negative gradient for ww predicts that a small increase in ww lowers loss. Gradient descent subtracts that negative gradient. BP-C11, BP-C15

A zero needs a diagnosis

For z<0z<0, ReLU stays zero under sufficiently small perturbations. Its slope is zero, blocking the signal to ww and bb for this example. With h=0h=0, gradients of vv and uu are also zero. Output bias cc can still have a nonzero gradient. A blocked hidden route does not imply every parameter stopped learning.

At exactly z=0z=0, ReLU has no ordinary derivative. Our code uses a zero backward rule there, consistent with PyTorch’s minimum-norm subgradient convention for this convex operation. This is an implementation choice, not an ordinary derivative. Autograd’s nondifferentiable-function rules give the convention. BP-C04, BP-C14

Another cause of zero is cancellation. If v=1,u=−1v=1,u=-1 while h=1h=1, contributions to hˉ\bar h cancel. The activation is positive. Inspect the routes before diagnosing an inactive activation.

Three kinds of accumulation

The path sum belongs to one graph. If a batch loss is a sum of example losses, its gradient is their sum; if it is a mean over a fixed batch of NN examples, the gradient includes the same 1/N1/N factor. Reduction changes the objective you differentiate.

There is also framework storage behavior: PyTorch accumulates gradients into parameter .grad buffers across backward calls. A usual fresh training step clears them with optimizer.zero_grad() before computing new gradients. That housekeeping does not replace the chain rule’s path sum. The optimization tutorial shows the ordering. BP-C06