Local slopes, global consequences

At each operation, multiply the sensitivity arriving from the loss by a derivative of that operation.

What a derivative says

A derivative is a local sensitivity. Writing $\partial L/\partial w=-1.5$ means that a tiny increase in $w$ , with other parameters fixed, changes loss at about −1.5 times that increase. It is neither a weight value nor a command to add −1.5.

\Delta L\approx \frac{\partial L}{\partial w}\,\Delta w

For $\Delta w=0.001$ , the first-order estimate is $\Delta L\approx-0.0015$ . This approximation concerns sufficiently small changes in a differentiable region; it is not an exact finite-step promise. BP-C15

We use compact notation: $\bar q=\partial L/\partial q$ . Read “q-bar” as the sensitivity of the loss to q. It is an intermediate derivative, often called an adjoint; it is not the forward value $q$ .

Multiply along a route

Suppose $r=f(q)$ , and the loss depends on $r$ . The chain rule connects the small operation’s derivative to the loss:

\bar q=\bar r\,\frac{\partial r}{\partial q}

Seed the calculation with $\bar L=1$ : the sensitivity of $L$ to itself is one. For $L=e^2/2$ , the local derivative is $e$ . Thus $\bar e=1(-0.5)=-0.5$ . Since $e=\hat y-y$ , its derivative with respect to $\hat y$ is one, so $\bar{\hat y}=-0.5$ .

BP-F02 · Follow the scalar loss backward
Operation	Local slope	Loss sensitivity
$L=e^2/2$	$\partial L/\partial e=e$	$\bar e=-0.5$
$e=\hat y-y$	$\partial e/\partial\hat y=1$	$\bar{\hat y}=-0.5$
$\hat y=a+d+c$	1 for each input	$\bar a=\bar d=\bar c=-0.5$

At the addition $\hat y=a+d+c$ , each local derivative is one. Each branch receives the same incoming sensitivity, −0.5. The forward sum does not divide it between its inputs. At multiplication $a=vh$ , the local derivative with respect to $v$ is $h$ ; with respect to $h$ it is $v$ .

\bar v=\bar a\,h=-0.5(1)=-0.5

\bar h_{\text{via }a}=\bar a\,v=-0.5(1)=-0.5

The official autograd explanation describes reverse-mode differentiation as applying the chain rule over recorded operations. BP-C01–02

Keep the two derivatives separate

The local derivative $\partial a/\partial h=v=1$ says how $a$ changes with $h$ . The loss sensitivity contributed through that branch is $\bar a\,v=-0.5$ . Forgetting the incoming factor confuses an operation’s behavior with its effect on the objective.

There is another issue: $h$ also changes the second branch. We have found one contribution, so we are not ready to propagate backward through ReLU yet.