Trust the trace. Check the derivative.

Perturb one parameter

A numerical check provides another route to a derivative. Perturb only parameter $\theta_i$ by a small amount $\varepsilon>0$ in both directions and recompute the complete forward loss:

Here $e_i$ is the coordinate vector that changes only the selected parameter, unrelated to our residual $e$. At the default smooth point, $\varepsilon=10^{-6}$ gives agreement with all five analytic gradients within the lab’s absolute tolerance $10^{-8}$.

That agreement is separate from the hand-computed values. Both matter: a checker can miss a shared bug if its “numerical” loss reuses the analytic derivative. Our finite-difference path calls the forward loss alone. Gradcheck mechanics explains the comparison. BP-C07

import { DEFAULT_PARAMS, DEFAULT_EXAMPLE,
  backward, finiteDifference } from './model.mjs';

console.log(backward(DEFAULT_PARAMS,
  DEFAULT_EXAMPLE).gradients);
console.log(finiteDifference(DEFAULT_PARAMS,
  DEFAULT_EXAMPLE, { epsilon: 1e-6 }));

The expected analytic gradient is $(-1.5,-0.75,-0.5,-0.5,-0.5)$. Run the full companion for recorded outputs and assertions. Small final-digit differences in the numerical estimate are floating-point arithmetic.

The corner is different

Set $b=-1$ with other defaults unchanged. Now $z=0$. Perturbing $w$ in opposite directions samples different sides of ReLU’s corner. The centered estimate approaches −3; our chosen backward gradient at the corner is zero. This disagreement is expected: there is no ordinary derivative for both methods to recover.

Even away from a corner, a large perturbation may cross it. An excessively small perturbation can lose precision through subtraction. PyTorch’s gradcheck reference warns about nondifferentiable points and notes its default settings target double precision. Our tests exclude corners from derivative-agreement assertions and check the convention separately. BP-C04, BP-C08

When numbers become vectors

For column input $x\in\mathbb R^D$, weights $W\in\mathbb R^{H\times D}$ and bias $b\in\mathbb R^H$, an affine layer gives $z=Wx+b$. Let $\bar z=\partial L/\partial z$ be an $H$-coordinate column of sensitivities.

The transpose collects contributions from every output coordinate into each input coordinate. The outer product gives one derivative for every weight entry. A gradient has the same shape as its parameter. These equations use one column-vector example; batch dimensions and reductions need explicit treatment. BP-C13

Diagnose a new case

Try this without the lab. Keep the architecture but change values: $x=3,\ w=1/3,\ b=1,\ v=2,\ u=-2,\ c=1,\ y=3$. A colleague reports “the early-layer gradient is zero, so ReLU must be inactive.” Is that justified? Find $h,L,\bar h$ and all five parameter gradients. Will every parameter stay fixed under a positive-rate step?

When a gradient surprises you, inspect saved forward values, local derivatives, all downstream routes and the loss reduction. Then inspect the optimizer separately. These questions make “the model learned” a sequence of checkable operations.