04
A gradient is not a step
The backward pass reports sensitivities. A learning rate turns them into a finite proposal—and the next forward pass judges it.
Make the proposal
Write parameters as and their gradient as . Plain gradient descent chooses a nonnegative learning rate :
We use the same single example each time: deterministic gradient descent. This is the no-momentum, no-weight-decay case of PyTorch’s SGD rule. All five deltas use the same old forward pass and gradient, then all parameters change together. Updating one before computing the remaining gradients would be a different procedure. BP-C05
At , the first weight delta is , so . The complete new vector is . A fresh forward pass gives prediction 1.865625 and loss about 0.00902832.
Try the next step
Before revealing a proposal, predict: with the initial example, will rate 0.5 lower the loss? The gradient is still correct; the question is how far its local information should be trusted.
Inspect the update
Synthetic scalar network · one fixed example · simultaneous gradient descent · full precision, rounded display
ReLU is active initially. Both branch contributions reach the early layer.
Changing case or rate restarts at update 0. Predict whether the next loss decreases, stays equal or increases.
Update 0. Current loss 0.125. Predict, then reveal the proposed update.
After 0 updates
Prediction 1.5
Loss 0.125
Hidden h 1
Current paths into h: -0.5 + (-0.25) = -0.75. z is positive: ReLU slope 1.
| Parameter | Value | Gradient |
|---|---|---|
| w | 0.5 | -1.5 |
| b | 0 | -0.75 |
| v | 1 | -0.5 |
| u | 0.5 | -0.5 |
| c | 0 | -0.5 |
Proposed next step
Next prediction 1.865625; next loss 0.009028 (lower).
| Parameter | Δ = −ηg | Next value |
|---|---|---|
| w | 0.075 | 0.575 |
| b | 0.0375 | 0.0375 |
| v | 0.025 | 1.025 |
| u | 0.025 | 0.525 |
| c | 0.025 | 0.025 |
| Update | Loss |
|---|---|
| 0 | 0.125 |
| Rate η | Next w | Prediction | Loss |
|---|---|---|---|
| 0 | 0.5 | 1.5 | 0.125 |
| 0.05 | 0.575 | 1.865625 | 0.009028 |
| 0.1 | 0.65 | 2.25 | 0.03125 |
| 0.5 | 1.25 | 6 | 8 |
| 1 | 2 | 12.375 | 53.820313 |
Initial loss 0.125. The gradient is the same for every row. Larger η changes the proposal; it does not improve the derivative. BP-C12 · deterministic toy calculation.
Closed/corner cases initially give zero gradients for w, b, v and u; c has gradient −2. Cancelling branches give gradients (0, 0, −2, −2, −2) while h stays positive.
At the initial parameters, changing only the rate never changes the gradient. Rate 0.05 lowers this loss; rate 0.5 produces prediction 6 and loss 8. A finite step can cross the target and make the error larger. These are toy consequences, not recommended rates for another network. BP-C12
After an update, compute the next gradient at the new parameters. Reusing the initial gradient for every step would follow a fixed direction. Large rates in this tiny system may later close ReLU and let the output bias alone fit the example; the first loss increase does not prove permanent divergence. In a closed or boundary case, that bias can move while the hidden route stays blocked.
Where autograd ends
This schematic PyTorch sequence explains responsibilities. The complete runnable companion for this guide is the dependency-free JavaScript lab linked in the notes.
optimizer.zero_grad()
prediction = model(x)
loss = 0.5 * (prediction - target).square()
loss.backward()
optimizer.step()backward() computes and accumulates derivatives for a scalar loss. step() applies the optimizer’s rule. Calculating gradients does not itself update model parameters. See Optimizing model parameters. Our browser lab implements the equations directly so you can inspect each term. BP-C03
These updates fit one invented pair. A falling loss here says nothing about a real demand forecast, held-out examples, data leakage or research validity. Those require a different experiment and evidence.