A path for gradients

The carry route can preserve a learning signal. It does not guarantee unlimited memory or easy training.

Two directions, two different questions

In a forward pass, we ask: what state results from these inputs? During training, we also ask: how would a small change in an earlier quantity affect a later loss? A derivative expresses local sensitivity; gradients collect such sensitivities for the parameters being trained.

Across many recurrent steps, gradient contributions involve products of stepwise derivatives. These can shrink or grow strongly, making optimization difficult—the vanishing and exploding gradient problems. This is not simply another name for a stored number fading in the forward pass. Pascanu, Mikolov, and Bengio, §§1–2 analyze these temporal products.

Isolate the direct carry path

Look only at the direct cell-state route and hold gate and candidate values fixed. A small change $\Delta c_{t-1}$ then passes through the carry term as:

\Delta c_t=f_t\,\Delta c_{t-1}

For one scalar coordinate, call the sensitivity along this restricted path from step $s$ to step $t$ $D_{s\to t}$ . Applying the chain rule repeatedly gives:

D_{s\to t}=\prod_{j=s+1}^{t}f_j

The product symbol means “multiply the forget values over these transitions.” Near-one factors can preserve this route's sensitivity better than repeatedly multiplying by small factors. This calculation is not the full recurrent derivative: it leaves out paths through hidden states and learned gate/candidate computations.

FIG 004One retained contribution, several horizons

Derived fᵏ, with no new writes and gates held fixed. The same product describes the direct cell-state sensitivity under those assumptions—not the full LSTM gradient or measured memory performance. k counts transitions after the starting value; f=1 is an idealized limit.

What the curves do—and do not—show

With constant $f$ , the restricted sensitivity after $k$ transitions is $f^k$ . The same factor describes the forward retention of an existing cell contribution when no new writes occur:

c_{s+k}=f^k c_s

These two interpretations share an algebraic factor, but answer different questions. The figure is an analytic fixed-gate comparison, not a measured training curve, gradient norm, or memory-capacity score.

The additive cell route provides a way for a signal to travel without repeatedly traversing a squashing operation on that route. Learned gates, other recurrent paths, the output transformation, and the task loss still matter. An LSTM does not eliminate every vanishing or exploding gradient problem, or guarantee that a distant cue will be learned.

So far we have chosen multipliers by hand. To turn this mechanism into a model, the next question is who chooses them—and how.