The cue is no longer in front of you

Imagine six events: a cue, two quiet gaps, a counter-cue, another gap, and a request to read the state. At the last step, the current event says only “read.” To respond to the earlier cue, a system needs a state that survives between steps.

We will use this invented sequence throughout. Its event names are a teaching story, not text being encoded by a trained model. Later, we will prescribe one numerical coordinate and inspect exactly what survives, changes, and becomes visible.

The question is not just whether the first cue can persist. What should happen when the counter-cue arrives? And can stored state remain present even when the system exposes nothing?

One update, used repeatedly

A recurrent neural network, or RNN, updates a state using the current input and a previous state. The same learned parameters are reused at successive positions; the inputs and states change. The state is a numerical representation, not a literal archive of every earlier event. See the recurrent update and unrolled computation in Pascanu, Mikolov, and Bengio, §1.1.

Write the current input vector as $x_t$ and the hidden state as $h_t$. The subscript $t$ counts sequence steps. An update takes $x_t$ and $h_{t-1}$ and produces $h_t$, which becomes available to the next step.

Separate carrying from exposing

An LSTM—Long Short-Term Memory—is itself an RNN architecture. Its update separates a carried cell state from an exposed hidden state, using learned multipliers called gates. We will study the standard form shown in the LSTM equations, without peephole connections or an output projection.

Start with what one step computes. Once the distinction between storage and exposure is clear, the sequence—and the reason this structure can help learning—will be easier to follow.

Two states, three different jobs

The cell state $c_t$ is the carried state. The hidden state $h_t$ is a transformed, gated view of it. Both are vectors with $H$ coordinates; the input $x_t$ has $D$ coordinates. We use column vectors.

The gates act coordinate by coordinate, not as one global yes/no decision:

• Forget gate $f_t$: how much of the previous cell value to retain.
• Input gate $i_t$: how much of a proposed value to write. That proposal is the candidate $g_t$, which can be positive or negative.
• Output gate $o_t$: how much of the transformed cell value to expose as hidden state.

A gate value is a multiplier, not a probability that an event is true or important. The candidate is not a fourth sigmoid gate.

Retain + write = new cell state

First multiply the old cell state by the forget gate. Separately multiply the candidate by the input gate. Add those contributions:

The symbol $\odot$ means elementwise multiplication: multiply matching coordinates. Then expose part of a squashed version of the new state:

These are the standard cell and hidden-state updates. Notice the addition: admitting a candidate does not automatically replace the old state. A negative write can offset a positive retained value. The forget and input gates are separate; they do not have to sum to one.

Bounded multipliers, not a bounded cell

A learned cell computes its gates with sigmoid and its candidate with tanh. For finite real inputs, mathematical sigmoid is strictly between 0 and 1, and tanh is strictly between −1 and 1. These ranges follow from their sigmoid and tanh definitions; floating-point saturation can round to an endpoint.

That does not bound $c_t$ to this interval: its update adds two terms. For example, an old cell value of 1 with $f=i=g=0.9$ becomes $0.9+0.81=1.71$. The standard hidden state is bounded coordinatewise between −1 and 1 because it uses $\tanh(c_t)$ and an output multiplier.

Holding the other gate and candidate values fixed, changing $o_t$ changes current $h_t$, not current $c_t$. In a learned recurrent cell, however, $h_t$ helps compute future gates, so later cell states can change. Our next experiment deliberately removes that feedback to isolate the arithmetic.

A controlled coordinate—not a trained model

This lab directly prescribes one scalar coordinate, starting from $c_0=h_0=0$. The candidate sequence is +1, 0, 0, −1, 0, 0. The first input gate is 1; the counter-cue's input gate is the control $w$; the others are zero. The same selected $f$ and $o$ apply at all six steps.

There are no trained weights, text encoder, or learning algorithm here. Exact gate values 0 and 1, and candidates −1 and 1, are idealized limiting values chosen to make retention and cancellation inspectable—not finite mathematical sigmoid/tanh outputs.

Before changing anything, predict: if you set output exposure to zero, will the cell state disappear too? Then compare the cell and hidden traces. The step control selects a calculation to inspect; it does not change the trajectory.

Follow the first write

At the defaults, $f=0.9$, $w=0$, and $o=1$. The first write makes $c_1=1$. Each later step keeps nine tenths of what remains, with no new write. At the selected fourth step:

There are three transitions after the first write—not four. Lower $f$ and watch that contribution fade faster. Set $f=1$ to see idealized perfect retention in this fixed-gate example. Neither curve measures a trained model's ability to remember a task.

A write can oppose what is already there

With $f=1$, raise $w$ from 0 to 1. Step 4 admits the candidate −1: the retained +1 and the written −1 cancel. The resulting zero does not mean that nothing was written.

Now set $w=0$ and lower $o$ to zero. Every hidden value becomes zero, while the cell retains the cue. In this toy, future gates and candidates are prescribed independently of hidden state, so the entire cell trajectory is unchanged. A learned LSTM need not preserve that future invariance.

Repeated multiplication explains forward retention. A related question appears during training: can a later error still influence an earlier computation?

Two directions, two different questions

In a forward pass, we ask: what state results from these inputs? During training, we also ask: how would a small change in an earlier quantity affect a later loss? A derivative expresses local sensitivity; gradients collect such sensitivities for the parameters being trained.

Across many recurrent steps, gradient contributions involve products of stepwise derivatives. These can shrink or grow strongly, making optimization difficult—the vanishing and exploding gradient problems. This is not simply another name for a stored number fading in the forward pass. Pascanu, Mikolov, and Bengio, §§1–2 analyze these temporal products.

Isolate the direct carry path

Look only at the direct cell-state route and hold gate and candidate values fixed. A small change $\Delta c_{t-1}$ then passes through the carry term as:

For one scalar coordinate, call the sensitivity along this restricted path from step $s$ to step $t$ $D_{s\to t}$. Applying the chain rule repeatedly gives:

The product symbol means “multiply the forget values over these transitions.” Near-one factors can preserve this route's sensitivity better than repeatedly multiplying by small factors. This calculation is not the full recurrent derivative: it leaves out paths through hidden states and learned gate/candidate computations.

What the curves do—and do not—show

With constant $f$, the restricted sensitivity after $k$ transitions is $f^k$. The same factor describes the forward retention of an existing cell contribution when no new writes occur:

These two interpretations share an algebraic factor, but answer different questions. The figure is an analytic fixed-gate comparison, not a measured training curve, gradient norm, or memory-capacity score.

The additive cell route provides a way for a signal to travel without repeatedly traversing a squashing operation on that route. Learned gates, other recurrent paths, the output transformation, and the task loss still matter. An LSTM does not eliminate every vanishing or exploding gradient problem, or guarantee that a distant cue will be learned.

So far we have chosen multipliers by hand. To turn this mechanism into a model, the next question is who chooses them—and how.

The same weights, new decisions

In the standard cell, concatenate the previous hidden state and current input into $z_t=[h_{t-1};x_t]$. The semicolon means stacking the column vectors. Learned weighted sums plus biases produce three gates and the candidate:

Here $\sigma$ is sigmoid. These are the standard gate equations written with combined input/hidden matrices. The previous cell state enters the subsequent retain/write update; it does not directly enter these gate equations in our no-peephole variant.

The parameters are shared across steps, but $x_t$ and $h_{t-1}$ vary, so the resulting gates can vary. An ordinary forward pass changes states, not weights. Training uses a loss, computes gradients through the unrolled computation—backpropagation through time, or BPTT—and updates the shared parameters. See §1.1 and Fig. 2; our lab performs none of that training.

The output gate is not the answer

An LSTM produces hidden-state features. A separate task readout can turn selected features into a prediction—for example, a forecast at each step or a classification from a final hidden state. That design choice is distinct from the internal output gate. The documented LSTM outputs are states; Gers et al., §2.1, Eq. (7) give an example with a separate output layer.

In our cue story, “read” is only an inspection point. No task head interprets the cell as a word, an answer, or a probability. To build a predictive system, we would need an input representation, a readout, a training objective, and evaluation on the actual task.

Try a new update

Leave the lab's fixed sequence. Suppose the previous cell value is $0.8$, with $f=0.75$, $i=0.5$, $g=-0.4$, and $o=0$. What are the retained contribution, written contribution, new cell state, and hidden state?

Then stipulate one more step with $f=1$, $i=0$, and $o=1$. What becomes visible? The exact endpoints are the same idealization used in the lab.

Keep the mechanism in view

An LSTM is a learned recurrent update with a useful separation: retain old state, write a candidate, expose a transformed view. Cell state and hidden state are related, but not interchangeable.

When reading an LSTM diagram or implementation, trace those two states first. Then ask what computes the gates, where the prediction comes from, and which assumptions support any claim about long-term memory.