↗LSTM
Chapter 1 of 5
Notes & sources
Contents ≡
Contents / Chapter 1

01

Remembering a sequence

A later decision may depend on an earlier event. A recurrent state carries something forward.

The cue is no longer in front of you

Imagine six events: a cue, two quiet gaps, a counter-cue, another gap, and a request to read the state. At the last step, the current event says only “read.” To respond to the earlier cue, a system needs a state that survives between steps.

We will use this invented sequence throughout. Its event names are a teaching story, not text being encoded by a trained model. Later, we will prescribe one numerical coordinate and inspect exactly what survives, changes, and becomes visible.

FIG 001The past has to reach the readout
The past has to reach the readoutAn invented six-event story: positive cue, gap, gap, negative counter-cue, gap, read. The current event alone does not specify which earlier cue should matter.1cue+12gap3gap4counter-cue−15gap6readA signal arrives early. The read comes later.?The past has to reach the readoutAn invented six-event story: positive cue, gap, gap, negative counter-cue, gap, read. The current event alone does not specify which earlier cue should matter.1 cue+12 gap3 gap4 counter-cue−15 gap6 readread state

Invented signal story, not trained behavior. The signed stems mark the two candidate writes used later in the lab; their time positions stay fixed. Whether either cue survives depends on the state updates.

The question is not just whether the first cue can persist. What should happen when the counter-cue arrives? And can stored state remain present even when the system exposes nothing?

One update, used repeatedly

A recurrent neural network, or RNN, updates a state using the current input and a previous state. The same learned parameters are reused at successive positions; the inputs and states change. The state is a numerical representation, not a literal archive of every earlier event. See the recurrent update and unrolled computation in Pascanu, Mikolov, and Bengio, §1.1.

Write the current input vector as xtx_t and the hidden state as hth_t. The subscript tt counts sequence steps. An update takes xtx_t and ht−1h_{t-1} and produces hth_t, which becomes available to the next step.

A plain tanh RNN, in one equation

One common baseline mixes the input and previous hidden state, then applies tanh⁡\tanh, a function that squashes each value between −1 and 1:

ht=tanh⁡(Wxxt+Whht−1+b)h_t=\tanh(W_xx_t+W_hh_{t-1}+b)

The matrices Wx,WhW_x,W_h and bias bb are learned parameters. They have no time subscript because this update reuses them. This is a plain tanh RNN, not the definition of every recurrent architecture. The PyTorch RNN equations use the same idea, with the bias written in two parts.

Separate carrying from exposing

An LSTM—Long Short-Term Memory—is itself an RNN architecture. Its update separates a carried cell state from an exposed hidden state, using learned multipliers called gates. We will study the standard form shown in the LSTM equations, without peephole connections or an output projection.

Start with what one step computes. Once the distinction between storage and exposure is clear, the sequence—and the reason this structure can help learning—will be easier to follow.