↗LSTM
Chapter 2 of 5
Notes & sources
Contents ≡
Contents / Chapter 2

02

Inside the cell

Keep some old state, add a proposed write, then decide how much to expose.

Two states, three different jobs

The cell state ctc_t is the carried state. The hidden state hth_t is a transformed, gated view of it. Both are vectors with HH coordinates; the input xtx_t has DD coordinates. We use column vectors.

The gates act coordinate by coordinate, not as one global yes/no decision:

  • Forget gate ftf_t: how much of the previous cell value to retain.
  • Input gate iti_t: how much of a proposed value to write. That proposal is the candidate gtg_t, which can be positive or negative.
  • Output gate oto_t: how much of the transformed cell value to expose as hidden state.

A gate value is a multiplier, not a probability that an event is true or important. The candidate is not a fourth sigmoid gate.

Retain + write = new cell state

First multiply the old cell state by the forget gate. Separately multiply the candidate by the input gate. Add those contributions:

ct=ft⊙ct−1+it⊙gtc_t=f_t\odot c_{t-1}+i_t\odot g_t

The symbol ⊙\odot means elementwise multiplication: multiply matching coordinates. Then expose part of a squashed version of the new state:

ht=ot⊙tanh⁡(ct)h_t=o_t\odot\tanh(c_t)
FIG 002Two terms stored. One branch exposed.
Two terms stored. One branch exposed.The previous cell state is multiplied by the forget gate, then added to input gate times candidate. The new cell also branches through tanh and the output gate to produce hidden state.cₜ₋₁×+cₜfₜretainednew celliₜ×gₜwrittentanh×oₜhₜGates and candidate come from [hₜ₋₁; xₜ].Two terms stored. One branch exposed.The previous cell state is multiplied by the forget gate, then added to input gate times candidate. The new cell also branches through tanh and the output gate to produce hidden state.cₜ₋₁×fₜretain+gₜ×iₜwritecₜtanh×oₜhₜ

Follow the heavy cell-state route: multiply, then add. The branch to h reads the new cell through tanh and the output gate. This diagram shows update arithmetic; chapter 5 gives the shared learned gate calculations.

These are the standard cell and hidden-state updates. Notice the addition: admitting a candidate does not automatically replace the old state. A negative write can offset a positive retained value. The forget and input gates are separate; they do not have to sum to one.

Bounded multipliers, not a bounded cell

A learned cell computes its gates with sigmoid and its candidate with tanh. For finite real inputs, mathematical sigmoid is strictly between 0 and 1, and tanh is strictly between −1 and 1. These ranges follow from their sigmoid and tanh definitions; floating-point saturation can round to an endpoint.

That does not bound ctc_t to this interval: its update adds two terms. For example, an old cell value of 1 with f=i=g=0.9f=i=g=0.9 becomes 0.9+0.81=1.710.9+0.81=1.71. The standard hidden state is bounded coordinatewise between −1 and 1 because it uses tanh⁡(ct)\tanh(c_t) and an output multiplier.

Holding the other gate and candidate values fixed, changing oto_t changes current hth_t, not current ctc_t. In a learned recurrent cell, however, hth_t helps compute future gates, so later cell states can change. Our next experiment deliberately removes that feedback to isolate the arithmetic.

Why specify which LSTM?

The name covers a family. Hochreiter and Schmidhuber's 1997 LSTM used input/output gates and a fixed cell self-connection. The later forget-gate extension is described in Gers, Schmidhuber, and Cummins's 2000 journal paper. The exact modern equations above should not be attributed unchanged to the original paper.