From cell to model

Learned weights choose the gates. A separate task readout turns hidden features into a prediction.

The same weights, new decisions

In the standard cell, concatenate the previous hidden state and current input into $z_t=[h_{t-1};x_t]$ . The semicolon means stacking the column vectors. Learned weighted sums plus biases produce three gates and the candidate:

\begin{aligned}f_t&=\sigma(W_fz_t+b_f)\\i_t&=\sigma(W_iz_t+b_i)\\g_t&=\tanh(W_gz_t+b_g)\\o_t&=\sigma(W_oz_t+b_o)\end{aligned}

Here $\sigma$ is sigmoid. These are the standard gate equations written with combined input/hidden matrices. The previous cell state enters the subsequent retain/write update; it does not directly enter these gate equations in our no-peephole variant.

Check the vector sizes

With input width $D$ and state width $H$ , $z_t$ has $H+D$ coordinates. Each combined matrix $W_f,W_i,W_g,W_o$ has shape $H\times(H+D)$ , and each bias has $H$ coordinates. Thus every gate and candidate has the same width as the cell state.

This is a mathematical convention, not a requirement for a library's physical weight or bias layout. Each coordinate gets its own multipliers; they need not correspond to human-named facts.

The parameters are shared across steps, but $x_t$ and $h_{t-1}$ vary, so the resulting gates can vary. An ordinary forward pass changes states, not weights. Training uses a loss, computes gradients through the unrolled computation—backpropagation through time, or BPTT—and updates the shared parameters. See §1.1 and Fig. 2; our lab performs none of that training.

The output gate is not the answer

An LSTM produces hidden-state features. A separate task readout can turn selected features into a prediction—for example, a forecast at each step or a classification from a final hidden state. That design choice is distinct from the internal output gate. The documented LSTM outputs are states; Gers et al., §2.1, Eq. (7) give an example with a separate output layer.

In our cue story, “read” is only an inspection point. No task head interprets the cell as a word, an answer, or a probability. To build a predictive system, we would need an input representation, a readout, a training objective, and evaluation on the actual task.

Try a new update

Leave the lab's fixed sequence. Suppose the previous cell value is $0.8$ , with $f=0.75$ , $i=0.5$ , $g=-0.4$ , and $o=0$ . What are the retained contribution, written contribution, new cell state, and hidden state?

Then stipulate one more step with $f=1$ , $i=0$ , and $o=1$ . What becomes visible? The exact endpoints are the same idealization used in the lab.

Work through the answer

\begin{aligned}\text{retained}&=0.75\times0.8=0.6\\\text{written}&=0.5\times(-0.4)=-0.2\end{aligned}

\begin{aligned}c&=0.6-0.2=0.4\\h&=0\times\tanh(0.4)=0\end{aligned}

The hidden value is zero, but the cell value is not. At the stipulated next step, the entire cell value is retained and no new value is written:

\begin{aligned}c_{\text{next}}&=0.4\\h_{\text{next}}&=\tanh(0.4)\approx0.380\end{aligned}

A closed output gate did not erase the current cell. We could read it later because the next gates were stipulated. A learned recurrent model could choose different next gates as a consequence of the changed hidden state.

Keep the mechanism in view

An LSTM is a learned recurrent update with a useful separation: retain old state, write a candidate, expose a transformed view. Cell state and hidden state are related, but not interchangeable.

When reading an LSTM diagram or implementation, trace those two states first. Then ask what computes the gates, where the prediction comes from, and which assumptions support any claim about long-term memory.