Gates over time

Keep the six events fixed. Change one multiplier and trace the consequence.

A controlled coordinate—not a trained model

This lab directly prescribes one scalar coordinate, starting from $c_0=h_0=0$ . The candidate sequence is +1, 0, 0, −1, 0, 0. The first input gate is 1; the counter-cue's input gate is the control $w$ ; the others are zero. The same selected $f$ and $o$ apply at all six steps.

There are no trained weights, text encoder, or learning algorithm here. Exact gate values 0 and 1, and candidates −1 and 1, are idealized limiting values chosen to make retention and cancellation inspectable—not finite mathematical sigmoid/tanh outputs.

Before changing anything, predict: if you set output exposure to zero, will the cell state disappear too? Then compare the cell and hidden traces. The step control selects a calculation to inspect; it does not change the trajectory.

MEMORY LAB

Keep. Write. Expose.

One scalar cell · prescribed gates · no trained network

Make a prediction (optional)

If output o becomes zero, must cell c become zero? Yes, memory is erased. No, it may stay stored.

Keep old memory f 0.9At every stepAdmit counter-cue i₄ 0.0Only at step 4Expose memory o 1.0At every step

● Cell c□ Hidden h

Fixed scale for this toy. Overlapping circle/square marks mean the states coincide. Select a step below.

Inspect step 4 · counter-cue

Step 4 · counter-cue Previous cell 0.810 · candidate -1.000

0.729retained0.9 × 0.810

0.000written0.0 × (-1.000)

0.729new cell cstored state

Hidden h = 1.0 × tanh(0.729) = 0.622

The counter-cue proposes −1. The input multiplier admits 0.0 of that negative write; it is added to retained memory.

Step 4: cell 0.729, hidden 0.622. Final cell 0.590, hidden 0.530. The counter-cue is blocked; the original cue is only scaled by retention. Changing output leaves c unchanged here because future gates are prescribed independently of h.

Gates and candidates are hand-set, independently of h. Exact gate 0/1 and candidate ±1 are limiting idealizations, not finite-real sigmoid/tanh values. This is forward arithmetic, not training or a measured gradient. Calculations use full precision; visible values round to three decimals. A learned LSTM can feed h into future gates, unlike this experiment.

Inspect every step's values

Derived toy states · displayed to three decimals
Step / event	g	i	c	h
1 · cue	1.000	1.000	1.000	0.762
2 · gap	0.000	0.000	0.900	0.716
3 · gap	0.000	0.000	0.810	0.670
4 · counter-cue ← inspecting	-1.000	0.000	0.729	0.622
5 · gap	0.000	0.000	0.656	0.576
6 · read	0.000	0.000	0.590	0.530

Static comparisons: keep, fade, cancel, hide

Same prescribed sequence; different multipliers. These final-step comparisons remain available without scripts.

Derived final state at step 6
Case	f	i₄	o	c₆	h₆
Keep the cue	1.0	0.0	1.0	1.000	0.762
Let it fade	0.5	0.0	1.0	0.031	0.031
Admit counter-cue	1.0	1.0	1.0	0.000	0.000
Close output	1.0	0.0	0.0	1.000	0.000

Closing output hides the cell from h; it does not erase the toy cell. Admitting the equal, opposite counter-cue at f=1 cancels the cell rather than proving that nothing was written.

Follow the first write

At the defaults, $f=0.9$ , $w=0$ , and $o=1$ . The first write makes $c_1=1$ . Each later step keeps nine tenths of what remains, with no new write. At the selected fourth step:

c_4=0.9^3=0.729

There are three transitions after the first write—not four. Lower $f$ and watch that contribution fade faster. Set $f=1$ to see idealized perfect retention in this fixed-gate example. Neither curve measures a trained model's ability to remember a task.

A write can oppose what is already there

With $f=1$ , raise $w$ from 0 to 1. Step 4 admits the candidate −1: the retained +1 and the written −1 cancel. The resulting zero does not mean that nothing was written.

Now set $w=0$ and lower $o$ to zero. Every hidden value becomes zero, while the cell retains the cue. In this toy, future gates and candidates are prescribed independently of hidden state, so the entire cell trajectory is unchanged. A learned LSTM need not preserve that future invariance.

Track both contributions to the final step

The first write crosses five subsequent transitions to reach step 6. The negative write at step 4 crosses two. Their contributions add:

c_6=f^5-wf^2

h_6=o\tanh(c_6)

With the defaults, $c_6=0.59049$ . The display rounds to three decimals; calculations use the unrounded values. This is an exact decomposition of the prescribed sequence, not a general formula for every LSTM input.

Repeated multiplication explains forward retention. A related question appears during training: can a later error still influence an earlier computation?