Vanilla RNNs and their memory limits
Recurrent neural networks process sequences by maintaining a hidden state that updates at each timestep. The hidden state at time t depends on the input at t and the hidden state at t-1, enabling the network to carry information forward. However, vanilla RNNs suffer from a critical flaw: the hidden state is recomputed at each step by multiplying by the same weight matrix. Over many steps, gradients either vanish (shrink to near zero) or explode (grow unbounded), making it impossible to learn long-range dependencies. Experiments show vanilla RNNs reliably forget information after 10-20 timesteps, no matter how much training. This limitation made them unsuitable for long sequences like sentences or documents.
LSTM gates and controlled information flow
LSTMs address vanilla RNN problems by introducing gates that explicitly control information flow. The forget gate decides what to discard from memory. The input gate decides what new information to store. The output gate decides what to expose. These gates are learned during training, allowing the model to maintain long-term dependencies by gating out irrelevant information and protecting important state. LSTMs proved capable of learning dependencies over hundreds of timesteps. They became the standard for sequence modeling for nearly a decade, powering machine translation, speech recognition, and language understanding. Modern transformers replaced RNNs by using attention instead of recurrence, eliminating the sequential computation bottleneck entirely while maintaining the ability to learn long-range dependencies.