AI
An intuitive memory framework links transformer layers to associative recall capacity
This technical analysis treats neural-network layers as associative memory stores and compares how interference limits recall. Under random-key assumptions, it estimates capacities of roughly 64 memories for a 64-dimensional linear store, about 2,700 for squared-ReLU-like nonlinear recall, and more than a million for self-attention at a stated sharpness. The author emphasizes that these are single-layer vector-recall estimates, not direct measures of factual knowledge or trained-model memory, and surveys implications for long-context attention.
Editorial summary · Publisher article
What matters
- Linear outer-product storage accumulates interference as more keys occupy limited dimensions.
- Least-squares storage cancels some interference and roughly doubles the estimated capacity.
- Nonlinearities suppress weak similarities, allowing MLPs to store more memories in superposition.
- Self-attention’s context-dependent keys and softmax weighting make its memory store non-parametric and expandable with context length.
Context & caveats
- The capacity figures rely on random keys and values and do not account for what trained models actually represent in their data or parameters.
Why this made the edition
A complete, technically substantial explainer derives an intuitive memory-capacity framework across linear layers, MLPs and attention, gives explicit assumptions and numerical examples, and clearly limits the conclusions to random key-value recall.
Source material