Loading...
Subject 01 · LLM & NLP Interactive Lesson
The Attention Mechanism
In machine learning, attention is a method that determines the importance of each component in a sequence relative to others. Unlike static weights computed during training, attention weights are "soft"—they are computed dynamically in the forward pass and change with every input sequence.
O(n²)
Complexity limit
Q, K, V
Projection vectors
Soft Weights
Dynamic values
Parallelizable
Replaced RNNs
Self-Attention vs. Recurrent Models
Inspired by ideas about attention in humans, the attention mechanism was developed to address the weaknesses of using information from the hidden layers of recurrent neural networks (RNNs). Recurrent neural networks favor information contained in words at the end of a sentence (recency bias), thereby tending to attenuate the significance of information earlier in the sentence.
Attention allows a token equal access to any part of a sequence directly, rather than only bottlenecking through successive hidden states. By shifting from recurrence to parallel self-attention, the Transformer model solved key training performance limitations, making it the foundation for models like BERT, T5, and GPT.
Attention Pipeline Diagram
Figure: The query vector q attends over the key matrix K to calculate soft similarity weights α, combining value matrix V into a context vector.
Interactive Alignment Explorer
Observe how alignment matching maps words dynamically between languages during decoding (e.g., translating "I love you" to French "je t'aime").
Step 1 of 3
Target Translation
"je"
Attends To
"I"
Attention Weight
94%
On the first pass of the decoder, 94% of the attention weight is focused on the source word 'I'. This guides the model to produce the translation 'je'.
Alignment Matrix: High attention weights (dark cells) reveal cross-lingual mapping.
Scaled Dot-Product Formula Explorer
Click on any part of the mathematical formula below to view its dimensions, purpose, and role in computing self-attention.
Attention(,,) =(Q Kᵀ/)V
Click on a variable in the formula above (Q, K, V, softmax, or √d_k) to view its mathematical description.
Mathematical Variants
Standard Scaled Dot-Product
For queries matrix Q ∈ ℝ^(m×d_k), keys matrix K ∈ ℝ^(n×d_k) and values matrix V ∈ ℝ^(n×d_v), the scores represent similarity projections. Equivariant to queries re-ordering and invariant to key-value shuffling.
Attention(Q, K, V) = softmax( QKᵀ / √d_k ) · V ∈ ℝ^(m×d_v)
Attention Alignment Architectures
1. Encoder-Decoder Dot Product
Both encoder and decoder states are needed to compute attention weights. Score defined by: w_ij = x_i · h_j
2. Encoder-Decoder QKV
Queries are projected from decoder states, while keys and values are projected from encoder outputs.[44]
3. Encoder-Only QKV
Decoders are omitted entirely. Queries, keys, and values are computed strictly from the encoder input states (self-attention).[46]
4. Encoder-only dot product
No decoder used. With only 1 sequence input, the weight matrix is calculated via an auto-correlation: w_ij = x_i · x_j.[45]
5. Fully-Connected Layers (PyTorch)
Computes attention scores using parameterized feedforward linear layers instead of raw dot-product similarity metrics.[47]
Attention Optimizations
FlashAttention
Standard attention has quadratic time/memory complexity O(N²). FlashAttention partitions computations into blocks fitting inside faster GPU SRAM, avoiding the storage of massive intermediate matrices.[48]
FlexAttention
A PyTorch-native flexible API developed by Meta allowing users to customize and modify attention score values before softmax while maintaining FlashAttention execution speed.[49]
Saliency & Vision Transformers
Visualizing attention weights as saliency heatmaps is standard practice to inspect decision making in Vision Transformers (ViT). Since models are often trained self-supervised, maps are not native class-sensitive.
Key Methodologies
- Attention Rollout: Recursive dot products combining maps across all layers.[51]
- CDAM: Combines attention maps and class [CLS] gradients.[52]
- GradCAM: Back-propagates gradients directly to attention outputs.[53]
History & Historical Context
1950s–1960s
Cognitive Psychology Foundations
Development of the Cocktail Party Effect (focusing on target auditory stimuli while filtering out background noise) and the filter model of attention.
1990s
Fast Weights & Dynamic Links
Early models of fast weight controllers and dynamic links between neurons, anticipating key-value routing mechanisms.
2014
RNN Seq2Seq with Attention
Bahdanau et al. introduced learned attention to handle translation of long sentences in sequence-to-sequence recurrent neural networks.
2017
The Transformer & Self-Attention
The landmark paper 'Attention Is All You Need' formalized scaled dot-product self-attention: A = softmax(QKᵀ / √d_k)V, discarding recurrence entirely.
2020+
Linear & Hardware Optimizations
Linformer, Reformer, FlashAttention, and FlexAttention introduced to reduce quadratic overhead and optimize GPU execution for long context windows.