Understanding Self-Attention

Self-attention is arguably the most important innovation in the transformer architecture. It allows the model to weigh the importance of different parts of the input when processing each element.

What is Self-Attention?

At its core, self-attention is a mechanism that allows each position in a sequence to attend to all positions in the same sequence. This is different from traditional recurrent networks, which process sequences one step at a time.

The Query, Key, Value Framework

Self-attention works by computing three vectors for each input token:

• **Query (Q)**: What am I looking for?
• **Key (K)**: What do I contain?
• **Value (V)**: What information do I provide?

The attention weights are computed by taking the dot product of the query with all keys, then applying a softmax to get a probability distribution.

Why It Matters

Self-attention allows transformers to:

1. Capture long-range dependencies efficiently
2. Process sequences in parallel
3. Learn rich representations of context

This is why transformers have become the dominant architecture for language models and beyond.

*Want to see self-attention in action? Check out the [Attention Visualizer](/apps/attention) app.*