Attention Is (Almost) All You Need
Almost every model you’ve heard of lately — the chatbots, the image encoders, the code assistants — descends from one 2017 paper with an unusually confident title: Attention Is All You Need (Vaswani et al.). The claim was that you could throw out the machinery everyone thought was essential for processing sequences — recurrence, convolution — and keep just one mechanism: attention.
What attention actually does
The idea is simpler than the name suggests. For each position in a sequence, the model asks: which other positions are relevant to me, and how much? It answers by comparing a “query” from the current position against a “key” from every other position; the better the match, the more weight that position gets. Then it pulls in a weighted blend of everyone’s “values.” A word deciding what it means looks at all the other words at once and takes what it needs.
The whole operation is one tidy formula — softmax(QKᵀ/√dₖ)·V — and the model runs several of these “heads” in parallel, each free to track a different kind of relationship.
Why it won
The old recurrent approach processed a sequence step by step: to relate position 1 to position 100, information had to travel through 99 intermediate steps. Attention connects any two positions directly, in a single hop. That has two consequences. Long-range relationships are easier to learn, and — crucially — the computation parallelizes beautifully instead of being forced into sequence order. The Transformer set new translation records (28.4 BLEU English-to-German, 41.8 English-to-French) at a fraction of the training cost of the systems it beat.
The quiet punchline
Here’s the detail worth sitting with: the original Transformers were tiny. The base model had about 65M parameters, the big one about 213M — a rounding error next to today’s models. The 2017 paper established the architecture; the scale came years later. Which is the whole reason it matters so much now — it turned out to be the substrate that scaling laws could ride, from vision transformers to large language models. The right shape came first. Then the field poured compute into it.
References
- Vaswani, A. et al. (2017). Attention Is All You Need. NIPS.