Vision Transformers Needed a Scratchpad

Make an attention map of a large vision transformer — light up the patches it attends to — and you expect it to trace the objects in the scene. Often it does. But scattered across the image, usually in the blandest regions (a patch of empty sky, a smear of out-of-focus grass), you find a few bright spikes that correspond to nothing at all. For a while these were shrugged off as noise. Darcet, Oquab, Mairal, and Bojanowski’s Vision Transformers Need Registers (2024) explained what they are, and removed them with an almost embarrassingly small change.

The artifacts

Train a big ViT long enough — supervised or self-supervised, DeiT or CLIP or DINOv2, it doesn’t matter — and a handful of high-norm tokens appear. These are patch tokens whose feature vectors have grown enormous compared to their neighbors, and they tend to land on low-information, redundant patches. The paper’s reading: the model has quietly repurposed them. Instead of describing their own little square of background, those patches have been drafted to store global, image-wide information the network needs a place to compute with. The model, given nowhere to keep its scratch work, grabbed a few patches it judged expendable and used them as memory.

Why it matters

If all you want is one vector per image, this is harmless — classification still works, because the global information ends up somewhere. The damage is to dense features: the per-patch outputs you need for segmentation, depth, tracking, and correspondence. A few patches are now lying about what they contain, so the feature map has holes exactly where you’re trying to read it pixel by pixel. It also makes attention maps too messy to use for things like unsupervised object discovery — a task DINOv2 notably stumbled on, precisely because of these artifacts.

The fix: give it somewhere to write

The solution is to stop making the model steal background patches by handing it dedicated scratch space. Add a few extra learnable tokens — registers — to the input sequence, alongside the patch tokens and the CLS token. They correspond to no region of the image; they are pure workspace. The model learns to offload its global bookkeeping into the registers instead of hijacking a patch of sky. At the end you throw the registers away and keep the patch features, which are now clean. Attention maps smooth out, dense-task scores rise, and object discovery starts working again.

DINOv3 (Siméoni et al., 2025) bakes this in as standard practice — four registers — and reports that with them the patch norms stay stable across a million training iterations, instead of a few exploding.

The honest footnote

Registers are not a cure for everything that ails a dense feature map. The DINOv3 team found a separate degradation that registers don’t touch: over very long training, the similarity between the CLS token and the patch tokens slowly creeps up, eroding the locality that makes patch features useful. That one needed a different fix entirely. Two diseases, similar symptoms — worth remembering before you assume one patch solves them both.

Still, the register story is a satisfying one. The spikes weren’t a bug in the data or a flaw in attention. They were the model doing something reasonable — needing scratch space — in a destructive way, because nobody had given it any. Sometimes the fix isn’t a cleverer loss or more data. It’s noticing what the model is trying to do and getting out of its way. A few spare tokens.

References

• Darcet, T., Oquab, M., Mairal, J., & Bojanowski, P. (2024). Vision Transformers Need Registers. ICLR (oral).
• Siméoni, O., et al. (2025). DINOv3. arXiv technical report — adopts four registers and documents a second, distinct dense-feature degradation.