Just Predict the Clean Image
A diffusion model makes pictures by denoising: start from pure static and, step by step, remove noise until an image emerges. There’s a design choice buried in that loop that most people treat as a footnote — what, exactly, should the network output at each step? A new paper from Tianhong Li and Kaiming He, Back to Basics: Let Denoising Generative Models Denoise (2025), argues the footnote was load-bearing all along.
Three targets that look interchangeable
At any step you have a noisy image, a blend of the clean picture x and some noise ε. You can train the network to predict one of three things: the clean image x, the noise ε, or the “velocity” v = x − ε. Given the noisy input, any one of these determines the other two by simple algebra, and a loss written in one form is just a reweighting of a loss in another. So the textbook conclusion has been that the choice is cosmetic — and since the DDPM era, the field mostly settled on predicting the noise.
Why it stops being cosmetic
The equivalence quietly assumes the network can hit any of the three targets equally well. Li and He point at the manifold assumption to argue it can’t. Real images occupy a thin, low-dimensional sheet inside the enormous space of possible pixel values — most random pixel grids look like nothing. Clean images live on that sheet. Noise, and therefore velocity, is spread across the whole high-dimensional space, off the sheet.
Asking a network to output a point on a low-dimensional manifold is a well-posed request. Asking it to output an arbitrary high-dimensional noise field — when each patch it sees is raw pixels, thousands of dimensions wide — is a much harder one. A toy experiment makes the point cleanly: embed 2-D data in D dimensions and only x-prediction recovers the structure as D grows; noise- and velocity-prediction fall apart by D = 512.
The payoff: throw everything else away
Once you predict the clean image, the rest of the machinery becomes optional. The authors’ model, JiT (“Just image Transformers”), is a plain vision transformer run directly on raw pixel patches — no VAE or tokenizer to compress the image first, no self-supervised pretraining, no perceptual loss borrowed from a pretrained network. On ImageNet it reaches an FID of 1.82 at two billion parameters. And the prediction target really is what carries it: in a controlled comparison at a fixed model size, keeping everything else the same and swapping the clean-image target for noise or velocity doesn’t merely do worse — it collapses, with FID leaping from single digits to somewhere between 96 and 395.
There’s a pleasing symmetry here. The same senior author’s Masked Autoencoders took the same stance — a plain ViT, straight on pixels, nothing fancy — but for learning representations. JiT is that minimalism pointed at generation instead. Progress by subtraction.
The open argument
This isn’t settled. A competing line of work (REPA, RAE) claims that to make high-dimensional pixel diffusion train well you do need to align it to a pretrained representation — they lean on DINOv2 features — and they post even lower FID (down to ~1.13). JiT’s rebuttal is that the alignment is unnecessary once the prediction target is right. Both results stand; the field hasn’t decided which is the better trade once you count the cost of that extra pretraining.
Either way, the lesson is worth keeping: the knob everyone had labeled “doesn’t matter” turned out to decide whether the simplest possible architecture works at all. It’s worth occasionally re-checking the assumptions we inherited as settled.
References
- Li, T., & He, K. (2025). Back to Basics: Let Denoising Generative Models Denoise (JiT). arXiv:2511.13720.
- Peebles, W., & Xie, S. (2023). Scalable Diffusion Models with Transformers (DiT) — the transformer-backbone diffusion model JiT builds on.
- He, K., et al. (2022). Masked Autoencoders Are Scalable Vision Learners — the same plain-ViT-on-pixels stance, for representation learning.