Running Out of Text

Scaling laws come with an asterisk that’s easy to skip past. The clean version — loss falls as a predictable power law in model size, data, and compute, so you can fit a few cheap runs and extrapolate to the expensive one — quietly assumes an endless river of fresh text, each token read exactly once. That assumption is running out. In her survey Scaling Laws, Carefully (2026), Lilian Weng calls the boundary the data wall: the point where the supply of high-quality unique tokens is exhausted and further scaling has to reuse what you already have.

So what happens when you show a model the same data twice?

Repetition is not free

The naive hope is that a repeated token is as good as a fresh one. It isn’t, and the ways it fails are specific.

• Hernandez et al. (2022) ran a controlled study — 90% unique tokens, 10% repeated — and watched test loss do something strange: double descent, with the repetition specifically damaging out-of-distribution performance and later fine-tuning. A small amount of repeated data poisoned more than its share.
• Muennighoff et al. (2023) turned this into a recipe, with roughly 400 experiments running up to 1,500 epochs. They model the effective amount of data as the unique tokens plus a discounted contribution from repeats, where each additional epoch is worth exponentially less than the last. The practical upshot inverts the usual advice: when data is the binding constraint rather than compute, spend your budget on more epochs over a smaller model, not on more parameters.
• Lovelace (2026) adds that bigger models are more sensitive to repetition — an explicit overfitting penalty that grows with the ratio of model size to unique data — though strong weight decay softens it.

Trust these even less

Here’s the part worth internalizing. Weng’s entire “carefully” thesis is that scaling-law extrapolations are fragile: they’re fit on small, cheap models and stretched orders of magnitude upward, so trivial choices — how you count parameters, whether you average or sum the loss, where you set the fit region — can swing the prediction. The famous Kaplan-versus-Chinchilla disagreement about how to spend compute turned out to hinge on exactly such a bookkeeping detail (non-embedding versus total parameter counts).

The data-constrained laws are shakier still. Weng stresses that they are pure empirical curve fits with no theory underneath — nobody has derived the repetition-discount exponents, they’re just measured. So they should be trusted even less than the single-epoch laws they extend. They tell you a wall is out there; they don’t pinpoint where it stands.

The boundary nobody put in the slogan

This is where the bitter lesson gets its footnote. “General methods that scale with compute win” was the qualitative claim; scaling laws made it quantitative; the data wall is the boundary condition the slogan never mentioned. Scale wins — with the correct compute allocation, and only while the unique data lasts. The models learned to read faster than we can write.

References

• Weng, L. (2026). Scaling Laws, Carefully. Lil’Log — the survey these findings are drawn from.
• Muennighoff, N., et al. (2023). Scaling Data-Constrained Language Models. NeurIPS.
• Hernandez, D., et al. (2022). Scaling Laws and Interpretability of Learning from Repeated Data. arXiv.