Reading room
Small-scale pretraining study finds larger efficiency gains from data than model recipes
Experiments comparing public model recipes and datasets from 2019 through 2025 find that better data delivered larger pretraining compute-efficiency gains than model changes. At the highest tested budget, the authors estimate gains of 12 times from data and 3.7 times from model recipes. The result highlights data engineering's economic significance, but the study uses small models and relatively easy benchmarks. It does not establish which improvements matter most at frontier scale or during reinforcement learning.
Editorial summary · Publisher article
What matters
- The experiments combine different model recipes and corpora at budgets up to 10 to the power of 19 FLOPs.
- Evaluation uses OLMES, which aggregates 10 relatively easy benchmarks, mostly multiple-choice questions.
- Additive model and data effects explain 88% of score variance in the tested comparison grid.
- Synthetic data, tokenizer improvements and inference-efficiency gains are outside the study's scope.
Why scale could change the result
The authors argue that architectural research also makes larger training runs feasible by improving stability and resource use. Such benefits can be missed by small-scale efficiency comparisons. They also acknowledge benchmark dependence, limited hyperparameter tuning and extrapolation uncertainty.
Why this made the edition
Original experiments include controlled comparisons, compute budgets, evaluation methods and extensive limitations. The findings inform AI research economics without establishing equivalent effects at frontier scale.
Source material