How Big Is Big Enough? Scaling Laws, Chinchilla, and the Measurement of AI

The Hook: A Chart That Steered an Industry

In January 2020, a group of researchers at OpenAI published a paper with the unassuming title Scaling Laws for Neural Language Models. It contained a series of charts that at first glance looked almost boring: straight lines on doubly logarithmic paper. Yet those straight lines would become some of the most consequential graphics in recent technological history. They described how good a language model gets when you make it larger, feed it more data, or pour more compute into it — and not vaguely, but as a precise mathematical relationship.

Before these charts, "make it bigger" was a gut feeling. People suspected that larger models would be better, but no one could say by how much. After these charts, "make it bigger" was a quantitative recipe with a measurable slope. You could predict, before even a single graphics processor was fired up: a model with this compute budget will reach roughly this prediction error. Funded training runs worth hundreds of millions of dollars were no longer built on hope, but on extrapolated curves.

And then, two years later, another lab — DeepMind — showed that nearly the entire industry had applied the recipe incorrectly. The largest models in the world, on whose training astronomical sums had been spent, were systematically undertrained. They had been built too large and with too little data. A model with less than a quarter of the parameters, but with substantially more training data, could beat the giants — at the same compute cost.

This article tells the story of neural scaling laws: what exactly they claim, why they work at all, how DeepMind's "Chinchilla" redrew the map, why even this famous result later came under scrutiny — and what all of this has to do with the heated debate over whether large models suddenly "emerge" new abilities or whether that is merely a deception of our measuring instruments. It is a story about how an empirical observation became an engineering law — and about how careful one must be with laws read off from curves.

Part 1: What a Scaling Law Actually Claims

The Three Knobs and the One Target

To understand what Kaplan and colleagues measured in 2020, we need four terms. Three of them are the knobs you can turn during training; the fourth is the target you want to optimize.

The first knob is N, a model's number of parameters — roughly, the number of adjustable weights in the neural network, a measure of its "size" or capacity. The second is D, the amount of training data, measured in tokens (word fragments). The third is C, the compute deployed, measured in FLOP (floating-point operations). These three are not independent: as a rule of thumb, training a transformer model costs about C ≈ 6 · N · D FLOP. Each parameter must be touched roughly six times per token (crudely: twice in the forward pass, four times in the backward pass). Compute is therefore essentially the product of model size and data volume.

The target is L, the loss or prediction error — specifically the so-called cross-entropy, a measure of how surprised the model is by each true next token. A low loss means the model predicts the text well. Loss can be expressed in "nats per token" or "bits per token" and is closely related to perplexity. Importantly, loss is a continuous, smooth quantity, not a "right/wrong." That fact will play a central role later.

Power Laws: Straight Lines in a Log-Log Plot

Kaplan and his co-authors trained over 200 transformer models spanning seven orders of magnitude in compute, four orders of magnitude in parameter count, and three orders of magnitude in data. And they found something remarkable: the loss follows a power law with respect to each of the three knobs. In formulas:

$$L(N) \propto N^{-\alpha}, \quad L(D) \propto D^{-\beta}, \quad L(C) \propto C^{-\gamma}$$

A power law is exactly what produces a straight line in a doubly logarithmic plot. Double the model size, and the loss drops by a fixed percentage — regardless of whether you go from one million to two million parameters or from one billion to two billion. This scale invariance is the fascinating part: across many orders of magnitude there is no "kink," no point at which growth suddenly stops or explodes. The improvement is smooth, continuous, and — this is the crux — predictable.

The exponents α, β, and γ are astonishingly small. In Kaplan's measurement the exponent for parameter count was about 0.076 (in a particular convention). This means: to halve the loss, you must make the model not twice, but many orders of magnitude larger. Scaling therefore delivers reliable, but dearly bought, progress. There is no free jump — every additional increment of quality costs exponentially more resources.

Three Regimes and the Decisive Question

Kaplan distinguished three regimes: sometimes the model is the bottleneck (too few parameters for the available data), sometimes the data volume (too little text for a huge model), sometimes compute. In each regime he observed the characteristic power-law decay.

The most practically important question, however, was: If I have a fixed compute budget C — how do I split it optimally between model size N and data volume D? Because of C ≈ 6ND, this is a trade-off: a larger model means, at the same budget, less training data, and vice versa. Here Kaplan's paper gave an answer that shaped the industry for the next two years — and that, as it would turn out, was wrong on a decisive point.

Kaplan's analysis suggested that with more budget one should above all grow the model, letting the data volume grow only sub-proportionally. The recommendation, in effect: put the money into parameters, not data. The result was an outright arms race toward ever more gigantic models — GPT-3 with 175 billion parameters, Gopher with 280 billion, Megatron-Turing NLG with 530 billion. All trained on, relative to their size, comparatively modest data volumes of a few hundred billion tokens.

Part 2: Chinchilla — The Correction That Shifted Everything

DeepMind's Counter-Experiment

In March 2022, a team led by Jordan Hoffmann at DeepMind published the paper Training Compute-Optimal Large Language Models. It would become the most famous correction in AI scaling research. The researchers trained over 400 models, from 70 million to over 16 billion parameters, on data volumes between 5 and 500 billion tokens. Unlike Kaplan, they systematically varied both axes and looked closely at which combination of N and D yielded the lowest loss for a given budget.

Their result contradicted Kaplan on a central point. Across three independent estimation methods, they found: model size and data volume should grow in lockstep. For every doubling of the parameter count, one should also double the number of training tokens. The optimal split is roughly N_opt ∝ C^0.46 and D_opt ∝ C^0.54 — both near 0.5, whereas Kaplan had assumed a substantially higher exponent for N (around 0.73).

Why the difference? One key reason was methodological: Kaplan had not cleanly adapted the learning-rate schedule to the respective data volume, which underestimated the role of data. DeepMind corrected this. That sounds technical, but it had enormous consequences.

The Parametric Formula

The most elegant part of the Chinchilla paper is a single equation describing loss as a function of model size and data volume:

$$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$$

This formula has a beautiful interpretation. The first term E is the irreducible loss — the entropy of natural language itself, a floor below which you cannot go even with infinite compute, because language has an inherent unpredictability. The second term penalizes a too-small model (finite capacity), the third a too-small data volume (finite training). The constants estimated by DeepMind are approximately: E ≈ 1.69, A ≈ 406, B ≈ 411, α ≈ 0.34, β ≈ 0.28. That A and B are nearly equal, and α and β nearly equal, is precisely the mathematical reason why N and D should scale in lockstep.

Chinchilla Beats Gopher

To prove their theory, the DeepMind researchers did something bold. They took the same compute budget with which the 280-billion-parameter model Gopher had previously been trained (on around 300 billion tokens) and invested it in a much smaller model with only 70 billion parameters — which they trained on 1.4 trillion tokens, almost five times the data. They called this model Chinchilla.

The result was unambiguous: Chinchilla beat Gopher across nearly the entire range of benchmarks — despite a quarter of the parameters. It was smaller, faster, and cheaper to operate, and yet better. The message was unmistakable: the entire generation of giant models was undertrained. Seduced by Kaplan's recommendation, people had put too much money into parameters and too little into data.

The famous rule of thumb that emerged from this paper: a compute-optimal model should be trained with roughly 20 tokens per parameter. A 70-billion model therefore needs around 1.4 trillion tokens. This number — "Chinchilla-optimal" — became the reference for years.

Part 3: When the Law Itself Is Audited

The Replication Debate of 2024

Scaling laws are empirical claims, and good science checks empirical claims. In April 2024, Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You of the research organization Epoch AI did exactly that. In their paper Chinchilla Scaling: A replication attempt, they tried to reproduce DeepMind's third estimation method — the fit of the parametric formula L(N,D) — using data reconstructed from the paper's plots.

They ran into inconsistencies. The estimates DeepMind reported for the third method did not quite match the first two methods of the same paper. Above all, the confidence intervals given were absurdly narrow — so narrow that producing them would have required over 600,000 experiments, whereas DeepMind had probably trained fewer than 500 models. Something was wrong with the statistical analysis.

The Resolution: An Optimizer Detail

The story has an illuminating ending. After the first version of the Epoch paper appeared, one of the DeepMind lead authors clarified the cause: in fitting their formula, they had averaged the so-called Huber loss values over the examples instead of summing them. This led to a very high loss scale in the optimizer used (L-BFGS-B), causing it to terminate too early — both in the original fit and in the bootstrapping, the procedure for estimating uncertainty. Hence the impossibly narrow intervals.

Notably, the bottom line: when Besiroglu and colleagues re-fit the parametric formula themselves, they arrived at values compatible with the first two estimation methods of Hoffmann et al. The central core claim — that N and D should scale roughly in lockstep — therefore held. What wobbled was not the law, but the overconfident error bars around some of its parameters. I am of the opinion that this is exactly what makes the episode so instructive: it shows that scaling laws describe robust regularities, but that the exact constants should be taken with caution — they depend on data reconstruction, optimizer settings, and fitting details.

The Second Correction: Inference Costs Too

Chinchilla optimized for a single quantity: the compute cost of training. But a model is trained once and then used billions of times. Each use (inference) also costs compute — and that cost is permanently lower for a smaller model.

In late 2023, Nikhil Sardana and colleagues drew the consequence in Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws. If you expect to use a model very often, it pays to train it smaller and longer than Chinchilla prescribes — you pay more at training, but save on each of the billions of requests. Their calculation: a developer expecting a Chinchilla-30B-quality model with very high demand (around 10⁹ requests) can reduce total FLOP by about 28% by instead training a 13.6-billion model on 2.84× the data.

This logic precisely explains Meta's model strategy. The Llama models were deliberately trained far beyond the Chinchilla optimum: Llama 1 with up to 142 tokens per parameter, Llama 2 with up to 284, and Llama 3 pushed this to an extreme 1,875 tokens per parameter (the 8-billion model on 15 trillion tokens). That is roughly a hundred times the 20-token rule. The reason is not ignorance of the Chinchilla insight, but its further development: for a model meant to run millions of times on end devices and in data centers, a small, "over-trained" model is the economically correct choice.

Part 4: Emergence — Real or Mirage?

The Exciting Claim

If scaling is as smooth and predictable as the curves suggest — where, then, does the feeling come from that large models suddenly do something qualitatively new? In 2022, Jason Wei and colleagues gave this feeling a name: emergent abilities. They defined an ability as emergent if it is not present in smaller models but is present in larger ones — in such a way that performance stays at chance level up to a certain model size and then rises sharply. As examples they cited multi-digit arithmetic, unscrambling letters, and question-answering tasks in various languages.

That was an exciting, almost unsettling claim. It suggested that scaling brings not merely gradual improvement but unpredictable phase transitions — that somewhere beyond a threshold, new abilities appear out of nowhere. For some this was evidence of astonishing potential; for others a safety nightmare: how do you control a system whose abilities you cannot predict?

The Objection: It's the Yardstick

In 2023, Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo of Stanford University put a large question mark next to this. Their paper Are Emergent Abilities of Large Language Models a Mirage? — recognized as one of the best papers at the NeurIPS 2023 conference — delivered a thesis as simple as it was revealing: emergence lies not in the model but in the choice of metric.

The heart of the argument: many emergence examples are measured with discontinuous metrics. Take "exact match" — a multi-digit addition counts as correct only if every digit is right. Or "multiple choice grade" — either the exactly correct option is chosen or not. Such all-or-nothing metrics punish any small improvement with zero points until the model solves the entire task flawlessly. The underlying ability — say, the probability of predicting the individual digit correctly — actually improves smoothly and continuously. But the hard metric translates this gentle curve into an apparent jump.

Schaeffer and colleagues showed: over 92% of the emergent abilities hand-annotated by Wei on the BIG-Bench benchmark appeared under exactly one of two metrics — Multiple Choice Grade or Exact String Match. Replace these with a continuous metric (say, the probability the model assigns to the correct answer), and the jump vanishes, revealing again the familiar smooth scaling. They could even produce the effect in reverse: by applying a sufficiently nonlinear metric to an image-recognition network, they made "emergence" appear artificially where there was none.

What Remains of Emergence?

The debate is not thereby settled once and for all, but it has sharpened the concepts. There is a difference between two statements: "The ability of the model changes abruptly" and "Our measurement of the ability changes abruptly." Schaeffer's work suggests that many famous emergence examples fall into the second category — they are artifacts of the yardstick, not of the model.

This does not entirely devalue the practical observation: if a user wants a correct, complete calculation, the all-or-nothing metric is indeed the relevant one. From a user's perspective the transition feels like a jump. But for the scientific question of whether something discontinuous happens inside the model, the answer is more sobering: much that looked like a phase transition is a smooth curve viewed through a coarse lens. I am of the opinion that the right lesson is this: before attributing supernatural jumps to a system, one should first examine one's own measuring instrument.

Part 5: Why Do Scaling Laws Work at All?

One unsatisfying aspect of scaling laws long remained: they are empirical. The straight lines were measured, but why nature obeys a power law was at first an open question. There are now serious theoretical explanatory approaches.

One influential family of explanations traces the power laws back to the geometry of the data. The idea, developed among others by Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma (published in the Proceedings of the National Academy of Sciences), distinguishes a "variance-limited" and a "resolution-limited" regime. In the resolution-limited case, the exponent of the scaling law can be directly related to the intrinsic dimension of the data manifold: a model interpolating a function on a d-dimensional data manifold improves at a rate that depends on d. Simplified: real-world data lie on a curved surface in a high-dimensional space, and the more finely a model "resolves" this surface, the better it gets — by a law that has exactly the observed power form.

Other approaches trace the power laws back to the statistical structure of language itself — for instance to the well-known Zipf distribution, according to which a few words are very frequent and very many words very rare. The distribution of the "concepts" a model must learn has a long tail of rare cases, and the rate at which a model captures this tail produces the power law. These explanations are not yet complete, but they turn the straight lines from a lucky accident into something one can understand from deeper principles.

It is worth drawing an honest boundary here: the theoretical grounding is an active field of research, and no single explanation is uncontested. What is empirically robust — the existence of the power laws over many orders of magnitude — is better established than any specific theory of why they hold.

Part 6: The Practical Meaning — A Compass, Not a Timetable

What does all this mean for someone who builds or deploys AI systems? First: scaling laws are a planning tool. Before launching an expensive training run, one can train small models, fit the curve, and extrapolate the large model's loss. You know in advance roughly what you get for your money. This has fundamentally changed the economics of AI development.

Second, the Chinchilla and the inference corrections shift the emphasis from "as many parameters as possible" to "the right ratio." The interesting question is no longer just how large a model is, but how many tokens per parameter it has seen — and for which use case (one-off research versus billions of uses) one optimizes.

Third — and this is perhaps the most important current concern — we are hitting a data wall. Chinchilla demands that the data volume grow with the model size. But the publicly available, high-quality text material on the internet is finite. Various estimates suggest that the stock of high-quality human text could become scarce for the largest training runs before this decade is out. This drives research toward synthetic data, toward multiple passes over the same data, and toward architectures that learn more per token. It is quite possible that the pure "more data, more parameters" era hits a natural limit and that the next advances must come from other sources — for instance from more compute at inference time (a model "thinking"), a trend already supplementing pure training-time scaling.

Scaling laws remain a compass: they reliably indicate the direction in which things get better, and how steep the road there is. But they are not a timetable guaranteeing that the road runs straight ahead forever. Every extrapolation beyond the measured range is a bet — a well-informed one, but a bet nonetheless.

A Comparison Table: The Evolution of Scaling Guidance

The Central Takeaway

The deepest lesson of scaling laws is not "bigger is better." It is subtler: progress in AI is, over long stretches, astonishingly regular — and this regularity can be measured before it is experienced. From this predictability grows a twofold relation of humility and caution. Humility, because the smooth curves show that many apparent "breakthroughs" are in truth expectable points on a known line. Caution, because the same curves tempt one to extrapolate them blindly — and because, as Chinchilla and the replication debate show, even carefully measured laws get corrected in their details.

For your own work this means, quite practically: when you hear a claim about "suddenly new abilities," first ask which yardstick was used to measure. And when you size a system, do not squint only at model size, but at the ratio of data to parameters and at the full lifecycle of training and inference. Scaling laws reward not the one who burns the most compute, but the one who understands their interplay most clearly.

A Question to Ponder

Scaling laws reliably tell us how a model's prediction error falls — but loss is not the same as "understanding," "usefulness," or "truth." When a quantity falls smoothly and predictably, yet we are not sure it measures what actually matters to us: do we then rely on a law — or on the convenience of finally having something measurable? And how would you decide which of the abilities you care about in an AI system can even meaningfully be plotted on a smooth curve?

Cross-References in the Vault

• The Ghost in the Machine: How to Read a Neural Network From the Inside – while scaling laws describe the external behavior of large models, mechanistic interpretability opens the black box from within.
• The Pyramid of Risk: How the EU AI Act Tames Artificial Intelligence – and Why It Concerns the Whole World – the AI Act's 10²⁵-FLOP threshold for "systemic risk" is ultimately a regulatory application of the very compute that scaling laws place at the center.
• The Predictive Brain: Predictive Processing and the Illusion of Perception – the brain, too, is modeled as a prediction machine minimizing its "prediction error"; an appealing comparison to a language model's loss.

Sources

• Kaplan, J. et al. (2020): Scaling Laws for Neural Language Models. arXiv:2001.08361. https://arxiv.org/abs/2001.08361
• Hoffmann, J. et al. (2022): Training Compute-Optimal Large Language Models (Chinchilla). arXiv:2203.15556 / NeurIPS 2022. https://arxiv.org/abs/2203.15556
• Besiroglu, T., Erdil, E., Barnett, M., You, J. (2024): Chinchilla Scaling: A replication attempt. arXiv:2404.10102 / Epoch AI. https://epoch.ai/publications/chinchilla-scaling-a-replication-attempt
• Wei, J. et al. (2022): Emergent Abilities of Large Language Models. arXiv:2206.07682. https://arxiv.org/abs/2206.07682
• Schaeffer, R., Miranda, B., Koyejo, S. (2023): Are Emergent Abilities of Large Language Models a Mirage? NeurIPS 2023 / arXiv:2304.15004. https://arxiv.org/abs/2304.15004
• Sardana, N. et al. (2023/2024): Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws. arXiv:2401.00448. https://arxiv.org/abs/2401.00448
• Bahri, Y., Dyer, E., Kaplan, J., Lee, J., Sharma, U. (2024): Explaining Neural Scaling Laws. PNAS. https://www.pnas.org/doi/10.1073/pnas.2311878121

← All articles