Description
Chinchilla is a 70-billion-parameter language model from Google DeepMind, introduced in an April 2022 paper, "An empirical analysis of compute-optimal large language model training." It was built to test a specific finding: that many large language models of the time were oversized for their training compute budget and undertrained on data.
Read More
DeepMind trained models of varying sizes and token counts to empirically estimate the optimal tradeoff between model size and training data for a fixed compute budget. The research found that, for the compute used to train Gopher (280 billion parameters), a model 4 times smaller trained on 4 times more data would have performed better. Chinchilla, at 70 billion parameters trained on 1.3 trillion tokens, tests this directly: it uses the same training compute as Gopher but far fewer parameters.
The results showed Chinchilla outperforming Gopher, GPT-3 and Megatron-Turing NLG on nearly every benchmark measured, including TriviaQA for question answering, HellaSwag, PIQA, Winogrande and BoolQ for common-sense reasoning, LAMBADA for reading comprehension, and MMLU for general knowledge, despite having 70 billion parameters against Gopher's 280 billion. Because Chinchilla is smaller, it is also cheaper and faster to run at inference time than Gopher, even though both models cost the same to train. DeepMind later noted that its scaling method predicts an optimal model of 140 billion parameters trained on 3 trillion tokens for the compute budget used to train PaLM, a 540-billion-parameter model released after Chinchilla.
The paper frames this as a broader lesson for the field: the prevailing approach at the time had been to keep training data roughly fixed at around 300 billion tokens while increasing parameter counts, a trend visible in models like Gopher and the 530-billion-parameter Megatron-Turing NLG. DeepMind's results argue that training compute should instead be split more evenly between model size and data volume, since training cost for transformers is driven by both factors together, not by parameter count alone.
Chinchilla is aimed at AI researchers and practitioners planning large language model training runs who need to weigh model size against training data volume for a given compute budget, rather than at end users looking for a chat product. It is documented through DeepMind's release post and paper rather than offered as a product with a public API or downloadable weights, and its data scaling approach has since informed how later large language models balance size against training data.
- CategoryAI Models & LLMs
Chinchilla Highlights
70-billion-parameter transformer trained on 1.3 trillion tokens
Trained with the same compute budget as DeepMind's larger, 280-billion-parameter Gopher model
Outperforms Gopher, GPT-3 and Megatron-Turing NLG on most benchmarks tested
Evaluated on question answering (TriviaQA), common-sense reasoning (HellaSwag, PIQA, Winogrande, BoolQ), reading comprehension (LAMBADA) and general knowledge (MMLU)
Lower inference time and memory cost than a larger model trained with equivalent compute
Introduces an empirical method for predicting compute-optimal model size and token count
Getting Started with Chinchilla
Read the release post: Start with DeepMind's paper on compute-optimal large language model training.
Review the scaling method: Study how DeepMind estimated the optimal tradeoff between model size and token count.
Compare benchmark results: Check Chinchilla's scores against Gopher, GPT-3 and Megatron-Turing NLG.
Apply the scaling law: Use the compute-optimal method to estimate model size and token count for a given training budget.
Chinchilla's Use Cases
- Compute-optimal training
- Model benchmarking
- Efficient inference
