Description
Gopher is a 280-billion-parameter transformer language model from Google DeepMind, the largest in a family of models the team trained ranging from 44 million to 280 billion parameters. It was introduced in December 2021 alongside two companion papers: one on the ethical and social risks of language models, and one proposing RETRO, a retrieval-enhanced training architecture.
Read More
DeepMind trained the model family to study where increasing scale helps language model performance and where it does not. Gopher advanced significantly toward human expert performance on the Massive Multitask Language Understanding (MMLU) benchmark compared with prior work, and the research showed gains from scale on tasks such as reading comprehension, fact-checking and identifying toxic language. Scale did not meaningfully improve performance on logical reasoning and common-sense tasks, however.
Beyond benchmark scores, DeepMind explored Gopher through direct interaction. When prompted in a dialogue format, the model could sometimes discuss topics such as cell biology with surprising coherence and a correct citation, despite having no dialogue-specific fine-tuning. The research also documented failure modes that persisted across model sizes: a tendency to repeat itself, reflect stereotypical biases, and state incorrect information with unwarranted confidence.
The release's companion paper built a taxonomy of ethical and social risks from language models, covering six thematic areas and 21 risks in depth, intended as a foundation for assessing and mitigating these issues. The second companion paper introduced RETRO, which queries passages of text during generation, reducing training energy cost and letting outputs be traced back to source passages, while matching a much larger standard transformer's performance with far fewer parameters.
Gopher is aimed at AI researchers studying language model scaling, evaluation and safety. It is a research release documented through DeepMind's blog post and papers rather than a product with a public API or downloadable weights, and DeepMind later used Gopher's training compute cost as the reference point for developing its more data-efficient successor, Chinchilla.
- CategoryAI Models & LLMs
- DeploymentCloud
- PlatformsWeb
Gopher Highlights
280-billion-parameter transformer language model, part of a family from 44 million to 280 billion parameters
Advances performance on the Massive Multitask Language Understanding (MMLU) benchmark
Evaluated across benchmarks including reading comprehension, fact-checking and toxic-language identification
Can sustain coherent, cited dialogue on technical topics without dialogue-specific fine-tuning
Documented failure modes across model sizes: repetition, stereotypical bias and confidently stated misinformation
Released with a taxonomy of ethical and social risks in language models, covering 21 risks in six themes
Companion to RETRO, a retrieval-enhanced architecture with lower training energy cost
Getting Started with Gopher
Read the release post: Start with DeepMind's blog post introducing Gopher and its two companion papers.
Review the benchmark results: Check Gopher's scores on MMLU, reading comprehension and toxic-language tasks against smaller models in the family.
Read the risk taxonomy: Review the companion paper on ethical and social risks to understand documented failure modes.
Compare with RETRO: Read about the retrieval-enhanced architecture released alongside Gopher.
Gopher's Use Cases
- Model scale research
- Language model risk assessment
- Retrieval-augmented modeling
