Description
This comprehensive report from Epoch AI delves into the evolving dollar training costs of machine learning (ML) systems, analyzing data from 124 publicly known ML systems between 2009 and 2022. The primary finding indicates a substantial growth rate of approximately 0.49 orders of magnitude (OOM) per year in the compute cost for final training runs.
Read More
Two primary estimation methods were employed: Method 1, which uses the overall GPU price-performance trend, and Method 2, which utilizes the price-performance of actual NVIDIA GPUs used in training. Method 1, analyzing 124 systems, estimated a growth rate of 0.51 OOMs/year. Method 2, with a smaller sample of 48 systems, yielded a slightly lower growth rate of 0.44 OOMs/year. For "large-scale" systems, which represent those using a disproportionately large amount of compute, the growth rate was found to be significantly slower, at approximately 0.2 OOMs/year, suggesting a potential slowdown in the cost escalation for the most advanced AI models.
The report also explores the implications of these trends, including forecasts for when the cost of compute for the most expensive training runs might reach significant economic thresholds, such as 1% of US GDP. Naive extrapolation suggests this could occur by 2032, while more adjusted forecasts, considering a potential slowdown, point to 2040. The study highlights the importance of understanding these cost trends for forecasting AI progress, identifying potential developers of transformative AI, and informing broader economic and investment strategies within the AI landscape.
Key caveats are noted, including the inherent uncertainty in cost estimates, the focus solely on compute for final training runs, and the distinction between internal tech company costs and consumer prices. The research also touches upon the energy costs associated with training, finding them to be a smaller fraction compared to hardware costs, and discusses various methods for estimating these costs, emphasizing the need for robust data and careful consideration of assumptions.
- CategoryOther AI Tools
Highlights
Analysis of dollar training costs for ML systems (2009-2022)
Estimated annual growth rate of compute cost: ~0.49 OOM/year
Separate analysis for 'all systems' and 'large-scale' systems
Comparison of two estimation methods: overall GPU trend vs. actual hardware
Forecasts for future training costs and economic thresholds
Discussion of uncertainties and limitations in cost estimation
Examination of energy costs relative to hardware costs
Dataset of 124 milestone ML systems
Insights into the economics of AI development and investment
Use It For
- AI Cost Analysis
- AI Investment Forecasting
- AI Research Planning
- Economic Impact Assessment
- Competitive Landscape
