David Meyer

<p>Although companies like OpenAI and <a href="https://fortune.com/company/alphabet/" target="_blank">Google</a> don’t disclose the precise costs of training AI models like GPT-4 and Gemini, it’s clearly a fiendishly expensive business—and the bigger and more capable these so-called frontier models get, the more it costs to train them.</p><div class="paywall-selector"></div><div class="paywall">



<p>When OpenAI released GPT-3 in 2020, cloud provider Lambda <a href="https://lambdalabs.com/blog/demystifying-gpt-3">suggested</a> the model—which had 175 million parameters—cost over $4.6 million to train. OpenAI hasn’t disclosed the size of GPT-4, which it released a year ago, but reports range from <a href="https://www.semafor.com/article/03/24/2023/the-secret-history-of-elon-musk-sam-altman-and-openai">1 trillion</a> to <a href="https://the-decoder.com/gpt-4-architecture-datasets-costs-and-more-leaked/">1.8 trillion</a> parameters, and CEO Sam Altman vaguely pegged the training cost at “<a href="https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/">more than</a>” $100 million. Anthropic CEO Dario Amodei <a href="https://twitter.com/sarahdingwang/status/1694415161837453534">suggested</a> in August that models costing over $1 billion would appear this year and that “by 2025 we may have a $10 billion model.”</p>



<p>Is this kind of exponential cost growth realistic? It’s certainly a real trend, say researchers, but whether it can be sustained is another matter.</p>



<p>In 2022, researchers in the U.K., the U.S., Germany, and Spain <a href="https://arxiv.org/pdf/2202.05924.pdf">found</a> that, since the deep learning field took off in the early 2010s, the amount of computational power needed to train the most capable new models doubled roughly every six months. According to Epoch AI director Jaime Sevilla, who was the lead author on that paper, the trajectory has held since then, with the cost of training roughly tripling each year—the 4X growth in compute requirements is offset by a 1.3X increase in efficiency.</p>



<p>“It’s still a straight line, and it keeps pointing up,” says Sevilla.</p>



<p>Here’s Epoch AI’s projection of the hardware cost involved in training the most expensive AI models, through 2030. This excludes AI researchers’ salaries, which are considerable these days. There is also huge uncertainty—as is clear from the vast range in Epoch’s estimations—about the exact trajectory. This results from how little is publicly known about the size and cost of models like GPT-4 and the statistical effects of disparate estimates of the exact investment growth rate, which compounds the more years out one tries to project. Sevilla describes the median forecast—the line that hits $140 billion by 2030—as “a naive extrapolation based on historical data, rather than an all-things-considered forecast.” </p>



<figure class="wp-block-embed is-type-rich is-provider-datawrapper wp-block-embed-datawrapper"><div class="wp-block-embed__wrapper">
<iframe title="Hardware cost of training frontier machine learning models" aria-label="Interactive line chart" id="datawrapper-chart-2GjGa" src="https://datawrapper.dwcdn.net/2GjGa/3/" scrolling="no" frameborder="0" style="width: 0; min-width: 100% !important; border: none;" height="500" data-external="1"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(a){if(void 0!==a.data["datawrapper-height"]){var e=document.querySelectorAll("iframe");for(var t in a.data["datawrapper-height"])for(var r=0;r<e.length;r++)if(e[r].contentWindow===a.source){var i=a.data["datawrapper-height"][t]+"px";e[r].style.height=i}}}))}();</script>
</div></figure>



<p>And here come the additional caveats, apart from that uncertainty. The first and most obvious is that, if this trend continues, the cost of training relative to the capabilities that are gained will at some point become too much for any company to bear.</p>



<p>GPT-3 was more accurate in its output than GPT-2, to the point that it was able to power GitHub’s Copilot code-generator. GPT-3.5, boosted by processes that again required additional training and further computing resources, was convincing enough to provide the foundation for ChatGPT’s first release. GPT-4 added multimodality—the ability to also accept images as inputs—to the mix, along with better reasoning abilities and a much better understanding of the context of users’ prompts. It’s not yet clear how great future leaps between versions will prove to be.</p>



<p>Lennart Heim, another coauthor of the 2022 paper who leads compute governance work at the Centre for the Governance of AI, <a href="https://blog.heim.xyz/this-cant-go-on-compute-training-costs/">warned</a> last year that the hardware cost of training one new frontier AI model could theoretically surpass the entire gross domestic product of the U.S. around the mid-2030s. “At some point you will hit the limits,” says Heim. “To spend 1% of GDP on a single training run, the capabilities need to be quite good.”</p>



<p>The amount of available training data may be a limiting factor, although Heim points out there could be ways around this, such as training on more kinds of data and showing the models the same data multiple times. Copyright lawsuits could factor into the equation here, if they stop the likes of OpenAI simply hoovering up every piece of data they can find online. But even if that happens, there’s always the option of synthetic data—for example, video data generated by the Unity or Unreal gaming engines. It’s also likely that companies will increasingly <a href="https://variety.com/vip/why-generative-ai-companies-will-pay-content-owners-and-licensing-models-that-will-emerge-1235944577/">license private data</a> for AI training purposes.</p>



<p>Another wild card lies in the practicality of expanding data centers, and the associated increase in energy and water consumption. Hyperscalers like <a href="https://fortune.com/company/microsoft/" target="_blank">Microsoft</a> (which is <a href="https://fortune.com/2024/04/02/microsoft-openai-stargate-100-billion-ai-supercomputer-star-wars-sdi/">reportedly contemplating</a> a $100 billion data center project with OpenAI called “Stargate”) are starting to look at attaching new energy sources like small modular nuclear reactors to their data centers, and work is underway to find <a href="https://www.ft.com/content/ba0a5ba0-980a-472f-a8d1-77f2602060f2">less energy-intensive alternatives</a> to existing AI infrastructure. But pushback against major new data centers has been growing everywhere from Ireland to West Virginia in recent years, and the cascading requirements of new AI models are only <a href="https://www.ft.com/content/d38ebd3e-84a2-4c22-b84c-6528a1e2dd3b">amplifying</a> that resistance. </p>



<p>And if the industry really does manage to create artificial general intelligence (AGI), with all the impacts that would have on employment and societal power imbalances, a whole new level of political resistance could manifest—though on the other hand, the advent of AGI could bring benefits that are game-changing enough to justify training costs that today seem outlandish. </p>



<p>However, not all AI is about the march toward AGI, with the massively large model sizes that would likely involve. If one looks at specific use cases—such as turning natural-language requests into SQL database queries—then smaller, fine-tuned models can outperform a beast like GPT-4 despite having only single-digit billions of parameters, argues Philipp Schmid, a technical lead at AI firm Hugging Face. </p>



<p>“We believe in the future … we will have closed-source, big, strong foundation models, but a lot of companies will adopt multiple small models for specific use cases,” Schmid said, adding that an open-source approach—as has been done by players like Meta and Mistral—would also help to address cost issues. </p>



<p>“If we collaboratively build on the work from each other, we can reuse resources and money spent,” he said.</p></div>

If current trends continue, new AI iterations would have to bring astounding benefits to justify the costs involved in training them, experts say.

The cost of training AI could soon become too much to bear

Coins on a laptop computer with a robotic hand.

<br><br><b>Who will be next to lead the Fortune 500?</b> See who made our list of the <a href="https://fortune.com/ranking/most-powerful-rising-executives/?&itm_source=fortune&itm_medium=article_tout&itm_campaign=MPRE">25 Most Powerful Rising Executives.</a><br><br><br><br><br><br>

P-Reported

A.I.

Tech

If current trends continue, new AI iterations would have to bring astounding benefits to justify the costs involved in training them.

Why the cost of training AI could soon become too much to bear

The cost of training AI could soon become too much to bear | Fortune

Brad Smith, Vice Chair and President of Microsoft

Microsoft president says AI is ‘the electricity of our age’ as company prepares to hit $80 billion spend

/2025/01/03/microsoft-president-ai-electricity-of-our-age-80-billion-spend/

Lionel Lim

Malaysia wants Big Tech to get licensed as Asia Pacific cracks down on social media platforms

/asia/2025/01/03/malaysia-social-media-license-google-x-meta-bytedance-tencent-telegram-australia-korea-vietnam/

Allie Garfinkle

crystal ball with the capital, bitcoin, stock charts, and money symbols plus robot hand

The Crystal Ball: Envisioning how AI will shape our world in 2025

/2025/01/03/the-crystal-ball-envisioning-how-ai-will-shape-our-world-in-2025/

Alexei Oreskovic

Mark Zuckerberg’s new minister of MAGA: Goodbye Mr Clegg, Hello Joel Kaplan

/2025/01/03/mark-zuckerbergs-new-minister-of-maga-goodbye-mr-clegg-hello-joel-kaplan/

Ryan Hogg

A passenger carries the folding bike classic Brompton on the platform at Munich Central Station.

Iconic Brompton bike maker suffers 99% profit decline amid post-COVID bike industry turmoil

/europe/2025/01/03/iconic-brompton-bike-maker-suffers-99-profit-decline-amid-post-covid-bike-industry-turmoil/

Norway accelerates towards full electrification as 89% of new cars go electric

/europe/2025/01/03/norway-full-electrification-89-new-cars/

Paolo Confino

Airline passengers who found bedbugs infesting their seats were allegedly rebuffed by flight crew

https://fortune.com/2025/01/02/airline-passengers-who-found-bedbugs-infesting-their-seats-were-allegedly-rebuffed-by-flight-crew/?itm_source=parsely-api

Brooke Seipel

Park City's wealthiest guests are furious after $20,000 ski trips became a disaster, and now investors are bailing

https://fortune.com/2025/01/03/park-city-utah-skiing-ski-patrol-strikes-long-lines-explained-vail-resorts/?itm_source=parsely-api

Amanda Gerut

Loom’s founder reveals painful insecurities after selling a startup for $1 billion: 'I am rich and I have no idea what to do with my life'

https://fortune.com/2025/01/03/looms-founder-startup-sale-identity-crisis-doge-elon-hawaii/?itm_source=parsely-api

Alena Botros

Carvana father-son duo lose hundreds of millions of dollars in net worth after short-seller report

https://fortune.com/2025/01/03/carvana-father-son-lose-millions-net-worth-short-seller-report/?itm_source=parsely-api

Lindsey Leake

Norovirus rates have skyrocketed by 340% this season. Here’s where the ‘winter vomiting disease’ is spreading and why

https://fortune.com/well/article/norovirus-winter-surge-2024-2025-cases-symptoms-vaccine/?itm_source=parsely-api

Orianna Rosa Royle

Gen Z and millennials aren’t buying lab-grown diamonds because they’re ‘sustainable’ says Pandora CEO, they only care about price and design

https://fortune.com/europe/2024/12/26/gen-z-millennials-lab-grown-diamonds-sustainable-price-bigger-ring-design/?itm_source=parsely-api