Computational Power and AI - AI Now Institute

This is a very broad and reasonably deep description of how compute power is used to build and train models, and why this is hard and very expensive.

Compute costs are predictably large: the final training run of GPT-3 is estimated to have cost somewhere between $500,000 to $4.6 million. Training GPT-4 may have cost in the vicinity of $50 million, but the overall training cost is probably more than $100 million because compute is required for trial and error before the final training run.

Those costs are huge, and it is a ton of power too. It is interesting that the big innovations over the last few years — crypto and AI — are both very high compute demand workloads. Crypto has an answer with proof-of-stake. It isn't clear what will make AI costs go down for model building.