TECH NEWS – The owner of Twitter says it’s not clear if we can generate enough energy to keep artificial intelligence progressing at the rate it’s improving.
Training large language models (LLMs) requires many GPUs. This is mainly provided by Nvidia, but the company cannot produce enough to satisfy everyone. However, Musk says there is another problem: power requirements. He thinks that Grok 3, the next-generation model (developed by his startup xAI), will need about 100,000 Nvidia H100 GPUs for training. That in itself is not an easy (and certainly not cheap) task.
An H100 GPU consumes up to 700W. Multiply that by 100,000 and you’re up to 70 megawatts, and while there’s no way each GPU is always running at maximum load, the additional hardware and infrastructure will add additional power requirements, so it could be over 100 megawatts. That is the consumption of a small city. Paris had 500 megawatts of data centers in 2022. And 100 megawatts for just one LLM is a big problem. In an interview with Nicolai Tangen (CEO of a Norwegian wealth fund), Musk said that while the lack of GPUs will continue to be a major obstacle to the development of AI models, the lack of an adequate power grid and the power it provides will become a major obstacle.
Musk also made an exaggerated prediction: he believes that AGI (Artificial General Intelligence) will be smarter than the most intelligent human in 1-2 years. In 2017, he claimed we were two years away from self-driving cars so reliable we could sleep in them; that hasn’t happened yet. On March 19, 2020, he said there wouldn’t be a single case of COVID in the U.S. by the end of April… again, he was significantly wrong.
The Grok 2, the current model, settled for 20,000 H100 GPUs, so it will have five times the system requirements of the next generation yet to be built. That doesn’t sound very good.
Leave a Reply