TECH NEWS – The owner of Tesla, SpaceX and Twitter has revealed how many GPUs he’s using to train the future version of Grok, which will be preceded by an earlier model that’s essentially just around the corner.

There’s a fierce race going on between AI-focused companies these days: everyone is trying to beat the others to release a new model of their LLM (large language model). xAI is part of this fight. Elon Musk’s company has already announced Grok, an AI assistant available to Twitter subscribers. Version 2 of this is coming soon.

Grok 3 end of year after training on 100k H100s should be really something special

— Elon Musk (@elonmusk) July 1, 2024

Grok 2.0 will be released by August, but Musk has already mentioned Grok 3 in his tweets. According to him, this version will be really special, as it will be trained on 100,000 Nvidia H100 GPUs and could even appear by the end of the year. This hardware and the amount of it could bring unprecedented results. Rumor has it that OpenAI’s LLM, GPT-4, was trained on only 40,000 Nvidia A100 AI GPUs. The A here is not the Ada Lovelace architecture, but the one before it, Ampere! And H is for Hopper, the consumer version of Ada Lovelace (GeForce RTX 4000 graphics cards). More modern hardware, and more of it: OpenAI may have a serious competitor.

In an earlier tweet, Musk claimed that coaching LLMs over the Internet is hectic and “takes a lot of work”. The Grok 2 model will make improvements in this regard, and Grok 3 builds on the first two versions, so it will be interesting to see what xAI comes up with. Musk has already said that he also wants to include Nvidia’s currently most powerful AI accelerator, the Blackwell B200 (the architecture that will debut in the GeForce RTX 5000 cards). Except that the H100 kit cost $3 billion, and he wants to buy $9 billion worth of the B200.

So Musk is putting all his eggs in one basket…

Source: WCCFTech