Technical Challenges to Scale Beyond GPT4 to 100K H100s
Up until late 2024, no one has been able to massively increase the amount of compute dedicated to a single model beyond the OpenAI GPT 4 model level. This information is from semianalysis and EIA. Google’s Gemini Ultra, Nvidia Nemotron 340B, and Meta LLAMA 3 405B had similar or slightly more compute than GPT-4, but ...
Link :
https://www.nextbigfuture.com/2024/12/technical-challenges-to-scale-beyond-gpt4-to-100k-h100s.html
Link :
https://www.nextbigfuture.com/2024/12/technical-challenges-to-scale-beyond-gpt4-to-100k-h100s.html
){kind=link}
