Nvidia challenger Groq just raised $640 million for its AI chips. Its college dropout CEO says a viral moment was ‘a game changer’

But it was a viral moment in February 2024 that set the stage for today’s massive funding round. It was “100% a game changer,” Ross told Fortune in a recent interview. 

Unlike Nvidia GPUs, which are used for both training today’s most sophisticated AI models as well as powering the model output (a process known as “inference”), Groq’s AI chips are strictly focused on improving the speed of inference—that is, providing remarkably fast text output for large language models (LLMs), at a far lower cost than Nvidia GPUs.

But Groq has struggled with how to show potential users the power of its chips. The answer, it turned out, was for Groq create its own ChatGPT-like experience. 

In February, Groq set up its own conversational chatbot on its website that it said broke speed records for LLM output on open-source models including Meta’s Llama. Then a developer posted a short video on X showing how Groq, powering an LLM from Paris-based startup Mistral, could provide answers to questions with hundreds of words in less than a second. Ross said the company’s fortunes immediately changed—there were suddenly thousands of developers clamoring to build their AI tools using Groq’s powerful AI chips. Just six months later, there are now 300,000 developers accessing Groq’s solutions and hardware through its AI cloud service. 

AI chips in the cloud

At the moment, though, the vast majority of those developers are using Groq’s services for free—so it remains to be seen how Groq, which currently has 200 employees, plans to become profitable. “We’ve only made paid access available to little over 30 customers at the moment,” said Ross, because of limited capacity. While he says he expects to have some “pretty good revenue” this year, “one of the benefits of being a private company is we don’t have to talk about our revenue.” 

While it might seem easy to question Groq’s long-term prospects with so little insight into revenue, Ross has a long history of surpassing expectations. After dropping out of high school because he “was bored,” he learned computer programming and, after a stint at Hunter College, managed to get into NYU. There, he took PhD classes as an undergraduate for two years and then, once again, dropped out. “I didn’t want to cap my earning opportunity by graduating from something,” he joked. That led to a job at Google, where he helped invent Google’s AI chip, called the TPU, before leaving to launch Groq in 2016. 

Ross says Groq has no intention of being a startup that lives off of VC funding rather than having a sustainable business. “You’ve got Sam Altman saying he doesn’t care how much money he loses,” he said. “We actually intend to recoup our investment with this money that we’ve raised, so we will actually get every dollar back on the hardware that we deploy.” Groq was able to raise over half a billion dollars, he explained, because “we have more demand than we can possibly satisfy.” The investment will allow the company to build out more hardware and charge customers who are eager for higher rate limits. 

Groq is not the only AI chip startup looking to challenge Nvidia: Cerebras, for example, recently filed confidentially for an IPO, while SambaNova, Etched, and Fractile are also in the mix. And of course, established GPU chipmakers like AMD are ramping up their AI efforts. But analyst Daniel Newman recently told Fortune that there is “no natural predator to Nvidia in the wild right now.” 

That said, even if Groq can only nibble a tiny portion of Nvidia’s pie, it will provide plenty of business. “I don’t know if Nvidia will notice how much of the pie we eat, but we will feel quite full off of it,” said Ross. “It’ll be a huge multiple in terms of our valuation going forward.”