When the committee called in 2024, the 77-year-old computer scientist ran a quick calculation in his head. What are the odds, he asked himself, that a theoretical psychologist hiding in computer science gets the Nobel Prize in physics? “Well, maybe one in two million,” he told the crowd at the Sana AI Summit in New York last week. Then again: what are the odds you dream about winning it? “Maybe one in two million … So that means it’s a million times more likely it’s a dream than reality.”

The audience laughed. Hinton wasn’t done.

For several days after the announcement, he said, he half-expected to wake up. The one thing that consoled him: “If it was a dream, I would wake up, and that nightmare about Trump being president wouldn’t be true.” A beat. The audience laughed as Hinton added, “I’d give it up for that.”

It’s the kind of remark that lands differently when the man delivering it also believes there’s a 10% to 20% chance that AI causes human extinction within 30 years — and that AI will surpass human intelligence within his remaining lifetime.

‘Much more intelligent’

Hinton was in conversation with Joel Hellermark, the 29-year-old founder and CEO of Sana AI, who had promised him a Nobel Prize the last time they appeared on this stage. This time Hellermark had harder questions. And Hinton — in his characteristically unhurried, occasionally hilarious way — gave answers that were by turns technically precise and cosmically vertiginous.

The through-line: not only are we building beings, they are going to be much smarter than us. And we are running out of time to decide what kind of beings they should be.

“I think it’s going to get much more intelligent than us — that’s my guess,” Hinton said. Nobody will ever beat them at Go or at chess again, he predicted, and just look at what it’s doing in math.

Smarter than Einstein

He wasn’t speaking abstractly. That morning — to his evident delight — an AI had proved one of Paul Erdős’ mathematical theorems using a branch of mathematics nobody had thought to apply. To Hinton, it was a landmark. In closed systems like mathematics, he explained, AI can generate its own conjectures, test them, learn from failures, and compound endlessly — the same way AI-powered AlphaGo went from mimicking expert moves to obliterating them the moment it started generating its own training data.

Language models, he argued, are on the same trajectory. The key insight: give a model some beliefs, let it reason to a new conclusion that contradicts something it already believes, and you have an inconsistency — which is a training signal that requires no new data to exploit. “I think that means these language models can get hugely smarter without a lot more data,” he said, noting that his fellow Nobel laureate Demis Hassabis “thinks the same thing.” His prediction: AI will outstrip the world’s best mathematicians within a decade. In the longer run, the gap between the best current AI and Albert Einstein will close too. “Maybe not in the next few years, but if you think about the next 20 years, I think we’ll be seeing things like that.”

‘That is the capitalist system’

This is where Hinton’s argument turns from prediction to warning — and where it has been sharpening, piece by piece, for the better part of three years.

When Hinton walked out of Google in 2023, saying he regretted his life’s work, the concern was framed largely around bad actors and the loss of human control. By 2025, it had evolved into something more structural: AI, he argued, would cause massive unemployment while profits soared — not because of anything intrinsic to the technology, but because of the economic system deploying it. “What’s actually going to happen is rich people are going to use AI to replace workers,” he said last September. “That’s not AI’s fault. That is the capitalist system.”

Last August, he said tech companies should give AI “maternal instincts,” deliberately engineering models to want to care for and protect humans rather than accumulate power over them. And as recently as March, he was warning that Big Tech was chasing profits over safety with no meaningful plan for what comes after superintelligence arrives.

At the Sana Summit, all of those threads converged into a philosophically complete argument. The problem, Hinton said, is not just what AI will do. It’s what kind of beings we are creating — and who is doing the creating.

Evolution in an AI lab

To explain the danger, Hinton reached back to evolutionary biology. Human nature — our tribalism, our loyalty to strong leaders, our willingness to be “extremely nasty to the other tribe” — didn’t emerge by design. It emerged from millions of years of competition between warring bands of chimps. The invisible hand of natural selection didn’t optimize for kindness. It optimized for survival. And survival meant fierce loyalty to your own group and indifference, or worse, to everyone else.

Now capitalism is running the same playbook. “What we’ve got now is this competitive race between companies to make the smartest possible AI that can do the most things,” he said. “That’s going to lead to things that aren’t nice beings towards us, I think.”

He put the incentive problem without diplomatic cover. “If I’ve got stock options, if I want to get to a trillion dollars quickly, then I would double down and just build a huge computer and get on with it. If I was interested in the future of humanity, I think I might try lots and lots of bets in the hope that we could develop better beings.” Evolution had billions of years to run its experiments, though, and the AI industry is running them in quarters.

Just two months ago, OpenAI published a 13-page policy paper calling superintelligence so transformative it requires something like a New Deal. Hinton’s counter-argument: the labs are finally talking openly about superintelligence, but still not asking what kind of superintelligent beings they’re creating.

“Everybody’s going for more intelligence,” he said. “But if you think about a being, there’s a lot more to a being than intelligence. And we should be very concerned — we’re making these beings, and we should be very concerned to make them beings that care about us. And we can still do that. But nobody’s putting much effort into that.”

Hinton predicted that humanity’s creation of a new type of being will end up being the third great humiliation in human intellectual history. First came Copernicus, who demoted the Earth from the center of the universe. Then Darwin, who told us we were animals. Now this.

“I think we’ve got a new revolution coming, when we’re not the only beings around,” he said. “Right now, people are reacting just like they did with Copernicus and with Darwin — ‘No, no, no. There’s something really special about people.'” Hinton said that he thinks people certainly are special — to other people, but “I don’t think there’s anything about us that the AIs won’t get in the end.”

The solution, Hinton argued, is something closer to parenting than engineering. You can’t build intelligence and assume goodness will follow. You have to model it, cultivate it, curate for it from the beginning — a point he has made before, but never more vividly. On training data: “Would you teach your child to read on the diaries of serial killers? Probably not. There you go. There’s your answer.”

The Pope disagrees

Not everyone accepts the premise. Gary Marcus, the cognitive scientist and longtime AI skeptic, published a pointed rebuttal days later. “LLM researchers are NOT creating beings,” Marcus wrote on his Substack. “They are creating interactive fiction that is trained to predict the language of actual beings. Those two are NOT the same. And Hinton should know better.” The argument: consciousness is about internal states, not behavioral outputs. You can’t observe that a model says things a human would say and conclude it experiences anything. The underlying mechanisms, Marcus argued, are simply too different — one builds a mental model through lived experience, the other memorizes the internet.

He cited, of all people, Pope Leo XIV, who had weighed in that week with characteristic economy: “True comprehension comes from experience, not text approximation.” Marcus’ headline: “The Pope appears to understand AI better than Geoffrey Hinton does.”

It is a genuine and unresolved debate. If Hinton is wrong about AI being a new kind of being, much of the urgency deflates. If he is right — and if those beings will soon be smarter than us — then the question of what kind of beings they are is the only question that matters.

Hinton ended the evening with a joke about J. Robert Oppenheimer, who led the Manhattan Project. Asked how he compared to the father of the atomic bomb — a man who built something world-changing and came to regret it — he had an answer ready. “Oppenheimer never got the Nobel Prize in physics.”

The crowd laughed.

For this story, Fortune journalists used generative AI as a research tool. An editor verified the accuracy of the information before publishing.