The emergence of super-intelligent computer entities are coming, but they may or may not be friendly to humans, one of the top researchers in the field said.
“I hope for super benevolent, super beneficial, super intelligence,” David Hanson, founder of Hanson Robotics, said at the Fortune Brainstorm Tech International conference in Guangzhou, China. “So what this means is it has to understand the consequences of its actions on a very large scale, civilization scale.”
The danger is that the headlong rush to make new technological breakthroughs around artificial intelligence, or AI, may ignore the need to make the super-intelligent entities benevolent and focused on doing the greatest good for the greatest number, Hanson warned.
“There are many things that could wrong,” he said. “There are many catastrophe scenarios, some of which involve AI, some of which may be possibly fixed by AI. But we may not survive the next century.”
Hanson’s warning comes amid a fierce debate over the potential dangers of AI. Billionaire entrepreneur Elon Musk has repeatedly warned that humans could be wiped out by super-intelligent AIs. That prompted Facebook founder Mark Zuckerberg to declare Musk was being “pretty irresponsible.” Either way, the rush to develop ever smarter and more powerful AI continues, fueled by billions of dollars in private and government funding.
Get Data Sheet, Fortune’s technology newsletter.
Many looming disasters for humanity such as climate change are coming because of technological developments that didn’t look out for possible unintended consequences and didn’t take accountability for human actions, Hanson argued.
“It’s not responsible for us to say ‘well we have to prove it to ourselves that we’re doomed’ before we start to take action,” Hanson explained. “I think that we need to look at these issues with a very tough candor…we’ve got to get smarter and we’ve got to get wiser.”
Hanson’s company, based in Hong Kong, is trying to make more humanistic robots. Hanson Robotics’s Sophia robot addressed the United Nations and was made a citizen of Saudi Arabia.
