Self-driving cars and robots that can zoom on their own around warehouses are just some of what’s possible because of artificial intelligence. But expect unforeseen consequences if researchers ignore the inherent ethical dilemmas in the emerging technology.
That’s one of the takeaways from a panel about AI ethics and education in San Francisco that was hosted by the Future of Life Institute, a research group focused on preventing societal problems created by the technology.
Although humans typically program AI-powered robots to accomplish a particular goal, these robots will typically make decisions on their own to reach the goal, explained Benjamin Kuipers, a computer science professor and AI researcher at the University of Michigan.
Get Data Sheet, Fortune’s technology newsletter.
It’s these smaller decisions that robots make on their own that can cause trouble because human programmers may fail to take all of a robot’s possible choices into account, Kuipers said.
“This is not the robot apocalypse,” said Kuipers. “What we’re seeing here are robots pursuing human-generated goals in unconstrained ways.”
Kuipers did not cite a specific example of a robot making a harmful decision that its human programmers overlooked. Instead, he cited the Disney animated film Fantasia as an example of what technologists need to take into account when building their robots.
In Fantasia, Mickey Mouse, as a young wizard apprentice, magically commands a broom to fill a cauldron with water. When Mickey falls asleep, however, the broom ends up flooding the room because the untrained wizard failed to take in account that the broom would continue to fill the cauldron even after it was full.
Illah Nourbakhsh, a robotics professor at Carnegie Mellon University, said that educators need to teach computer science and robotics students a basic understanding of ethics, because the technologies they are creating are so powerful that they “are actually changing society.” He cited the examples of drones being used in warfare and AI technologies being used in advertising as ways cutting-edge technology is being used on a global scale and changing consumer behavior.
Having a basic understanding of ethics can help technologists better understand the potential ramifications of the AI-powered software and robotics they are creating, he explained. One ethical dilemma he cited is how robotics can increase factory productivity; while this may lead to a boost in a nation’s GDP, it can also increase the wage gap between the poor and the rich.
Nourbakhsh does not believe that technology is “neutral,” and that it is ultimately up to other people to determine how it should be used, for better or worse.
Technologists should think about how their creations will impact society and even the choice of words they use to describe them. For example, calling the technology that powers self-driving cars either a safety-enhancing system or a labor-saving system has big consequences for how society perceives the technology, he explained.
