Technologists are still working to perfect A.I. and automation technologies that can accomplish one or two tasks—say, driving or fraud detection—in a very complex world. But it’s not hard to envision a future in which multiple computer agents will work together to solve even bigger problems. In a recent experiment, Alphabet Inc’s DeepMind team set out to illustrate how learning machines work together—or occasionally, as it turns out, don’t.
The team ran thousands of iterations of two simple games, allowing programs controlling them to learn the most effective strategies. In one, two A.I.s competed to gather the most resources (green ‘apples’), and could also choose whether or not to sabotage their competition with the yellow zaps seen in the video below.
Get Data Sheet, Fortune’s technology newsletter.
What they found raises some concerns about our imminent A.I.-driven future. As the team detailed in a blog post, computer agents in the apple-gathering scenario tended to coexist peacefully when there were plenty of resources. But when apples became scarce, they quickly turned to sniping one another to get an edge.
More concerning still, “agents with the capacity to implement more complex strategies try to tag the other agent more frequently, i.e. behave less cooperatively,” regardless of how many resources were around. Smart robots, in other words, have no inherent hesitation to harm one another in pursuit of their goals—or, presumably, to do the same to any human who might get in their way.
In a different scenario, though, the programs found more reason to work together. When set to ‘hunt’ for shared rewards, A.I.s would figure out how to collaborate—and were actually more likely to do so the smarter they were.
The assumptions behind these experiments deserve some scrutiny. The researchers specifically equate their hyper-rational A.I. agents with “homo economicus”—that is, the imaginary, purely gains-maximizing human that made up the flawed core of 20th century economics. But in the real world, humans aren’t all that consistently rational, casting some doubt on the researchers’ claims that their game theory scenarios could help us understand human systems like the economy and traffic.
And future robots themselves aren’t unavoidably fated to exist as pure maximizing agents. Even learning and evolving A.I.s could be designed with behavioral guidelines that would supersede their narrowest imperatives, ala Isaac Asimov’s Three Laws of Robotics. It’s not without its challenges, but the uneven behavior of DeepMind’s competitive algorithms suggests some form of empathy should be on programmers’ to-do lists.
