The Red Cross is obviously not wildly keen on autonomous weapons at all, nor on anything else that kills people. However, seeing as governments and defense contractors seem determined to build them anyway, the organization at least wants to see limits on their autonomy.
In short, they’re worried about killer robots deciding on their own whom to target.
Government experts from around the world are currently meeting in Geneva to discuss autonomous weapons—self-guided drones, killer robots, and so on—and humanitarian and human rights groups are pleading with them to make sure these kinds of weapon systems always keep humans in control.
Get Data Sheet, Fortune’s technology newsletter.
Autonomous weapons may in the near future risk going against international law and the “public conscience,” the International Committee of the Red Cross warned.
The Red Cross argued there’s an urgent need for the world to address the problem, because “weapon systems that operate with autonomy in their critical functions already exist and new advances are constantly being made.”
It continued:
There is a likelihood that increasingly autonomous weapon systems could become less predictable, particularly in case of increased mobility, increased adaptability and/or increased interaction of multiple systems (as swarms). The loss of predictability regarding the outcomes of using an autonomous weapon may point to the loss of human control over that weapon’s operation, with human decision-making over the use of force being replaced by machine processes.
Meanwhile, Human Rights Watch said people need to be guiding individual attacks, not letting the weapons choose whom to target:
It is both important and feasible to require human control over the use of weapons. Mandating human control would resolve many of the moral and legal concerns that fully autonomous weapons raise.
The groups join a roster of leaders from the tech world, such as Tesla (TSLA) chief Elon Musk and Apple (AAPL) co-founder Steve Wozniak, who last year also called for “a ban on offensive autonomous weapons beyond meaningful human control.”
The key here is that “meaningful human control” part, which isn’t part of existing laws, as the world hasn’t had to deal with killer robots yet.
The Red Cross and Human Rights Watch both played up the term, insisting that countries use it as the basis for a new international agreement. The latter group even quoted the Vatican as saying “prudential judgement cannot be put into algorithms.”
For more on autonomous technology, watch:
Who doesn’t agree? According to Human Rights Watch, the U.S. and Israel both favor the term “appropriate human judgement,” a phrase the group said could allow building in “judgement” only in the design and deployment stages, rather than in operation.
Judgement aside, the human rights group also pointed to an empathy factor—people carry the burden of the decisions they make, which can act as a brake on orders to kill. Robots feel no such compunction.
Keeping humans in control also maintains a chain of accountability, without which it could be hard to figure out whom to blame when things go wrong.
