A.I.'s Hidden Biases Are Continuing to Bedevil Businesses

Bias will continue to be a fundamental concern for businesses hoping to adopt artificial intelligence software, according to senior executives from IBM and Salesforce, two of the leading companies selling such A.I.-enabled tools.

Companies have become increasingly wary that hidden biases in the data used to train A.I. systems may result in outcomes that unfairly—and in some cases illegally—discriminate against protected groups, such as women and minorities.

For instance, some facial recognition systems have been found to be less accurate at differentiating between dark-skinned faces as opposed to lighter-skinned ones, because the data used to train such systems contained far fewer examples of dark-skinned people. In one of the most notorious examples, a system used by some state judicial systems to help decide whether to grant bail or parole was more likely to rate black prisoners as having a higher risk of re-offending than white prisoners with similar criminal records.

“Bias is going to be one of the fundamental issues of A.I. in the future,” Richard Socher, the chief scientist at software company Salesforce, said. Socher was speaking at Fortune’s Brainstorm Tech conference in Aspen, Colo.

Dario Gil, director of research at IBM, also speaking at Brainstorm Tech, echoed Socher’s concerns. “We need robust A.I. engineering to protect against unwarranted A.I. bias,” he said.

At IBM, Gil said, the company was increasingly looking at techniques to provide businesses with a “data lineage” that would record what data a system used to make a decision, how that data was generated and how and when it was used to make a recommendation or prediction.

Gil said this kind of A.I. audit trail was essential for ensuring accountability, something he said must always reside with human-beings. “We have to put responsibility back to who is creating this software and what is their purpose and what is their intent,” he said. “The accountability has to rest with the institutions creating and using this software.”

Both Gil and Socher said that eliminating A.I. bias was not an easy problem to solve, especially because machine learning systems were so good at finding correlations between variables in data sets. So, while it was possible to tell such software to disregard race when making, for example, credit recommendations, the system might still use a person’s address or zip code. In the U.S., at least, that information can also be highly correlated with race, Socher said.

Gil said that IBM has been developing software—such as its AI Fairness 360 toolkit—that can help businesses automatically discover such hidden correlations in their data.

But, Socher said, discovering such correlations is one thing. Knowing exactly what to do about them is, in many ways, a much harder problem.

Socher said that in some cases, such as marketing breast pumps, it might be alright to only recommend a product to women. Meanwhile, in other contexts, the same sort of gender discrimination in recommendations would be illegal. For a company like Salesforce that is trying to build A.I. tools that are general enough that companies from any industry can use them for almost any use case, this presents a particular dilemma, he said.

This is one reason, both Gil and Socher said, many businesses are choosing to train A.I. systems from their own data rather than using pre-trained software packages for tasks chatbots or automated image-tagging. Building their own A.I., Gil said, gave businesses more control and more chances to detect hidden biases.

Both Socher and Gil said that one of the great things about A.I. is that it can help companies uncover existing bias in their business practices. For instance, it can managers who don’t promote women or financial institutions that don’t extend credit equally to minorities. “A.I. sometimes puts a mirror in front of our faces and says this is what you have been doing all the time,” Socher said.

He also said that certain types of bias were unlikely to be resolved until the people building A.I. systems were themselves more diverse. At the moment, he said, too many of the computer scientists creating A.I. software are white men. He also said too many of the A.I. applications developed so far reflect the concerns of affluent urbanites. He said this is one reason Salesforce has been supporting projects like the Deep Learning Indaba, a conference designed to bring together A.I. researchers from across Africa.

A.I.’s Hidden Biases Are Continuing to Bedevil Businesses. Can They Be Stopped?

More must-read stories from Fortune: