For business, algorithmic bias has become a big issue. Many companies, and the software vendors selling them A.I., worry that their automated systems unfairly discriminate against protected classes of individuals, risking legal, regulatory, and reputational trouble. Fear of algorithmic bias has slowed deployment of A.I. in many sectors. It has also opened up a whole sub-industry of consultants and software vendors that sell tools to detect algorithmic bias and try to mitigate it.
Much of the focus has been on how to fix biased data. Minority groups are underrepresented in most datasets—in fact, due to overt and implicit discrimination and human bias, they are often far less represented in the data than they are even in a given population. This mean A.I. systems trained on this data won’t perform as well for minority groups. One solution has been to try to curate better datasets, with better minority representation. Another has been to test rigorously to make sure that historical biases, such as unfair lending practices, are not replicated by A.I.
But it turns out that mitigating bias is far harder than even many experts assume, as a story last week in the tech publication The Register pointed out. The story cited a research paper published in the science journal Science Advances that looked at racial bias in an A.I. system designed to classify brain activity from functional magnetic resonance image (fMRI) scans. It turns out that the datasets of fMRI data used to train these kind of A.I. systems don’t have much data from Black Americans. So it’s not surprisingly that the algorithms performed poorly on scans of Black patients. But what is surprising is that even when the researchers curated a training dataset consisting only of data from Black Americans, the resulting system still performed worse on Black patients than the standard system performed on white patients.
The question is why? And the answer is that the researchers really aren’t sure. But they have some hunches. One of them is that our very understanding of the geography of the brain—where the borders of brain regions are—is itself biased, having been developed primarily from data from white patients. They also suspect that the how fMRI machines are calibrated to register blood flow around the brain is racially-biased, again using a calibration model that was developed from white patients. It is this kind of hidden bias, located far deeper in the data pipeline than just the training dataset itself, that often escapes scrutiny even from data scientists.
As The Register story also notes, the U.S. government’s National Institute of Standards and Technology (NIST) has been trying to grapple with a set of standards around detecting and mitigating bias in A.I. Any standards NIST promulgates could have a big impact on how A.I. is used in industry—and definitely affects how any A.I. systems that the U.S. government deploys are developed.
To its credit, NIST realizes how deep the problem is and how crude and unsatisfactory many of our approaches to solving it are so far. In the report NIST published in March, it noted “current attempts for addressing the harmful effects of AI bias remain focused on computational factors such as representativeness of datasets and fairness of machine learning algorithms. These remedies are vital for mitigating bias, and more work remains. Yet…human and systemic institutional and societal factors are significant sources of AI bias as well, and are currently overlooked.”
NIST and government alone won’t solve this problem. Business will have to play its part too. Already there are troubling signs of a developing dystopia where automated systems, clothed in a veneer of “objectivity,” continue to unfairly impact minority groups. It will take a critical thinking and concerted effort—not just “push button” bias detection and mitigation solutions—to prevent this dystopia from becoming reality.
***
Brainstorm Tech, Fortune’s premier technology conference, is back in-person and back at its original location in Aspen, Colo. The event, from July 11 to July 13, will feature such speakers as Stewart Butterfield, the co-founder and CEO of Slack; Rene Haas, the CEO of Arm; and Lila Ibrahim, the COO at DeepMind; as well as Jonathan Kanter from the Department of Justice’s antitrust division. To apply to attend, click here.
***
P.S. If you’re interested in how you can help achieve a better work-life balance (and who isn’t?) and lead a healthier, more fulfilling life, then you should check out Fortune Well, our brand-new vertical dedicated to health and wellness and geared toward busy executives. Fortune Well provides actionable tips for readers on how to manage stress, stay active, be more productive, maintain better relationships, and make better choices around their own wellbeing. Check it out here.
Jeremy Kahn
@jeremyakahn
jeremy.kahn@fortune.com
A.I. IN THE NEWS
Google's A.I. research arm roiled by dismissal of another scientist. Satrajit Chatterjee, who was a prominent A.I. researcher at Google Brain, says he was dismissed after his team wrote a paper challenging research performed by colleagues that had been published in the prestigious scientific journal Nature, according to a New York Times article. The disputed research concerned using A.I. to design parts of computer chips. Chatterjee says Google refused to publish his critique. His dismissal comes a year after two other Google A.I. ethics researchers were fired following an incident in which one of the researchers had co-authored a paper challenging the utility of some of the large language models that Google has developed. In that case, Google had also said the paper had not met its publication standards. Google has defended its decision not to publish Chatterjee's research and says the A.I. scientist was "fired with cause," although it declined to elaborate on the circumstances. Sergio Guadarrama another Google A.I. researcher said on Twitter that Chatterjee's firing was unrelated to his research and criticized Chatterjee for attempting to turn himself into an academic freedom martyr by trying to link his firing to the earlier dismissal of the two A.I. ethics researchers, Timnit Gebru and Margaret Mitchell.
Autonomous ship makes second attempt at Atlantic crossing. An autonomous ship named after the boat that transported the Pilgrims to America in 1620, is making a second attempt to cross the Atlantic. The Mayflower Autonomous Ship, which was launched by the non-profit ProMare and built with technological help from IBM, left Plymouth, England, last week and is expected to arrive in the Potomac River by May 16. If it succeeds, it will be the first such oceanic crossing by a completely autonomous vessel. An initial crossing attempt, last year, had to be aborted after the ship suffered a mechanical problem. So far, this second attempt has proved uneventful. You can track the ship’s progress here. The ship is a critical proving ground for A.I. technology. I wrote about the ship here and highlighted some lessons businesses can learn from the attempt here.
Anthropic scores $580 million Series B. The San Francisco-based A.I. research company, which was founded primarily by alumni from OpenAI and which is focused on "A.I. safety," has raised $580 million in a funding round lead by cryptocurrency exchange billionaire Sam Bankman-Fried. Anthropic will spend the money to study ultra-large A.I. models, which are expensive to build and train, the company said in a blog post announcing the fundraise.
Alphabet acquires A.I. company Vicarious. The San Francisco company, which had been working on A.I. technology that was modeled more directly on the human brain than loosely brain-inspired neural networks for more than a decade and had raised $250 million from tech heavyweights such as Mark Zuckerberg and Jeff Bezos, will see its employees and technology distributed between two different Alphabet units: relatively new robotics startup Intrinsic and DeepMind, TechCrunch reports. The purchase price was not disclosed.
EYE ON A.I. TALENT
Atlassian, the Sydney, Australia-based company that makes software development tools, has hired Rajeev Rajan to be its chief technology officer, GeekWire reports. Rajan had been vice president and head of engineering at Meta in the Pacific Northwest region.
Palantir, the data analytics company and major government contractor founded by billionaire Peter Thiel, has hired Indra Joshi, Bloomberg News reported. Joshi had been head of A.I. for the digital arm, NHX, of the U.K.'s National Health Service.
EYE ON A.I. RESEARCH
A.I. can enable contextual advertising. Is that a good thing? That's the question asked by Emil Haglund, a doctoral student, and Johann Bjorklund, a professor, at Umea University in Sweden, in a research paper published this week on the non-peer reviewed research repository arxiv.org. Programmatic advertising, in which ads are served up to an individual surfing the web at the moment a site is viewed, is more valuable when an advertiser has access to lots of data about the individual viewing the ad. But allowing advertisers and platforms to have access to that data raises big privacy and ethical concerns. Contextual advertising, where ads are served up based largely on the nature of the web page in which they will appear, not on the basis of information about who is viewing the page, is increasingly seen as a good alternative. And artificial intelligence, in particular better natural language processing algorithms that can understand more about the context of that web page, including understanding what topic the web page is dealing with and what sentiment it is expressing, could play a big role in boosting contextual ads.
But as Haglund and Bjorklund point out, using A.I. for contextual advertising hardly solves all the ethical dilemmas posed by the way digital ads are served up to segmented audiences. Contextual advertising, the authors write, may reinforce the tendency to rely on gender and demographic stereotypes. They also point out that "reactive advertising"—ads that would actually change their content on the fly in response to context—could pose a control and credibility issue for publishers, especially if the language of the ads directly contradicts facts presented by the web page, and that many may ultimately choose to ban the practice.
FORTUNE ON A.I.
Snapchat CEO Evan Spiegel says he’s banned the word ‘metaverse’ at his office—by Carmela Chirinos
An algorithm that screens for child neglect raises concerns—by Garance Burke and The Associated Press
Hewlett Packard Enterprise used AI driven data to help accelerate Covid-19 vaccine research—by Susie Gharib
BRAIN FOOD
Is A.I. built on exploitative labor practices? That's the thesis of a recent article from MIT Tech Review that focused largely on the plight of desperate workers in Venezuela who were contracted to label data for companies such as Scale AI. The story is excellent journalism and a sobering, depressing read:
To keep their prices competitive, the firms similarly source workers from impoverished and marginalized populations—low-income youth, refugees, people with disabilities—who remain just as vulnerable to exploitation, [Milagros] Miceli, [a PhD candidate at the Technical University of Berlin who studies data labeling companies] says.
This has been particularly evident during the pandemic, when some of these companies began to loosen their standards. They lowered their wages and lengthened working hours as clients tightened budgets and the market’s sudden oversupply of labor drove down the average cost of data annotation. It has affected employees like Jana, a Kenya-based worker who asked us not to use her real name and says her diminishing income no longer supports her child. She now juggles two jobs. By day, she works full time at a firm seen as a pioneer in ethical data labeling. By night, she logs on to Remotasks and works from 3 a.m. until morning. “Because of corona, you don’t have an option. You just hope for better days,” she says.
But those better days won’t come without coordinated international advocacy and regulation to limit how low the industry can go, [Julian] Posada, [a PhD candidate at the University of Toronto who studies data annotators] says: “Platforms can move. If not the Philippines, then Venezuela. If not Venezuela, then somewhere else.”
