Before introducing its digital voice assistant in 2016, Google deployed a group of employees to make sure the service could field questions involving race and gender without stumbling. Any problems after publicly introducing the service would be a major embarrassment to the company, which has been criticized for creating products and features that discriminated against minorities.
The so-called product champions found 28 problems in all, including that the assistant failed to understand the significance of Black History Month and Hispanic Heritage Month. It also sometimes used incorrect pronouns—making unfair associations including that doctors were men—and fell well short when responding to someone who admits they’re gay for the first time to the assistant.
The effort was led by Google’s product inclusion team, a small group of employees who train others in how to design products for a broad base of users, test technologies for those users, and create a set of best practices for internal product development teams. The goal is to help Google create products that are without bias related to ethnicity, gender, sexual orientations, or disabilities.
“What we want to do is consult with teams,” said Annie Jean-Baptiste, head of product inclusion, research, and activation. “So if they’re starting a new feature or product, we want to be alongside them to make sure they’re building inclusively.”
The fixes are becoming increasingly important as big tech companies like Google face growing scrutiny for their track record on discrimination. It also comes at a time when tech companies are being hammered for the lack of diversity within their ranks.
In the past few years, Google, alone, has been slammed for a series of diversity problems.
Two years ago, Google came under fire for a sexist viral letter one of its employees wrote and for allegedly paying women less than men. Last year, 20,000 employees walked out over allegations that Google had mishandled sexual misconduct issues. Google’s search feature at one time tagged images of African-Americans as “gorillas,” and most recently, a tool the company developed in conjunction with another Alphabet subsidiary to detect hate speech and abuse was racially biased.
Kellie McElhaney, founding director of the Center for Equity, Gender, and Leadership at the University of California at Berkeley’s Haas School of Business, said she’s not convinced Google is doing enough to prevent another product from discriminating against minority groups.
“I’m not sure it’s a top priority right now,” she said. “They’ve been so reactive to put out fires.”
Google’s product inclusion team started in 2017 as a side project of Jean-Baptiste, formerly a human resources employee focused on diversity at Google. But about a year later, Jean-Baptiste became the formal head of product inclusion, as part of a newly formed team that oversees the product efforts as well as initiatives to provide tools and skills training to unrepresented communities.
The team is small—just four people. But with Jean-Baptiste’s help, it has formed dozens of working groups across the company to implement new principles and design, conducted 80 internal talks to share the latest insights about building for diverse sets of users, and created internal resources like videos and meet ups to answer questions employees may have about the process.
It also has recruited thousands of Googlers to help test products under the theory that a more diverse set of testers will spot more problems. And it has trained 35 directors and executives in overseeing that their product groups are considering a diverse set of users during each step of development.
The initiative is helping Google identify more problems before introducing new products and will continue to be key to avoiding future snafus, according to Google’s leaders.
“To work, this has to be core to the business,” Jean-Baptiste said.
But since many of the product leads voluntarily implement ideas from the product inclusion team, therein lies the problem, according to Google’s skeptics. The only employees held accountable for their inclusion work are those whose job functions depend on it. And beyond the product inclusion team, it’s unclear how many people that includes.
Erica Joy Baker, a former Google engineer, considers Google’s efforts to address potential problems with its products fragmented, at best.
“There needs to be somebody in the group with the social and political capital who can say, ‘These things need to be prioritized, and if not, a bad outcome of some sort will happen,’” she said. “There are still people within Google who don’t want to see more diversity and inclusion at Google.”
Joy has been a critic of Google’s culture for years. In 2015, she leaked a spreadsheet showing the salaries Google employees, to expose the inequalities in pay. The action led to many employees asking for and receiving better pay. But Google reprimanded her by rejecting bonuses that should have come as a result of positive peer reviews, Baker has previously said.
Google admits that it’s still perfecting how it builds products for all people, and much of the work is done on the honor system.
“Part of ‘Googliness’ is holding yourself accountable to do right,” said Liane Aihara, manager of Android/Pixel Insights, a team that applies user feedback to product development for the Android and Pixel devices. “You take it upon yourself to be accountable.”
Aihara was trained by Jeanne-Baptiste’s group and has been working to ensure her team is always thinking about the broad set of users for each product or feature. That means asking questions like is the product designed for all demographics? How do you build a camera that will be equally flattering for all skin tones in the pictures it takes? Are phone features equally accessible to left-handed consumers as they are to right-handers?
But for product managers and developers, who are often challenged by tight deadlines and ever-changing requests for features, adding another to-do can be daunting. Aihara said she doesn’t expect her team to accomplish all of the product inclusion goals all at once, but rather take baby steps to continuously improve.
“If there are five things to do, my ask is to do at least one,” she said about the efforts. “A little step is better than no step.”
The challenges across the company to make sure every product takes all types of users into account are unique for each product. The possible pitfalls for G Suite, Google’s family of productivity apps like Gmail and Google Docs, are totally different than those of the Google Assistant digital helper, for example.
“Our mantra is incremental steps are the best way to make change,” said Beth Tsai, policy lead of the Google Assistant. “We’re not going to solve inclusion and diversity in one fell swoop.”
Ideally, efforts to build products for all people should be reinforced daily by Google CEO Sundar Pichai, driven by employees at the ground level, and supported by middle managers who should have incentives tied to their efforts, said U.C. Berkeley’s McElhaney. But that’s not what she believes is happening at Google.
“They need to do it in a more strategic and comprehensive way,” she said, about Google’s product inclusion efforts. “They’re still in the early stages of their journey.”
More must-read stories from Fortune:
—Apple CEO Tim Cook has plenty to say as new hardware looms
—How Russian PR firms plant stories for companies in U.K. news outlets, social media
—Behind the ouster of eBay’s CEO? A trend every tech company should pay attention to
—From porn to scams, deepfakes are unnerving business leaders and lawmakers
—Why Apple is offering cheaper streaming and iPhones
Catch up with Data Sheet, Fortune’s daily digest on the business of tech.
