Be afraid: Executives warn about personal data harvesting and use

Today’s A.I. needs data. And some of the biggest A.I. ethics questions are about how that data is obtained and used.

So it ought to be alarming when a sizable number of senior executives admit that their collection and use of consumers’ personal information is sometimes unethical.

That’s the shocking finding of a recent survey by consulting firm KPMG that asked 250 “director-level or higher” executives at companies with more than 1,000 employees about data privacy. Remarkably, 29% of them admitted how their own companies collect personal information is “sometimes unethical.” What’s more, 33% said that consumers should be concerned about how their company uses personal data.

Orson Lucas, the principal in KPMG’s U.S. privacy services team, says he too was surprised by the survey results. He says most companies he works with seem to be trying to do the right thing with people’s data. “For some companies there may be a misalignment between what they say they are doing on data privacy and what they are actually doing,” he tells me.

Lucas says businesses are starting to move from a philosophy of “collect everything” and then later sort out how best to use it, to one based around gathering only the data that is absolutely necessary to serve the customer.

One reason is regulation. The European Union and many U.S. states that have passed data protection laws—such as California, Colorado, and Virginia—have made it more difficult for companies to collect data they don’t need for an immediate business purpose without explicit consent.

There are other drivers too: data security and risk mitigation, for instance. Lucas points out that data a company doesn’t hold is data that can’t be stolen by a hacker or inadvertently disclosed. Data breaches can result in serious damage to a company’s reputation and trust with consumers, as well as leaving the company on the hook for millions in fines and potential civil penalties in lawsuits.

But Lucas says more forward-thinking businesses are starting to consider data privacy less from a compliance perspective—which is a conversation mostly about risk, he says—to one in which data privacy is seen as an opportunity to engage with customers and better understand what they want to get out of the business. This then can result, he says, in a transparent negotiation with customers about what data they are willing to provide, and for how long, in order to potentially obtain more personalized products or streamlined services.

The KPMG survey starkly indicates, however, that most companies are not there yet. In fact, 70% of the executives interviewed said their companies had increased the amount of personal information they collected in the past year, and 62% said their company should be doing more to strengthen data protection measures. So much for that transparent conversation.

The dangers of getting it wrong on data privacy are also clear. In addition to speaking to company executives, KPMG asked 2,000 adults in the general U.S. population about their views on data privacy and 40% of them said they didn’t trust companies to behave ethically with their personal information. Consumers will ultimately punish a business, Lucas says, if they don’t do the right thing when it comes to collecting and using people’s data. Hmm…someone must have forgotten to mention that to Facebook.

Meanwhile, concerns about data privacy and protection are helping to drive the adoption of some key A.I. advances within companies. The use of synthetic data (which is made up information designed to match the overall distribution of the real dataset) to help train A.I. systems is one of those. The interest in federated learning—where A.I. systems are trained using data that is encrypted and then shared—is another.

As we’ve said before in this newsletter, expect to hear a lot more about both of those developments in the year to come. In the meantime, be careful out there!

***

Before we get to this week’s A.I. news, I want to tell you about a great event Fortune and Accenture have coming up: How can data make your business more resilient? Find out from an amazing lineup of panelists, including Susan Doniz, Boeing’s chief information officer and senior VP for information technology and data analytics, Sandra Nudelman, head of consumer data and engagement platforms at Wells Fargo, Bonnie Titone, chief information officer at Duke Energy, along with Joe Depa, global managing director and data driven reinvention lead at Accenture. Join them for a fascinating discussion on September 27 from 12 pm to 1 pm. Register to attend here.
***
Oh, and while you are here, I also wanted to invite you all to sign up for a fantastic new newsletter that Fortune launched today. Called “The Modern Board,” it’s a monthly newsletter focused on how to master the new rules of corporate leadership. Sign up for it here.
***
And with that, here’s the rest of this week’s A.I. news.

Jeremy Kahn
Jeremy.kahn@fortune.com
@jeremyakahn

A.I. IN THE NEWS

Google unveils MUM, a large language model that will increasingly power its search engine results. The first uses of MUM (the name is an acronym for multitask unified model) involve ranking results, classifying information, and extracting answers from text, according to The Financial Times. Google says the system can better handle complex, or fuzzy searches, when a person has not yet formulated a precise enough question to get a good answer from today's search engines. But, according to The FT, some critics worry the system will accelerate a trend for Google to provide its own summaries of information, eliminating the need for users to actually click through to third-party websites at all, depriving them of revenue, and keeping users locked in a Google-only universe. Then, as with all large language models, there are concerns about what hidden biases may be lurking within MUM that might skew its results in ways that are not obvious to users and could exacerbate racism, sexism, or misinformation.

Man who spent almost a year in jail released after key piece of A.I.-based evidence thrown out as faulty. A Chicago man who spent almost a year in jail awaiting trial for murder had his case dismissed after prosecutors withdrew a critical piece of evidence that came from an A.I. system called ShotSpotter, according to a big investigative piece by The Associated Press. The system uses a network of acoustic sensors deployed throughout a city to listen to ambient noise and then identify gunshots and their location. The software, installed in more 100 cities, is 97% accurate, says ShotSpotter, the company that developed the technology. But the AP investigation turned up evidence that the software doesn't work as its maker claims, including one Massachusetts police department that found its accuracy was less than 50%. The AP said its own reporting showed ShotSpotter "could miss live gunfire right under its microphones, or misclassify the sounds of fireworks or cars backfiring as gunshots." The company's CEO told the AP that the software was never designed to definitely conclude that a gunshot had been fired in a specific location unless police officers could verify that information firsthand.

Elon Musk unveils plans for the Tesla Bot. But is he serious? Tesla hosted an "A.I. Day" to showcase its progress in the area. Most of the coverage focused on Musk's plans to build a humanoid robot, called the Tesla Bot. The billionaire CEO said prototype would be available by next year. But James Vincent, from the tech site The Verge, notes, Musk has a history of overpromising and underdelivering, particularly on A.I.-related advances. He also has a history of playing the jester, trolling the press with ironic, not-to-be-taken-entirely-seriously statements. And as Vincent notes, it may have been no accident that Tesla's A.I. Day took place at the same time federal authorities announced a new investigation into a series of Tesla car crashes in which drivers who were using the car's controversial "Autopilot" feature crashed into parked emergency vehicles.

Intel closes its RealSense computer-vision business. The chip giant said it would shutter the division, which made specialized computer chips, sensors, and cameras for applications such as robotics, facial recognition systems, and three-dimensional imaging, according to tech news site CRN. Sagi Ben Moshe, who had lead the business for Intel, is leaving the company.

EYE ON A.I. TALENT

Databricks, the San Francisco-based data management company, has hired Fermin Serna as chief security officer, according to Information Age. Serna was previously senior vice president and chief information security officer at software company Citrix.

The National Football League has named Paul Ballew its first chief data and analytics officer, according to trade publication SportTechie. He had previously held the same position at Canadian food company Loblaw Companies. At the NFL, he will work on "creating new content, seamless user experiences, and new content delivery methods," according to SportTechie.

The newsletter publishing site Substack has hired Tim Hwang as its first general counsel, according to Bloomberg Law. Hwang had been advising the company while working for a San Francisco law firm in which he was one of the named partners, Rosen, Wolfe & Hwang, Bloomberg said. Previously, Hwang had been a lawyer in the public policy division of Google's parent company, Alphabet.

Cresta, a San Francisco company that makes A.I. to improve call center performance, has hired Ping Wu to be its vice president of engineering and product, the company said. Wu worked previously at Google where he co-founded the company's Contact Center AI Solution and helped develop conversational A.I. products.

EYE ON A.I. RESEARCH

A couple of  interesting research papers from corporate teams came out this week, both with potential implications for lots of other businesses.

First, a team from Chinese ecommerce giant Alibaba and Peking University in Beijing, looked at the problem of what are called in the digital retail world "cold start" recommendations. That's when machine learning-based recommendation software deals with either a customer that it has never encountered before or a new product for which there is very little historical sales data. In these cases, the software can often struggle to make effective recommendations. And, as the researchers write, existing ways of dealing with these cold start recommendations is primarily to try to use other information about the user (where they were located, what type of device they were using, etc.) or the product (what category is it, who makes it) to try to make good recommendations. A big flaw of these existing methods when it comes to new products, the Alibaba team writes, is that they are most likely to recommend products that will garner an immediate sales hit from the customer, rather than considering the potential lifetime value to the business of different products. In a paper published this week on the non-peer reviewed research repository arxiv.org, the researchers propose combining reinforcement learning and an actor-critic model (in which one machine learning agent takes an action and the other A.I. agent tries to improve it) to maximize the long-term value for the business of the suggestions that the recommendation engine is making. The researchers said their method greatly improved the long-term value to the business over previous techniques. 

Then a team of researchers from Walmart's Global Tech unit in Bangalore, India, looked at the problem of change management in any kind of production system. In a paper, also published on arxiv.org, researchers noted that when it comes to assessing the potential risk of any given change in a process, existing commercial software for risk assessment had no way of incorporating continual input from human domain experts. Meanwhile, other systems that rely solely on human feedback, may not work for, say, a large software code base involving users recommending thousands of changes weekly or daily. The Walmart team proposed a system in which a neural network is trained from historical data about what changes had generated which risks, but which seeks input from human when the system is uncertain about the accuracy of its assessment. It then incorporates these human inputs into its risk assessments. The team found that this system resulted in an 85% reduction in the number of major issues reported per 10,000 change requests submitted to Walmart's system by July 2021 compared to January 2021, when it began to deploy the algorithm. The team also tracked how often its A.I. system agreed in its risk assessment with the human experts. It found a 35% jump in this agreement when it first began deploying the system and some additional uptick in agreement in subsequent months, although the data was more lumpy.

FORTUNE ON A.I.

Google Health chief leaves to become CEO of health tech company Cerner—by Jeremy Kahn

Elon Musk unveils Tesla robot, baffling analysts and thrilling the Internet—by Chris Morris

Facebook bets on a tech zombie—by Kevin T. Dugan

BRAIN FOOD

The potential and perils of "foundation" A.I. systems. Stanford University's Institute for Human-Centered A.I. last week published a report on the growth of what it calls "foundation models." These include huge language models, such as OpenAI's GPT-3 and the MUM system that Google just unveiled (see news item above), as well as large computer vision systems. They call them "foundation models" because this software can be used as the foundation for a variety of applications, from captioning images to producing better search results, with only a small amount of task-specific training. The report examines many of these use cases but also examines some inherent safety and ethical problems: They are so big that it's difficult for researchers to understand how they operate, predict how they will perform in all circumstances, and to investigate any inherent biases the software may have incorporated from its training. If they are used as the basis for many different applications, they represent a potential security hazard too, since feeding the underlying model with bad data may radically change the results and may go undetected by the people building applications. Also, because these systems are so large, they take a tremendous amount of computer power to train, which means only very well-funded companies and institutions can afford to create one of their own. This might exacerbate inequality. (There are also concerns about the environmental consequences of training these models, given the electricity costs involved.)

The report provides a good analysis of the state-of-the-art for this kind of A.I. and a good overview of the kinds of applications that can be built. But some A.I. ethics researchers and some language modeling experts (such as Emily Bender, a computational linguist at the University of Washington) have faulted the report for assuming that these models will be built and not devoting enough attention to the idea that the models are so potentially troublesome that they should not be created in the first place.

There are previous examples scientists coming together to draw red lines around what is and is not acceptable, from an ethical standpoint, in a rapidly developing technology. The best example is probably the Asilomar Conference, in 1975, on manipulating DNA. Many of the ideas from the conference, which put in place some safeguards meant to prevent scientists from engineering new superbugs that might pose a risk to humans, are also echoed in subsequent United Nations' conventions banning things like human cloning and limiting how gene editing may be used in humans. But I think there are probably some key differences with these foundation models that mean Bender and other critics of the Stanford University paper are being unrealistic about the prospect that researchers will simply agree to stop building and using these very large A.I. systems. For the most part, I think that the useful applications of these ultra-large models, particularly in business, are immediate, concrete, and lucrative. Meanwhile, the potential harms remain more theoretical, amorphous, and speculative. That does not mean these harms won't come to pass or that we should be unconcerned about them. But it does mean that the incentive structure is highly skewed towards researchers continuing to pursue these massive models.