Digital ‘watermarks’ will not save us from deepfakes and AI-generated fraud

It is fast becoming apparent that we are living in a new age of inauthenticity, midwifed in no small measure by generative AI. This week brought news of robocalls in New Hampshire using a voice impersonating President Joe Biden—almost certainly created with easily available voice cloning AI software— urging Democrats to “save their vote” for November and not to vote in today’s Republican primary. New Hampshire has open primaries, meaning voters do not have to be registered party members to vote, and some Democrats had hoped to hurt Donald Trump’s chances of securing the GOP nomination by casting ballots for his rival Nikki Hailey.

State election authorities are investigating the robocalls but there is little the government can do to prevent AI-aided dirty tricks such as this. It seems likely that with more than 4 billion people globally eligible to vote in various elections this year, not least the U.S. presidential vote, voice cloning is on track to superpower election interference. It is also enabling frightening new scams in which fraudsters use voice cloning to impersonate the victim’s loved ones in distressing situations (usually held captive by kidnappers or arrested and in need of bail money). More significant for Eye on AI’s business readers, perhaps, are scams in which con artists impersonate a CEO or company director and ask a finance or corporate treasury executive to make an urgent payment to “seal a secret acquisition deal”—that just happens to send the money straight to an offshore account the fraudsters control. None of this seems to have dissuaded investors from piling into synthetic media companies such as AI voice cloning pioneer ElevenLabs. The London-based startup just raised an additional $80 million from some of Silicon Valley’s best-known venture capital funds, including Andreessen Horowitz and Sequoia Capital, in a deal that values it at $1.1 billion.

Many people have suggested that “digital watermarking” will help turn off the spigot on the firehose of falsehood AI generates. But another story from this week shows why that is unlikely to be the case. Samsung’s new Galaxy S24 smartphone (which my Eye on AI colleague Sage Lazzaro reported on in Thursday’s newsletter) comes with generative AI photo editing tools. Realizing that this would raise concerns about the ease of creating manipulated scenes and deepfakes, the company decided that all images created with its AI editing tools would bear a visible digital watermark—an icon with four star shapes—in the lefthand corner. But, as journalists for Gizmodo quickly reported, the digital watermark can be easily removed using (checks notes) the phone’s own AI editing tools.

Now, the S24 also automatically adds metadata to the image file that indicate AI effects have been added. This metadata may be a bit harder to scrub clean than removing the watermark. But probably not that hard. And this is why digital watermarking, which has now been rolled out by Adobe, Microsoft, and others, is no silver bullet for the problem of deepfakes and other kinds of generative AI fraud. It is simply too easy for bad actors to figure out ways to remove the watermarks or bypass the watermarking process. (Compared to images, there is even less consensus about exactly how to watermark AI-generated text and audio. OpenAI was working on a cryptographic-based watermark for the text ChatGPT produces, but it has not been implemented in production, and some researchers have already figured out ways to get around it.)

This brings me back to my conversation from a few weeks ago with Getty Images CEO Craig Peters. First, Peters related an astounding statistic that attests to the urgency of the problem: There have been more images produced with AI in the past 12 months than lens-based photographs. And that trend will only accelerate. Peters says what’s needed is a layered approach to authenticity. He’s in favor of metadata that provides an indication of photo manipulation, but knowing metadata can be altered, he says this alone is insufficient. In addition, he says there should be a global effort to create a provenance standard that includes a cryptographic hash stored in an immutable database that anyone could check to verify if an image is AI-created or if it comes from an authentic source.

Currently, Adobe has been promoting a Content Authenticity Initiative that includes encrypted metadata to track the provenance of images, along with related “Content Credentials” for AI-generated ones. Both are based on a cryptographic standard called C2PA. Besides Adobe, hundreds of organizations have signed up to this standard, including most notably camera-makers like Nikon and Leica, as well as Microsoft, which labels all of Bing’s AI-created images using C2PA-compliant Content Credentials. But, as Peters notes, C2PA is itself a flawed standard and there have already been cases where people have managed to change the metadata of AI-generated images to make them appear legitimate. He says we need something better. Getty is working on it, he says—but it isn’t there yet.

So, no watermarking is not going to save us this election year. We’ll need to muddle through another year of post-truth, our skepticism ramped up to 11. But we better find a solution soon. Distrust is insidious and corrosive to democracy and society.

With that, here’s more AI news.

Jeremy Kahn
jeremy.kahn@fortune.com
@jeremyakahn

Correction: Jan 25. An earlier version of this story misstated a statistic Getty CEO Craig Peters used to illustrate the growth of AI-generated imagery. He said more AI-created images have been produced in the past 12 months than lens-based photographs, not that the amount of AI-generated images produced in that period already exceeded the number of photographs produced throughout history.

AI IN THE NEWS

Sam Altman in talks with Middle Eastern investors, SoftBank, and TSMC about Nvidia rival. More details are emerging about the OpenAI cofounder and CEO’s quest to set up an AI computing chipmaker that could rival Nvidia, which currently has a near monopoly on the production of the graphics processing units (GPUs) that are most commonly used for AI applications. Altman has, according to a report in the Financial Times, held talks about an investment into the project with influential Abu Dhabi-based investor Sheikh Tahnoon bin Zayed al-Nahyan, who chairs two of Abu Dhabi’s sovereign wealth funds and UAE-based AI company G42—and who also serves as the national security advisor for his brother, the UAE’s president. Altman has also been in discussions with Taiwan Semiconductor Manufacturing Co. (TSMC), which currently produces most AI chips, including Nvidia’s, about setting up a new global network of chipmaking fabs to support the venture. Bloomberg reported the OpenAI CEO has talked to Softbank about investing in the project as well. But solicitation of Middle Eastern investors, and in particular the possible involvement of G42, has raised eyebrows among U.S. lawmakers and national security hawks who are concerned about G42’s links to the Chinese government.

AI startup Cohere in talks to raise up to $1 billion in further venture funding. That’s according to the Financial Times, which says the amount would exceed the total funding the Toronto-based developer of LLMs, cofounded by former Google researchers, has secured from investors to date. It also says the round will value the company at far more than the $2.2 billion valuation it achieved in its last funding round in June 2023. Cohere’s fundraising and valuations have lagged behind rivals such as Anthropic and OpenAI. The potential investors in the funding deal were not named but Cohere previously raised money from Nvidia, Oracle, and venture firms Inovia Capital and Index Ventures. The newspaper said the funding round would be a key test of investors’ appetite for continuing to fund AI startups at loftier and loftier valuations. OpenAI has received most of its funding through its strategic partnership with Microsoft, while Anthropic has established a close relationship with Amazon and Google in exchange for large funding checks. Cohere, on the other hand, has tried to present itself as a neutral option, not aligned with any one Big Tech player.

Are high-flying generative AI startups overvalued given their gross profit margins? That’s the key question raised by an eye-opening piece in The Information that says hot AI startup Anthropic has a gross profit margin of between 50% to 55%. The article cites two sources “with direct knowledge” of the figures. That margin might sound hefty, but in the enterprise software industry, it’s unusually slim. The average cloud-based software-as-a-service provider's gross margin is 77%, according to stats from Meritech Capital that the publication cites. The article calls into question how profitable Anthropic and other generative AI startups will be in the long term and whether investors who have been willing to fund these companies so far are vastly overpaying. Anthropic, which has reportedly told investors it is on track to make about $850 million in sales this year, has, according to other news stories, been seeking a new round of venture capital investment at an $18.4 billion valuation. Most enterprise software companies are valued at just five to six times forward revenues. The gross profit margins also don’t include what Anthropic pays to train its large language models, an expense that could tip into triple-digit millions if comparable to what OpenAI CEO Sam Altman has suggested it cost to train GPT-4. The question is whether valuations will come down to earth once the hype around generative AI dies down, and which investors will still be there, when the music stops, probably holding hefty losses.

A new standard seeks to certify AI models as “fairly trained.” The idea is to create a kind of Good Housekeeping Seal of Approval for generative AI models, certifying that their creators have obtained permission to use any copyrighted materials in training the software. The new Fairly Trained label and initiative is the brainchild of Ed Newton-Rex, the machine learning researcher, entrepreneur, and composer who was Stability AI’s vice president of audio for a time. Newton-Rex resigned from Stability over the company’s position that training AI software on copyrighted works without permission constituted “fair use.” The new certification is likened to organic or fair trade labels, according to a Bloomberg story. Fairly Trained has certified nine startups so far, mainly in sound generation. Its certification requires the software makers provide detailed information about training data sources and licensing terms. Certification fees are based on company revenue.

Award of Japanese literature prize to novel written with ChatGPT’s assistance sparks debate. Novelist Rie Qudan won the award for her book Tokyo-to Dojo-to which used ChatGPT for about 5% of its content. Many literary critics and members of the public questioned the judges’ decision to give Qudan this year’s Akutagawa Prize, arguing it was unfair to other novelists and would turn literature into battles between AI technologies, the Japan Times reported. They argued literary prizes should ban the use of AI writing aids, in the same way chess and go tournaments have barred computer assistance. But others saw Qudan’s novel as an interesting experiment, fusing literature and technology.

EYE ON AI RESEARCH

Machine translation is making large language models worse. Machine translation was supposed to be this great democratizing force, making it easier for people and businesses to connect across the globe. But so-called “low-resource languages,” which are those that do not have vast troves of digitized text on the internet, have always posed a problem. Most machine translation of these languages is poor to abysmal. And now this problem is being compounded by new “multi-way parallel” translation datasets that include the same snippet of text translated into multiple languages. These are being used to train LLMs. That's a problem, as a new paper by Amazon researchers points out, especially because it turns out the majority of these datasets are themselves produced through machine translation.

While the translations from English to French, English to Spanish, and English to Portuguese are likely ok, the translation from Amharic to Portuguese is undoubtedly terrible, constructed by taking a not-very-good machine translation from Amharic to English and then machine translating that not-great English to Spanish, and then that Spanish to Portuguese and so on. Chances are the result of this machine translation “game of telephone” is pitiful. And that means that LLMs will not provide any benefit for speakers of these low-resource languages.

It would be better, as the Amazon researchers note, to try to tackle this problem by actually creating much better organic datasets of text for low-resource languages. Several innovative startups, such as Lesan and GhanaNLP, have been working on this problem. But their efforts are overshadowed by purely statistical techniques, such as Meta’s “No Language Left Behind” effort that don’t address the root cause of the issue. You can read the Amazon paper on the non-peer-reviewed research repository arxiv.org here.

Ok, having said all of that, probably the biggest research news in the past week was Google DeepMind’s publication in Nature of its work on a system that can solve complex geometry problems from the International Mathematics Olympiad about as well as human competitors. It’s a potentially important breakthrough for reasons I explain here. I encourage you to read that story.

FORTUNE ON AI

‘ChatGPT moment’ with full self-drive technology—now owners may be weeks away from finding out if he can finally deliver —by Christiaan Hetzner

AI chatbot calls itself ‘useless,’ writes elaborate poem about its shortcomings, and says it works for ‘the worst delivery firm in the world’ —by Marco Quiroz-Gutierrez

The world needs an International Decade for Data–or risk splintering into AI ‘haves’ and ‘have-nots,’ UN researchers warn —by Tshilidzi Marwala and David Passarelli (Commentary)

KPMG’s U.S. CEO: ‘The ubiquity of GenAI and just how disruptive it will be is creating greater demand for our services’ —by Paul Knopp (Commentary)

BRAIN FOOD

Why we should be so wary of law enforcement use of facial recognition software. If there was any doubts that the U.S. needs better regulation around law enforcement’s use of facial recognition software, a shocking story in Wired this week should silence them.

The story recounts how police in Berkeley, California (the East Bay Regional Park District Police Department to be exact) tried to generate a new lead on a cold case homicide. They first took DNA recovered from the crime scene that they thought was likely the suspect’s and gave it to a company called Parabon NanoLabs, which claims to be able to take DNA samples and run them through AI software that can output a “snapshot phenotype report” that is essentially the AI’s guess of what the person might look like. This is controversial enough, with some doubting the DNA analysis can produce images as detailed as the facial images Parabon NanoLabs produces. But then the police took that AI-generated, DNA-derived facial image and ran it through facial recognition software to try to match it to real people’s faces.

It sounds like some cool sci-fi trick. The only problem is that there is absolutely no evidence that it works. (And, in fact, using the snapshot phenotype in this way violated Parabon NanoLabs’ terms and conditions and the officer involved also was not fully transparent with the regional police body he asked for help running the image through facial recognition, telling them he had a digital image of the suspect when in fact he had nothing of the sort.)

As civil liberties groups noted, it is highly likely to result in innocent people falling under suspicion. “It’s really just junk science to consider something like this,” Jennifer Lynch, general counsel at civil liberties nonprofit the Electronic Frontier Foundation, told the publication. Worse, Wired interviewed a number of police officials who basically said they saw nothing wrong with what the police in Berkeley had tried. (And the reporter found other cases where police departments had fed composite sketches to facial recognition software, which is almost as bad as what the Berkeley cops did.)

This is further evidence that the police simply can’t be trusted to use AI software responsibly. The police officers involved don’t seem to actually understand the underlying technology and its limitations and how those limitations might impact an investigation. Instead, they want magic wand solutions—and the companies selling this technology to law enforcement are only too happy to market their software to them in this way. This is why state and federal rules requiring the police to understand the risks of the software they are using and to take steps to mitigate those risks is essential.