“We know that people uniquely trust Google Earth for a reliable view of the world,” a Google spokesperson told Fortune. “It’s important to note that generated images didn’t appear in the main Google Earth experience for others… and were watermarked as AI generated.”

The spokesperson’s comments come after it quickly became apparent users began abusing the system to create false images meant for disinformation. Some generated images of a nuclear plant in Iran, or refugees near the Mexico-US border.

The feature allowed users to overlay AI-generated scenes onto real satellite, aerial and 3D imagery. Google envisioned legitimate uses such as visualizing planning concepts or historical reconstructions, but users almost immediately began generating fabricated disasters, false military installations and conflict scenes tied to real-world locations.

Adverse effects common in AI use

Google’s Nano Banana technology allows users to generate AI images using simple queries, built-in to its Gemini chatbot. Unlike earlier AI-image generators, Nano Banana was designed to allow users to edit their images. Akin to Photoshop—the AI would add, remove or alter objects within an image while preserving its realistic features.

Google released the Nano Banana software integration with Google Earth with a few safeguards. Images generated using the system were separated from authentic Google Earth imagery and were embedded with watermarks designed to indicate they had been created with AI, a Google spokesperson told Fortune. The company also employs the use of “SynthID,” a cryptographic watermarking technology designed to identify AI-generated content. Still, experts say the minor guidelines set by the company aren’t sufficient.

“To their credit, Google has a system that makes fabricated images detectable,” NYU computer science and engineering professor Emily Black told Fortune. “Even if there were, however, recent research has shown that’s not a full solution.”

The European Union AI Act, with transparency requirements for AI-generated media, required providers such as Google to make synthetic content detectable. Google’s compliance to that approach was by embedding the SynthIDs into their generated images.

Black also notes that SynthID is “invisible to the naked eye,” and is a way for computers to check if an image was generated with AI. For AI-generated images to be noticed by users, it has to include a watermark over the image. In the case of Google Earth and Nano Banana, only digital watermarks were used, meaning images were not immediately noticeable as AI-generated. This also means images that were screenshotted, or recreated outside of Google’s platform would not include the same metadata from the original Nano Banana fabrication.

“We need policymakers to be thinking about this and collaborating with experts to come up with better solutions,” she told Fortune. The professor also pointed to changes in US AI policy.

Black currently researches how AI outputs can influence user decisions and trust. This can lead to real-life negative results, like poor investment decisions based on viewing incorrect financial information hallucinated by large language models. According to the professor, her research—while focusing on AI hallucinations and text-based AI systems—demonstrates that simply disclosing information is AI-generated may not prevent people from trusting that information.

“In our ongoing work on user trust of AI hallucinations, we’ve found that run-of-the-mill disclosures are often not effective at dissuading users from following harmful advice,” Black said. “Even with disclaimers like, ‘AI chatbots can make mistakes. Check important information,’ the presence of incorrect AI-generated advice significantly increases a user’s reported likelihood of taking a harmful future action.”

Google told Fortune it was rolling back the feature “while implementing stronger guardrails,” acknowledging that the generated imagery that users were manipulating appeared to violate company policies.

Artificial intelligence creates artificial trust

Generative AI has become commonplace across chatbots, design software and image generators, where users generally understand they’re creating fictional or artistic content. Google Earth, historically, occupied a different role. For nearly two decades, it has served as a visual record of the physical world, relied upon by everyone as an up-to-date feed of the planet. That reputation changes how users interpret what they see. Google’s trust from consumers means users often approach the imagery inside the platform with an assumption that what they are seeing corresponds to reality.

Now, with AI being integrated in what used to be considered a real-feed of planet Earth, it causes even Google Earth images to be scrutinized under a microscope.

Henk Van Ess, an international expert in online research methods in AI, wrote in a note following Google’s introduction of the Google Earth system: “An official can now look at a genuine photograph of a genuine atrocity and say: AI. He doesn’t need the tool for that. He needs everyone to know the tool exists.”

Google did not immediately respond to a request for comment on whether the Google Earth feature will return after additional safeguards are developed.