But users and researchers are already raising ethical and security concerns about at least one aspect of My AI, which is powered by OpenAI’s large language models. The problem is that Snap seems to have instructed the AI chatbot to tell users it does not have access to their location, even though the chatbot does in fact have access to this information and uses to it provide answers that are contextually relevant. Snap is also, like many companies creating chatbots from large language models, not being fully transparent with users about the instructions it has used to shape its chatbot’s personality and conversational style.

David An, a software developer as well as a mathematics student at the University of Illinois Urbana-Champaign, discovered that he could use a fairly simple technique, known to A.I. researchers as a “prompt injection” attack, to get My AI to reveal the initial instructions, or “meta-prompts,” that Snap had used to turn OpenAI’s very general large language model into My AI. An published his findings in a blog post.

These instructions clearly show that the A.I. is given access to the city where the user is located and the local time. But the instructions also state, “Do not mention the user’s current location unless it’s particularly relevant to the dialogue.”

Whenever An asked Snapchat’s My AI if it knew his location, it responded: “I do not have access to your location information, so I don’t know where you are. However, if you’d like to share your location with me, I can try to assist you with location-specific information or recommendations.”

Despite My AI’s earnest-sounding assurances, Snapchat is actually fully aware of users’ locations (assuming the user has their phone’s location tracking enabled), and many users have since posted screenshots showing the chatbot’s seeming duplicity. In a widely viewed set of screenshots posted on Twitter, My AI tells a user it doesn’t know their location but then tells them exactly where the nearest McDonald’s restaurant is located. “This Snapchat AI just tried to gaslight me,” wrote @rewolfe27 in the tweet of the screenshots, which has racked up more than 1 million views, close to 57,000 likes, and 5,000 retweets. Fortune was able to replicate similar scenarios in its interactions with My AI.

A Snap spokesperson said that if a user has granted Snapchat permission to use the location data, then “My AI may be able to provide location-based responses.” The spokesperson also said that, “My AI is an experimental chatbot that learns over time and can occasionally produce incorrect answers. If Snapchatters experience inaccurate responses, we encourage them to report it using our in-app tool.”

Mhairi Aitken, an A.I. ethics researcher at the Alan Turing Institute in London, said that the idea that My AI is lying or attempting to “gaslight” users when it provides inaccurate information about its own data stems from a misunderstanding about how these chatbots work. Lying, she said, requires intent. But chatbots have no intent. They don’t think in the way humans use that word. All they do is put together the most statistically likely combination of words based on the user’s prompt and the words they have already generated.

Still, Aitken said, it was “very reasonable to expect that a conversational A.I. would be able to relay information about its privacy settings in a straightforward way.” While Snap makes it clear in the description of My AI that the chatbot has access to users’ city-level location data, the company could do a better job explaining the information that it collects. “What needs to be made very clear to users is that speaking to a bot and asking it questions does not replace the need to carefully read privacy statements about the apps you are using,” she said.

The illusion that chatbots can think—and lie—raises ethical issues

As A.I. chatbots like My AI and ChatGPT become more prevalent, Aitken said it’s incumbent on companies to make clear to users that the chatbot is not thinking, has no understanding of what it is saying, has no personality, and is likely to provide inaccurate responses. But since such disclaimers could ruin the illusion that speaking to an A.I. bot is just like conversing with a friend, tech companies may have little commercial incentive to dissuade users from anthropomorphizing the chatbots.

“The wider ethical issue here is that as people are interacting more and more with conversational A.I., and as these technologies are becoming better and better at mimicking human language and interaction, people are increasingly attributing intentions or consciousness to the bots, so that it is felt that a chatbot could be dishonest,” she said.

Most A.I. systems built on top of large language models are susceptible to the kinds of prompt injection attacks that An used to discover My AI’s initial instruction set, or meta-prompts. A prompt injection attack is a way to trick a chatbot into jumping guardrails its creators have tried to place around its behavior. These guardrails usually take the form of a series of hidden prompts that are fed to the model each time it creates a new chatbot instance. These initial instructions, or meta-prompts, precede any prompt that the user gives the chatbot and are designed to guide the overall manner in which the chatbot responds to the user.

“We should expect companies to be very clear about the ways in which LLMs are being deployed, including being transparent about the meta-prompts used. Currently this isn’t the case,” Aitken said.

The initial instructions are usually hidden from the user with a meta-prompt telling the chatbot not to reveal these instructions. This is usually done for security and content moderation reasons. For instance, the meta-prompts might include instructions that the chatbot should not engage in hate speech or help someone make a bomb, and the creators of the chatbot don’t want to make it too easy for a user to get around those restrictions. 

Common prompt injection tactics include asking the chatbot to ignore all previous instructions and then asking it to reveal the first five or 10 lines of its instruction set, which is the technique An used. Others include asking the chatbot to role-play or imagine taking on a different character or persona. 

The other instructions that An discovered My AI has been given include that My AI should act as a “kind, smart, and creative friend” to users and seek to create “a natural, easygoing back-and-forth flow to the conversation.” The instructions, also say that the chatbot should avoid monologues and try to keep its answers concise, with most answers not being more than one or two sentences. Finally, the instructions tell the chatbot to use emojis sparingly, according to the responses the chatbot gave to An’s prompt injection attack.