Is your chatbot hallucinating? A ‘bot debate’ could produce better A.I. answers, according to new research

CFOs are slow to embrace generative A.I. and the fact that a chatbot can hallucinate doesn’t help. 

Generative A.I. large language models (LLMs) that fuel chatbots are designed to understand and generate humanlike text. However, because they leverage billions of data points to predict the next word in a string of text, sometimes when not knowing the right answer to a prompt, they hallucinate, or create a response that may sound plausible but is factually incorrect or unrelated to the context. 

A group of MIT researchers released a new paper that finds a debate between chatbots can improve the reasoning and factual accuracy of LLMs. It’s like a bot debate club, except the bot can essentially debate iterations of itself. 

“The debate procedure allows a language model to critique and reflect on its opinions and opinions of other agents which allows it to sharpen its reasoning and answers,” Yilun Du, a researcher at MIT, and a coauthor of the paper, tells me. The researchers documented multiple instances of language models debating with each other over multiple rounds and reaching an improved shared answer.

How does this work? “The debates can occur in a single model (or bot),” says Du, who is a former researcher at OpenAI. “A single language model is replicated multiple times to generate multiple bots. Given a question, each bot then generates a different answer (the learned model behind the bot is the same across bots). The bots can then debate each other.”

However, the study also found that competing chatbots can spar with each other. “We also showed that you can have debates between different models like [OpenAI’s] ChatGPT and [Google’s] Bard to solve a task,” Du says. “But the majority of experiments use the same model.”

Michael Schrage, a research fellow at the MIT Sloan School Initiative on the Digital Economy, is not one of the authors of the paper but says he thinks the research is well done. “This kind of collective intelligence/voting approach is not uncommon,” Schrage says. “But to my knowledge, this is the first publication where I’ve seen it in a LLM context.”

Schrage has been exploring generative A.I. and LLMs with a focus on harnessing them as next-generation recommender systems. “I have already used large language models to generate business scenarios (some finance-related, others not) for both clients and classes,” he says. “I’ve found these scenarios constructive, provocative, and believable. But, again, these are LLMs, not large computational models.”

Foundational LLMs need to be fine-tuned and connected to software where calculations and computations are likely to be accurate, as well as transparent, explainable, and interpretable, he says. “That said, I think any financial analyst or auditor or accountant would be wildly irresponsible and unprofessional to rely on LLM-driven financial calculations at this time,” Schrage says.

He continues, “I strongly believe that—with guardrails and thoughtful, intentional prompts—FP&A folks and other financial modelers can get a lot of value very quickly by skillfully employing LLMs. The MIT research paper shows just how much is going on in the ‘computationally credible’ LLM space.”

Does Du think the issues with hallucinations or false information are valid concerns for finance professionals? “Yes,” he says. It’s very important to treat the responses from generative A.I. “not as ground truth, but rather just a possible source of information,” he says. Du suggests using responses as “ideas,” but then “separately verify yourself that they are correct.” He adds, “I believe my research is a step to making this source of information more accurate.” 

Let the debate begin.

Sheryl Estrada
sheryl.estrada@fortune.com

Big deal

A new report by Pew Research Center found that 58% of U.S. adults surveyed have heard of ChatGPT. Of that percentage, 19% say they have used it for entertainment, 14% have used it to learn something new, and 12% are currently working for pay and have used ChatGPT at work. Adults under 30 who have heard of ChatGPT are far more likely than those 65 and older to have used the chatbot for entertainment (31% vs. 4%). Pew also asked respondents about their experience with the chatbot. Fifteen percent say it has been extremely useful, and 20% say it's very useful. Meanwhile, 39% say it has been somewhat useful. The data is based on a survey of more than 10,000 U.S. adults conducted March 13-19, 2023.

Going deeper

"Rise of AI: Is Your Company Prepared for Generative AI?" is a new episode of the Wharton School's Ripple Effect podcast. Professor Rahul Kapoor explains why now is the time for business leaders to develop new frameworks to manage the changes ahead.

Leaderboard

Julia Brau Donnelly was named CFO at Pinterest, Inc. (NYSE: PINS), effective June 20. Donnelly will be taking on the role from Todd Morgenfeld. As previously announced, Morgenfeld will transition from Pinterest to pursue new career opportunities on July 1. Donnelly joins Pinterest from Wayfair, where she was most recently VP and global head of finance and accounting. During her more than seven-year tenure, she held several positions of increasing responsibility within the finance function. She led a global team of 250 employees across all of accounting and finance, including strategic finance, investor relations, corporate development, FP&A, accounting, tax and finance operations. Before Wayfair, she was a private equity investor in technology and media companies at Thomas H. Lee Partners in Boston.

Yafei (Roxi) Wen is resigning from her position of CFO at Invitae (NYSE: NVTA), a medical genetics company, effective June 30. Wen is pursuing other opportunities; the company has initiated a search for a new CFO. Wen will continue in her role through the end of the second quarter. Christine Gorjanc, longtime chair of the audit committee of the board of directors, will assume the role of interim CFO, effective July 1. Wen's resignation is not the result of any disagreement with the company on any matter related to operations, policies, or procedures, according to Invitae.

Overheard

"You're not going to fix these things if you are just sitting across the Pacific yelling at each other. So, I'm hoping we have real engagement."

—JPMorgan Chase CEO Jamie Dimon said on Wednesday during the JPMorgan Global China Summit in Shanghai, Reuters reported. Simon was answering a question about diplomatic relations between China and the U.S. and emphasized the need to have "real engagement" to resolve their security and trade matters.