ChatGPT can boost or tank employee performance, depending on what tasks it’s used for

 A new study from Boston Consulting Group’s Henderson Institution think tank and academics at four top business schools, including Harvard Business School and MIT Sloan School of Management, provides some early insight into the benefits of generative AI when used for the right tasks—and the destruction it can create when used for the wrong ones.

Researchers experimented on more than 750 BCG consultants, assessing ChatGPT-4’s efficacy in assisting them on two different tasks. The first task, “creative product innovation,” asked the test subjects to brainstorm ideas for a new product, develop a business case, test and launch the product, and create a marketing campaign. The second task, “business problem solving,” asked subjects to fix a fictitious company’s challenges, assess business performance data, and analyze an interview transcript with company executives.

The findings show a 40% performance boost for consultants using the chatbot for the creative product project, compared to the control group that did not use ChatGPT, but a 23% decline in performance when used for business problem-solving. 

The chatbot likely performed better with the creative project because it’s easier for large language models (LLMs) to develop innovative, novel, or valuable ideas due to the large swath of data they’re trained on. Meanwhile, the chatbot decreased business problem-solving performance because LLMs are more prone to error when analyzing quantitative and qualitative data to solve complex problems.

The researchers found that ChatGPT enhanced the performance of nearly all (90%) consultants who used it for the creative product assignment. But that boost was more prominent among low-performing employees, who saw a 43% performance increase when using the chatbot versus a 17% increase for top-performing employees.

There was also a 41% decline in the diversity of thought when using generative AI for the creative product assignment because ChatGPT churned out similar responses to prompts.

For HR leaders, the findings underscore the need for strategic workforce planning, namely, the human capabilities a company will need and how AI can best drive that performance.

“We’re just scratching the surface of this revolution. And this is why it’s fascinating to be a CHRO at the moment because you will be in a position to really lead on [human-AI interactions],” says Francois Candelon, global director of BCG’s Henderson Institute. “To experiment, to understand how it works, who I should hire, what capabilities I need over time, and so on. So it’s a lot of work for CHROs.”

Paige McGlauflin
paige.mcglauflin@fortune.com
@paidion

Reporter's Notebook

The most compelling data, quotes, and insights from the field.

BCG’s research also found that test subjects who received a simple 30-minute training on ChatGPT, such as how to write prompts and its limitations, had worse individual performance decline in the business problem-solving exercise than those who did not receive training. The reason? People became overconfident in the chatbot’s results.

But that doesn't mean no training is the best option; employees just need more in-depth training. “We need to make people fully aware of the limitations," Candelon says.

Around the Table

A round-up of the most important HR headlines.

- More people of color have been promoted to executive positions since the summer 2020 racial reckoning. In 2021, the S&P 100 added more than 300,000 jobs, 94% of which went to people of color. Bloomberg

- An Amazon SVP apologized to employees at an internal meeting for imperfect return-to-office protocols and asked that employees who felt rushed to relocate report the incident to human resources. The executive now believes a complete return to the office could take about three years. Insider

- Companies experimenting with four-day workweeks are seeing positive returns, such as happier employees and more interest in job opportunities. But, some employees say that fitting five days of work into four can be stressful. Wall Street Journal

- Joe Biden became the first sitting president in modern history to join a picket line yesterday during a visit to striking auto workers at a GM facility in Michigan. The president said he backed the United Auto Workers' demand for a 40% pay raise. Reuters

Watercooler

Everything you need to know from Fortune.

New work friend. Economist and Stanford University professor Erik Brynjolfsson says using AI to complete some tasks and collaborate with employees could cause worker productivity to double. —Geoff Colvin

Abandoned cash. Meta is paying £149 million ($181 million) to prematurely end its lease on an eight-story office building in London because there aren’t enough employees to fill it. The early exit from the building, which is just one of Meta’s four London offices, could be a result of its recent layoffs. —Eleanor Pringle

Bridging the gap. Women pay over $15 billion more than men yearly on out-of-pocket medical expenses. But for roughly $133 per employee per year, employers can eliminate this disparity. —Erin Prater