A parade of business executives broke ties with Trump on Wednesday, a day after he blamed white nationalists and counter-protesters in equal measure for the weekend clashes that left one woman dead.

Now, frustrated aides could be next. Trump’s remarks have left some wondering if sticking by the president comes at too high a cost to their reputations.

“A lot of us joined this administration thinking we could bring to it the experience and expertise that the president didn’t have an opportunity to gain in his business career, and to encourage some restraint in what he says publicly and to our allies,” said one senior official who is contemplating whether to resign.

“After yesterday, it’s clear that there is no way for anyone, even a Marine general, to restrain his (Trump’s) impulses or counter what he sees on TV and reads on the web.”

It was hoped that retired General John Kelly, Trump’s new chief of staff, could impose some form of discipline on Trump that his predecessor, Reince Priebus, could not.

But Kelly stood with his eyes fixed on the floor when Trump veered off-script at his Manhattan office tower on Tuesday. The president accused the protesters, who rallied against neo-Nazi and white supremacist groups in Charlottesville, of being “very, very violent.”

In the uproar that followed, chief executives at companies such as Merck & Co Inc, Under Armour Inc, Intel Corp, Campbell Soup Co and 3M Co quit advisory councils to the White House. Trump then dissolved the councils.

The exodus of executives sparked talk that Gary Cohn, Trump’s top White House economic adviser and a key liaison to the U.S. business community, might resign in protest as well.

Cohn, who is Jewish, was upset by Trump’s remarks, though he is remaining with the administration for now, sources said.

Cohn, along with Treasury Secretary Steven Mnuchin and Transportation Secretary Elaine Chao, stood by Trump during his remarks at Trump Tower on Tuesday. Cohn in particular looked self-conscious and uncomfortable.

“He just did not want to be in that position … and he was not good at hiding his body language for that,” said a former administration official who knows Cohn.

Cohn did not comment on the president’s language.

David Shulkin, U.S. secretary of Veterans Affairs, told reporters on Wednesday that as a Jewish American, he was “outraged” by neo-Nazis and other white supremacist groups and felt obligated to speak out against them.

“I am not going to condone in any way the behavior of Nazis. I believe this clearly cannot be tolerated,” Shulkin said when asked about whether it was appropriate to compare the actions of the white nationalists to the protesters opposing them.

Shulkin, however, defended Trump’s approach. “I think he’s been clear that this is totally unacceptable,” Shulkin said.

Cohn, who came to the White House from a successful career at Goldman Sachs Group Inc, is mindful the effect his Trump tenure could have on his professional reputation.

“He’s worried about his reputation being trashed, which is much more valuable to him than anything else,” the former administration official said.

Cohn has served as a point man on top White House priorities such as tax reform and rebuilding the nation’s infrastructure, but both of those efforts have been muddled by Trump’s increasingly combative relationship with Congress, one that was strained even further by his comments on Charlottesville.

Cohn’s departure would further set back those efforts and perhaps give the upper hand in the White House to a group of advisers seeking to scale back foreign trade, said a Wall Street executive who asked not to be named.

“Gary knows he’s a moderating influence,” the executive said. “It may give you short-term satisfaction to see Gary go, but it may be bad for the country in the long term. The calculation is: What do you think is best for the country versus what’s best for Gary?”

Steve Bannon, a White House senior adviser with close ties to far-right groups, told the American Prospect in an interview published Wednesday that he constantly butts heads with Cohn over issues such as trade with China. “That’s a fight I fight every day here,” Bannon said.

Cowan and Company, a financial services firm, said on Wednesday that the departure of the pragmatic and business-friendly Cohn could adversely affect markets.

“For us, the biggest question is what is the tipping point that would cause National Economic Council Director Gary Cohn to quit?” the firm wrote.

Trump thinks highly of Cohn and has spoken often of the financial sacrifices he made to leave Goldman to join the administration. He is widely considered to be a leading candidate to chair the U.S. Federal Reserve should Trump choose not to retain Janet Yellen.

That decision would insulate Cohn from the day-to-day drama of the Trump White House, but likely is months away.

In the meantime, Cohn has to decide whether he can stick it out. Another Wall Street executive told Reuters that Trump’s remarks may prove to be too much for him.

“Until yesterday, Cohn did a great job insulating himself from Trump and staying in the economic lane. But all of a sudden he was standing behind him when he goes off on a rampage and the true price of working for him comes home,” the executive said. “What can you do?”