“We’re all behind him now,” Senate Majority Leader Mitch McConnell warned, adding that it’s time for unifying the party, not “settling scores and grudges.” “I hope he’ll change his direction.”

So far, Trump has refused, reiterating in interviews broadcast Sunday that U.S. District Judge Gonzalo Curiel’s Mexican heritage means he cannot ensure a fair trial involving a billionaire who wants to build a border wall to keep people from illegally entering the United States from Mexico. Curiel was born in Indiana to Mexican-born parents—making him, in Trump’s view, “a hater of Donald Trump.”

“I couldn’t disagree more” with Trump’s central argument, McConnell said on NBC’s “Meet the Press.”

“I don’t condone the comments,” added Sen. Bob Corker, chairman of the Senate Foreign Relations Committee, on ABC’s “This Week.”

And Newt Gingrich, who became speaker of the House promising to open the GOP more to minorities, delivered the harshest warning of all.

“This is one of the worst mistakes Trump has made. I think it’s inexcusable,” Gingrich, a former presidential contender, said on “Fox News Sunday.”

Their remarks solidify the line GOP leaders have drawn in recent days between themselves and Trump, with whom they’ve made a fragile peace over their shared sense that almost anyone would be a better president than Democrat Hillary Clinton.

The GOP pushback against Trump comes two days before presidential primaries in California, home to more Latinos than whites. It’s the final major battleground between Clinton and Vermont Senator Bernie Sanders. Far ahead of Sanders in the delegate race, Clinton is poised to clinch her party’s nomination in the next few days.

Trump has no more competition for the GOP nomination, but he does have significant issues with the most senior elected members of the party he hopes to lead.

On Thursday, House Speaker Paul Ryan tepidly endorsed Trump—but 24 hours later disavowed the billionaire’s remarks about Curiel.

Trump University is the target of two lawsuits in San Diego and one in New York that accuse the business of fleecing students with unfulfilled promises to teach secrets of success in real estate. Trump has maintained that customers were overwhelmingly satisfied. Trump’s legal team has not sought to have Curiel removed.

Trump already has rejected calls for him adjust his approach.

“I’m not changing,” he said Tuesday at a fiery news conference at Trump Tower.

On Sunday, Trump doubled down on the idea. Asked on CBS whether a Muslim judge would be unfair given Trump’s plan to ban Muslims from the U.S, Trump responded: “Yeah. That would be possible, absolutely.”

For a party that in 2012 explicitly pinned its survival on drawing support from Hispanics, Trump’s words create an ugly series of headaches.

Asked three times whether Trump’s attack on Curiel was racist, McConnell thrice refused to respond directly and repeated a statement about disagreeing.

“I think it’s a big mistake for our party to write off Latino Americans,” said McConnell, a Republican from Kentucky.

Gingrich answered: “I think that it was a mistake … I hope it was sloppiness. [Trump] says on other occasions that he has many Mexican friends, et cetera, but that’s irrelevant. This judge is not Mexican. This judge is an American citizen.”

Corker, a Republican from Tennessee, expressed the same discomfort many other Republicans in Congress have complained about when they’re asked to respond to, or justify, Trump’s remarks. “I thought this interview was going to be more about the foreign policy arena,” Corker said on ABC.

Like Ryan, all three Republicans have endorsed Trump. But their comments carried the implicit caveat that their support depends at least in part on Trump dropping his criticism of Curiel. All three also suggested ways Trump could move beyond his legal issues.

Corker, who recently met with Trump in New York, said Trump “has a tremendous opportunity” to build out his foreign policy agenda.

Gingrich urged Trump to become more of a statesman.

“Trump has got to, I think, move to a new level,” he said. “This is no longer the primaries. He’s no longer an interesting contender. He is now the potential leader of the United States and he’s got to move his game up to the level of being a potential leader.”

McConnell’s advice was blunt.

“This is a good time, it seems to me, to begin to try to unify the party and you unify the party by not settling scores and grudges against people you’ve been competing with,” he said. “I’d like to see him reach out and pull us all together and give us a real shot at winning this November.”