His comments come as the legal battle over the 93-year-old Redstone’s $40 billion media empire heats up. Some investors unhappy with the pace of improvement at Viacom hope that change at the top is imminent.

Gabelli had shown more patience, saying as recently as May that Dauman had six months to turn the company around. But the billionaire investor’s latest comments, to Reuters on Wednesday, mark the first time he has said the CEO is out of options.

“He has to leave,” Gabelli said of Dauman in an interview. “It’s a matter of when, not if.”

Since the Redstones own 80% of the voting shares of Viacom (VIAB) and CBS through their privately owned National Amusements, Gabelli said, Dauman is fighting a battle he is unlikely to win.

Dauman contends in a Massachusetts lawsuit that Redstone is being manipulated by his daughter, Shari Redstone, and through a Viacom spokesman said: “The truth about Mr. Redstone’s wishes remain very much in dispute.”

In the litigation, Dauman is contesting the removal of him and Viacom director George Abrams from the Sumner M Redstone National Amusements Inc Trust, which eventually will control the fate of Viacom and CBS (CBS). The lawsuit says Redstone is incompetent and being manipulated by Shari Redstone.

A spokesman for Redstone declined to comment.

Viacom, which owns television networks Nickelodeon and Comedy Central—as well as film studio Paramount Pictures—has been struggling to improve its ratings as younger viewers watch more content online.

A spokesman for National Amusements said in a statement Thursday that with a more independent board and a new CEO, it believes Viacom would become “an entertainment powerhouse.”

“Viacom owns some of the best assets in the entertainment industry,” the spokesman said.

Viacom’s shares ended regular trading on Thursday up 4%. The stock is up 10% since Redstone removed Dauman from the trust in late May.

The judge in the Massachusetts case is expected to rule in the next few days on whether he will allow that case to proceed in his court or in California.

Redstone has also moved to oust Dauman and four other directors from the Viacom board. Viacom’s lead independent director, Fred Salerno, has filed a lawsuit contesting that move in a Delaware court. Oral arguments on that case are scheduled to begin July 29.

Gabelli said he did not understand why the board and CEO would fight what he sees as their inevitable departures.

“Let’s say I don’t understand the dynamics that are occurring,” he said. “She (Shari Redstone) has 80% of the vote and we have 10%.” He declined to comment on what he would like to see happen to Viacom.

Even if Paramount’s “Star Trek,” movie does well in theaters when it comes out later this month, it will not matter for Dauman, Gabelli said.

“It’s irrelevant,” Gabelli said. “Either way he leaves.”