On Monday night, the company announced that it would go ahead with its Alibaba spin-off, disregarding the fact that the IRS refused to say whether it would consider the spin-off tax free. Even so, Yahoo has said it believes it won’t have to pay Uncle Sam. And it has obtained a 100-page opinion from its outside law firm, Skadden, Arps, Slate, Meagher & Flom, written by a former top IRS lawyer to back up its claim. Independent tax expert Robert Willens told Fortune that he, too, thinks the spin-off will avoid taxation.

But one important group remains skeptical: Yahoo shareholders. Shares of the company rose just $0.88, or 3.2%, on the news of the spin-off. “People remain unconvinced, no question,” says Willens. “I would have expected the stock to increase a lot more than it has.”

The Yahoo-Alibaba saga has been somewhat of a slow motion train wreck for Yahoo CEO Marrisa Mayer. Yahoo holds 384 million shares of Alibaba, which is what’s left of an investment it made in the Chinese online retailer from years ago. It would like to sell those shares. But if Yahoo were to just go ahead and sell its stake, like any investor, it would owe capital gains taxes. So Yahoo (YHOO) came up with a clever way to avoid paying that bill: It decided to spin-off its Alibaba stake into a separate company, with existing shareholders getting the shares. Spin-offs are tax free as long as they have a business purpose. So, Yahoo said that it would spin off a division of its company that focuses on small business services along with its Alibaba (BABA) shares, creating a new company called Aabaco. However, Yahoo made it clear the small business unit it was shedding was not a very big operation. It was clear that the primary purpose of Aabaco would be to hold Yahoo’s Alibaba’s shares.

The deal got the attention of the IRS. And the regulator has been grumbling about the deal since at least mid-May. Earlier this month, the IRS said it wouldn’t bless the deal. An IRS official also recently said the agency is looking at closing the loophole that makes deals like Yahoo tax-free.

Since Yahoo has owned its Alibaba stake for a while, it’s hard to know what value investors put on it. That would give you a sense of how much they think Yahoo’s eventual tax bill might be now. Yahoo’s market cap rose by about $10 billion to just over $49 billion last October when the company started talking about the tax-free spin-off. Assume that jump was essentially the gain investors expected from not having to pay taxes. Since then, as the prospects of a tax-free spin off have dimmed, Yahoo’s shares have slumped, and its market cap is now $27 billion, or $22 billion less than what it was back in October. Alibaba’s shares have dropped as well during that time, erasing $14 billion from the value of Yahoo’s stake in the company. Yahoo also owns 35% of Yahoo Japan, and that stake has risen slightly in the same time, about 2%, which would tack another few hundred million on to the net worth of Yahoo.

That means Yahoo shareholders, even after the small jump in the stock price on Tuesday, are still putting Yahoo’s tax bill for its Alibaba shares at slightly more than $8 billion. That’s only a billion less than what Willens, the tax expert, estimates Yahoo’s tax bill would be if it were to abandon the spin-off and sell its shares outright.

Willens, though, thinks investors have got it wrong. He points out that the IRS has said that even if it closes the spin-off loophole, it won’t make its ruling retroactive. And the IRS isn’t likely to act in the next few months. Yahoo has said it plans to complete its spin-off by the end of the year. The IRS could still challenge the spin-off under the current rules and take Yahoo to court to force it to pay taxes. But Willens says Yahoo can point to several other similar deals that have never been challenged by the IRS. He thinks that would make it a tough case for the IRS to win.

Of course, there are some caveats to my math. Others things have happened in the past year. Stocks have fallen in general recently, which means investors may think that Yahoo is simply worth less than it used to be. What’s more, tax bill or not, Aabaco is likely to trade at a discount to Alibaba’s regular shares because of its weird spin-off structure.

Nonetheless, investors are assuming that Aabaco will trade at a big discount, and at least some of that could be on account of a whopper of a tax bill.