The Internal Revenue Service ordered the popular bitcoin exchange Coinbase to turn over millions of customer accounts this month, leaving many crypto-currencies collectors to worry just what the tax man has in store for them.

On its face, it looks like big trouble. The IRS request was a sweeping one as the agency asked for info about all of the Coinbase customers who bought bitcoin between 2013 and 2015, citing “suspicion [the group] includes U.S. taxpayers who are not complying with the law.”

Of course, some of these Coinbase users have nothing to worry about. For those who didn’t sell their bitcoins, there’s no capital gain and so nothing to pay. And those who declared their gain and paid the taxman are right as rain.

But what about those who bought a few bitcoins and sold them when the price went up? Or those who used their bitcoins to buy goods and services online? Some of these people may not be aware that because bitcoin is considered property not currency, every bitcoin purchase can amount to a capital gains event.

In theory, the IRS pay soon demand back taxes and penalties from thousands or even millions of Coinbase customers—and then go after other bitcoin sellers like Circle or Xapo next.

Get Data Sheet, Fortune’s technology newsletter.

It turns out, though, it probably won’t go down like that. According to Marco Santori, a crypto-currency lawyer at Cooley LLP, the scope of the IRS investigation will be much more modest. Speaking at a Coindesk event in New York last week, Santori noted that Coinbase is refusing the initial IRS demand as too broad, and that a judge is unlikely to enforce the request as it stands.

“This is a blanket request—it’s an opening offer [by the IRS]. They have to come in with something crazy, then agree to limit it something reasonable,” he said.

Santori predicted that, after a legal tussle, the IRS will scale back its demand and instead ask Coinbase to turn over just the bigger fish among its clients. He added that everyone must pay their taxes, and that bitcoin owners should consult their accountants if they are worried about gains for previous years.

On Wednesday, a judge granted a request by the IRS for a “John Doe” summons that requires Coinbase to turn over all the accounts. But Coinbase says it will mount a legal challenge to the order — meaning the IRS has cleared an initial hurdle but that Coinbase may still succeed in limiting the scope of the order.

“We are aware of, and expected, the Court’s ex parte order today,” Coinbase said in a statement. “We look forward to opposing the DOJ’s request in court after Coinbase is served with a subpoena. As we previously stated, we remain concerned with our U.S. customers’ legitimate privacy rights in the face of the government’s sweeping request.”

Meanwhile, those who took a loss face a different problem. As bitcoin authority Ryan Selkis points out, the maximum loss that can be captured is $3,000 for a single year since bitcoin is technically property not a currency. And given bitcoin’s wild ride between 2013 and 2015, when the price ranged from $13 to over $1,100, surely there were some investors who lost a lot more than that.

This story was updated on Dec. 1 to include the Coinbase statement and news of the court granting the summons.