FORTUNE — Cuddly Beanie Babies have become a bizarre icon of tax horror ever since their billionaire American creator, Ty Warner, was socked last September with jaw-dropping fines for not disclosing his secret Swiss bank accounts.
Now the harshness — and, at times, unevenness — of the Internal Revenue Service in pursuing offshore tax dodgers has an official critic: the ombudsman of the agency.
The Congressionally appointed watchdog, Nina E. Olson, unleashed a fresh critique Thursday of the IRS’s amnesty programs for Americans with undeclared offshore bank accounts. The unmistakable message from Olson, also known as the national taxpayer advocate: Many Americans get screwed over financially by entering the programs.
“The IRS Offshore Voluntary Disclosure Program Disproportionately Burdens Those Who Make Honest Mistakes,” the report says.
The program, comprised of three initiatives since 2009, including one ongoing, is intended to lure tax cheats out of the woodwork and discourage wealthy Americans from hiding assets overseas. The U.S. taxes its citizens and residents on their worldwide income, making cash held in a Swiss bank account liable to U.S. tax — even if the American account owner lives and works outside the U.S.
MORE: So what if the Fed’s profits are down?
Since 2009, some 38,000 people have come forward to pay $5.5 billion in back taxes, fines, and penalties under the initiatives, according to IRS data, in exchange for reduced fines and penalties and, in most cases, avoidance of prosecution.
A third initiative, rolled out in January 2012, continues to bring in people, especially since last August, amid a landmark settlement between the U.S. Justice Department and entire Swiss banking system, home to an estimated $2 trillion in offshore assets. Under that agreement, some 300 Swiss banks — the entire industry, minus some 14 Swiss and Swiss-style banks under criminal investigation by the Justice Department, including Credit Suisse (CS) and Julius Baer — must disclose American account holders to the IRS or risk hefty fines and penalties.
But tax lawyers estimate that perhaps 7 million Americans or more have not filed an annual required disclosure, known as a Report of Foreign Bank and Financial Accounts, or Fbar, to the U.S. Treasury for any foreign account holding more than $10,000 for which the taxpayer holds an interest or signatory power. That means that millions of people could face draconian fines and penalties for offshore tax evasion.
Olson’s report details other challenges to the IRS, which had a tough year in 2013 with budget cuts and a headline-making scandal about the agency’s targeting of conservative tax-exempt groups.
Until 2011, the IRS said that holders of offshore accounts who accidentally failed to file Fbars could pay a penalty capped at $10,000, in addition to back taxes plus interest. Ordinarily, failing to report accounts carries a 20% penalty, computed as a percentage of the back taxes owed to the Treasury.
MORE: 10 economic surprises for 2014
But willful failure to file an Fbar — i.e. knowingly gaming the system — carries much stiffer penalties — 50% of the highest account balance for each year covered. That draconian penalty has left many taxpayers with undisclosed foreign accounts owing more than they actually had in them. The penalty is due even if you don’t actually owe any taxes on the offshore accounts.
The first two amnesty initiatives, in 2009 and 2011, capped the willful failure to file penalty at 20% and 25% of the highest account balances, respectively. But in February 2011, the IRS said it would no longer entertain the “I didn’t know” argument from taxpayers, and instead would treat the failure to file Fbars as willful and intentional, not accidental, leaving in place the higher penalties.
The toll on taxpayers has been hefty.
Olson’s report found that for the 2009 program, launched after a landmark crackdown on Swiss bank giant UBS AG for enabling wealthy Americans to dodge taxes through hidden private accounts, the average taxpayer ended up paying penalties averaging nearly four times, or 386%, of the basic tax owed.
Less wealthy taxpayers fared even worse. Olson found that the bottom 10% of those who entered the 2009 program, as measured by an average account balance of $44,885, got hit with penalties equal to nearly six times the amount of tax owed. Taxpayers wending their way through the program without legal representation got hosed even more, paying penalties of eight times. By contrast, the largest tax dodgers got socked with penalties equal to three times their unreported tax bills — perhaps a sign of good lawyering affordable by the wealthy.
“The penalties are extraordinarily high as compared with most other penalties the IRS administers,” Olson’s report says.
Olson also found new evidence of another trend: Entering an amnesty program but then dropping out in exchange for being audited is a better deal, financially. Such taxpayers ended up paying on average penalties totaling only 70% of the unpaid taxes and interest — hefty, but far less than nearly 600%.
MORE: You may be saving too much for retirement
“Taxpayers who exercised their right to opt out” of programs “after making a disclosure and accept an audit instead fared considerably better than taxpayers who remained in the OVD program.” OVD refers to Offshore Voluntary Disclosure, one of several IRS acronyms for the amnesties.
Many tax lawyers say that taxpayers, and even some tax lawyers themselves, are clueless and simply do not know what an Fbar is, let alone that they need to file one. Think kin of Holocaust survivors who inherited old Swiss accounts; clueless widows who inherited their deceased husbands’ money; American expatriates living and working overseas and unaware of the complex, arcane reporting and disclosure requirements.
“They force everyone into assuming they’re criminals,” says Ryan L. Losi, a certified public accountant and executive vice president at PIASCIK, an accounting firm headquartered in Glen Allen, Va., with an international tax practice. “People are basically having to accept that they willfully non-complied. It’s a terrible deal for the taxpayer.”
Christopher S. Rizek, a tax lawyer at Caplin & Drysdale in Washington, D.C., says that the amnesty has made taxpayers “go through a cost-benefit analysis. And some end up paying huge, unfair penalties, while others are actually deterred from getting into the program.”
The IRS did not have an immediate comment.
Amid the harshness of the penalties, some tax lawyers report an increase in so-called quiet disclosures, in which taxpayers simply file amended or delinquent tax returns and Fbars that include the foreign income, and pay any taxes owed on it, and escape the draconian fines. “The goal is to get in under radar,” said Losi, adding that the Warner case had prompted an increase in “quiet disclosures.”
Warner, the owner of the Four Seasons Hotel in New York and chief executive of Ty Inc., which still makes Beanie Babies, faces up from 46 months to 57 months — nearly four to five years — in jail when sentenced on charges of federal tax evasion on Jan. 14 in Chicago.
Last October, Warner pleaded guilty to concealing some $93.6 million in an account originally held at UBS and then transferred to Zurcher Kantonalbank, a regional Swiss bank under criminal scrutiny by the Justice Department, in 2002. UBS (UBS) outed Warner to the authorities in early 2009 as part of a deal that allowed the bank to avert indictment on charges of enabling wealthy Americans to evade taxes.
Warner’s account, which at one point had up to $107 million, generated nearly $3.2 million in foreign income, which would have carried U.S. taxes of $1.25 million, court papers show. In 2007, Warner filed an amended return for 2002 showing that he owed a reduced $885,000 on the foreign income.
At the time, he still hadn’t been outed to the IRS or come clean. He tried to enter the first amnesty program but was kicked out when UBS turned over his name.
Now Warner, who grew rich despite never attending college, is paying the price. He owes Fbar-related penalties of a whopping $53.6 million — the largest offshore penalty case in U.S. history, according to court papers. The back taxes plus interest come to an additional $6 million, making the Fbar penalty nine times of the tax owed.
While Olson’s report does not mention the Warner case, it criticizes what it calls the IRS’s harsh treatment. “The penalties,” she writes, “are often Draconian and may deter other taxpayers from coming into compliance.”
