Budget cuts, reduced staffing and an increase in the number of tax returns will likely mean longer waits on the phone and slower processing of filings, Nina E. Olson, the national taxpayer advocate, said in a report Wednesday.

“Taxpayers who need help are not getting it, and tax compliance is likely to suffer over the longer term if these problems are not quickly and decisively addressed,” Olson wrote.

Her prediction is ominous sign for taxpayers, many of whom already have a low opinion of the IRS. Many filers rely on the IRS help lines to quickly handle questions, trouble shoot problems and smooth out what is almost universally considered a convoluted tax system.

The IRS job is huge. It handles 150 million tax returns by individuals and approximately 100 million phone calls per year.

In 2004, a high point in the agency’s customer service, staff answered 87% of taxpayer calls. Those that got through waited on hold for an average of 2.5 minutes.

This time around, Olson expects a far more frustrating experience. Call response rates may be as low as 43%, with those who do get through on the phone needing to wait for as long as 30 minutes.

Olson, who is employed by the IRS, but represents the interests of taxpayers, cited a lack of staffing as a significant factor this year. Over the past few years, the IRS has reduced its workforce by 12,000 employees with more cuts slated for this year.

“The only way the IRS can assist the tens of millions of taxpayers seeking to speak with an IRS employee is to have enough employees to answer their calls,” Olson said. “The only way the IRS can timely process millions of taxpayer letters is to have enough employees to read the letters and act on them. And the only way the IRS can meet the needs of the millions of taxpayers who visit its walk-in sites is to have enough employees to staff them.”

She called for Congress to work with the IRS to ensure there’s a solution going forward. A big part of that is getting more workers back at the IRS and pumping up the budget.

“We need to recognize that the IRS and its employees play a vital role in the economic welfare of this country,” Olson said. “And we need to find a way to support the agency even as we hold it accountable for what is often a thankless task.”

She continued: “We do not think it is acceptable for the government to tell millions of taxpayers who seek help each year, in essence, ‘We’re sorry. You’re on your own.'”

[fortune-brightcove videoid=3648838412001]