The upcoming tax season is likely to be the worst in years -- for the IRS and potentially for taxpayers.

The reason: The IRS budget keeps getting smaller, and the tax code keeps getting more complicated.

Nina Olson, the national taxpayer advocate, said at a conference recently that she worries it could rank as the worst since 1985, which was regarded as a disaster due to a computer failure, lost returns and delayed refunds.

The budget: Congress has shrunk the IRS budget over the past five years, while at the same time requiring the agency to administer even more complex laws.

The IRS topline budget for 2015 is about 10% less than it was in 2010. But the real drop is actually steeper since the 10% doesn't account for cost increases that have occurred in the past five years.

During roughly the same period, the number of IRS personnel has fallen by at least 8%. And the amount of money the agency has for staff training has dropped by more than 85%.

Meanwhile, the number of taxpayers has grown by about 7 million people, IRS Commissioner John Koskinen noted at the same conference where Olson spoke.

End result: Come January 20, when the IRS will start accepting returns, there will be less taxpayer service and, Koskinen warned, possibly delayed refunds.

Olson, a watchdog for taxpayers, estimates that 47% of the calls coming into the IRS probably won't be answered during the filing season. The other 53% of people lucky enough to get through will have to wait an average of 34 minutes to talk to a human being.

Related: How IRS drives taxpayers crazy

Tax preparers calling on behalf of their clients may use a "priority service" line. But it will take them about 52 minutes to talk to someone, Olson said. "So my advice is to bring your knitting while you sit on the phone trying to get through."

Koskinen said the IRS will do all it can to keep everything running smoothly. But the agency will be hamstrung.

"All we can try to do is maximize the resources available in that January to May time frame to make sure that ... we do as well as we can. And 'as well as we can' is still going to be miserable," he said.

But if that misery includes seriously delayed refunds -- still a very big if -- that's when average Americans may take note of just how under the gun the IRS is. And they won't like it.

"If the refund doesn't come, that's going to make them angry. This is dysfunction right in your face," said Christopher Bergin, president and publisher of Tax Analysts.

More work and complexity despite less money: The upcoming tax season will be the first when the IRS will have to administer the premium tax credits and individual mandates under the Affordable Care Act.

Firsts are never perfect, of course. But given how controversial health reform has been, any hiccups in the administration of Obamacare will be broadcast loudly by politicians, the press and frustrated tax professionals.

One concern Olson has: The IRS won't receive accurate information from the insurance exchanges. "The IRS will probably get blamed for a lot of things that it is not responsible for, and that will not do well on the Hill as we try to make our case for additional funding," she said.

It will also be the first tax filing season when the IRS will be administering the Foreign Account Tax Compliance Act. FATCA is intended to recapture money the United States loses because of unpaid taxes on Americans' overseas accounts.

Olson worries the process won't be seamless for U.S. ex-pats trying to reconcile their tax bills given the new reporting requirements. "Who are they going to call? There is no toll-free number to call overseas and you're certainly not going to sit there for 34 minutes overseas letting the clock tick on a toll call."

The ramifications of budget cuts amidst a heavier workload goes beyond 2015, Koskinen said. "Any marked deterioration in taxpayer service and enforcement creates a long-term risk for the U.S. tax system, which is based on voluntary compliance."

In other words, Americans, who have one of the best compliance rates in the world, may not be so willing to oblige if they can't get the help they need, or if they know they won't get caught if they cheat.

"If the compliance rate goes down by 1 percent, it costs the government $30 billion a year. That's almost three times the entire budget of the IRS," Koskinen noted.