The U.S. government is recalling 46,000 IRS workers to the office, without pay, to handle tax refunds.

The IRS has been operating with a skeleton staff since the shutdown began Dec. 22, but it announced last week that it would issue tax refunds despite the closure. Earlier on Tuesday, Treasury Secretary Steven Mnuchin told a Bloomberg News reporter he expects refunds to go out on time during the shutdown.

There had been growing concern that the shutdown would delay refunds going out because the money wouldn't be available for them from Congress.

Get Breaking News Delivered to Your Inbox

Last week, the administration said customary shutdown policies will be reversed to make the money available and refund payments on time possible.

The National Treasury Employees' union, which represents some 70,000 IRS workers, has filed two lawsuits against the government, alleging that it's illegal to force people to work unpaid.

"There is no doubt the IRS needs to get ready for the 2019 filing season that starts Jan. 28, and IRS employees want to work," NTEU National President Tony Reardon said in a statement. "But the hard, cold reality is that they've already missed a paycheck and soon they'll be asked to work for free for as long as the shutdown lasts."

The IRS said that while it would have some staffers answering phone lines, "taxpayers should be prepared for longer wait times." In addition, in-person taxpayer assistance centers are still closed, and people with appointments at the IRS "should assume their meetings are cancelled."