Phoenix (CNN) -- On Wednesday, critics called for the resignation of the controversial Arizona sheriff known for his tough stand on illegal immigration, alleging he misspent millions of dollars in taxpayer money.

Auditors say Maricopa County Sheriff Joe Arpaio used nearly $100 million designated for jail funds to pay deputies' salaries, CNN affiliate KNPX reported.

Arpaio has called the April findings from a county budget office audit a "payroll discrepancy issue," and pledged to work with his employees to fix the problem "without any direct impact to the taxpayer," KNPX said.

But Mary Rose Wilcox, a supervisor on the Maricopa County Board of Supervisors, said that wasn't enough.

Speaking at a news conference Wednesday along with a group of community activists, Wilcox urged Arpaio to resign and called on the Department of Justice to take administrative control of the county's jail system.

"Time after time, you have fiscal problems, abuse problems of civil and human rights and you have abuse from within, corruption from within," Wilcox said. "The sheriff cannot sit there and not take responsibility for that."

Lisa Allen, a spokeswoman for Arpaio, said the sheriff declined to comment. She added that he was not going to resign.

Arpaio, who has earned the nickname of "America's toughest sheriff," drew national attention for his tough stand on illegal immigration.

Many of his prisoners are housed in tents and forced to wear pink underwear, and he once boasted of feeding them on less than a dollar a day per prisoner.

He also faces a Justice Department investigation into whether his policies cracking down on illegal immigrants discriminate against Hispanics.

Journalist Valeria Fernandez contributed to this report.