(CNN) -- Federal authorities are moving to rein in the man dubbed "America's Toughest Sheriff" after complaints that immigration raids by his deputies amounted to unconstitutional roundups of Latinos.

Maricopa County Sheriff Joe Arpaio supervises an inmate relocation in Phoenix, Arizona.

Sheriff Joe Arpaio and the Maricopa County, Arizona, sheriff's department have had an agreement with the Department of Homeland Security since 2007 that allows his department to enforce federal immigration laws. But Arpaio says the federal agency is moving to revise the agreement to limit that power to checking the immigration status of inmates already in his Phoenix jail.

Arpaio has cultivated his image as "America's Toughest Sheriff," a nickname earned by his treatment of Maricopa County inmates. Many of his prisoners are housed in tents and forced to wear pink underwear, and he once boasted of feeding them on less than a dollar a day.

'Latino in America' The Latino population is set to nearly triple by 2050. Soledad O'Brien journeys into the homes and hearts of a group destined to change the U.S. Witness the evolution of a country as Latinos change America and America changes Latinos.

October 21 & 22, 9 p.m. ET see full schedule »

Now he faces a Justice Department investigation into allegations of civil rights abuses, and the American Civil Liberties Union of Arizona is suing the sheriff over immigration raids conducted by his department. The class-action lawsuit alleges that Arpaio has abused the power delegated to him under his agreement with Homeland Security, known as the 287(g) program.

"He's unconstitutionally acted to racially profile many persons in the community, persons who appear or are Latino," ACLU lawyer Dan Pochoda told CNN. Pochoda said the five-term sheriff has launched a series of high-profile raids to appeal to "his anti-immigration base."

Arpaio told CNN's "American Morning" the allegations were "garbage" and said he would continue to use state laws to crack down on undocumented immigrants in his county.

"We do not go on street corners and grab people that look like they're from another country," he said. "Pursuant to our duties, when we come across illegals, we take action." Watch what Arpaio has to say »

Homeland Security Secretary Janet Napolitano, who was Arizona's governor before taking her Cabinet post, told CNN that Arpaio is reacting prematurely to decisions that have not yet been finalized. But Arpaio says he's now become the poster boy of the emotionally charged immigration debate and is losing authority for political reasons.

"They don't want to aggravate the Hispanic community, aggravate the businesspeople who hire the illegals, and they want amnesty," he said.

Arpaio said he planned to continue his operations "with no changes."

"We do enforce traffic violations. All law enforcement does that, and comes across some serious criminals, which we do in our crime suppression," he said. "We go into workplaces under the state law, and we arrest people in the workplace there illegally with identity theft situations and human smuggling."

But according to a 2008 policy report on effective law enforcement by the Goldwater Institute, a libertarian-leaning watchdog group based in Phoenix, Arpaio's department "falls seriously short of fulfilling its mission." The report found that Maricopa County has "diverted resources away from basic law-enforcement functions to highly publicized immigration sweeps, which are ineffective in policing illegal immigration."

CNN Correspondent Thelma Gutierrez contributed to this report.

All About Joe Arpaio • Immigration • Maricopa County