Story highlights Lawsuit threat comes after Florida began removing non-eligible voters from its voter rolls

The state identified more than 100,000 names of non-eligible voters

Critics say the plan unfairly targets minorities and uses inaccurate information

In the latest volley in the ongoing battle over Florida voter lists, the Justice Department has sent a letter stating it will take legal action against the state, citing violations of voting rights laws.

"Because the State has indicated its unwillingness to comply with these requirements, I have authorized the initiation of an enforcement action against Florida in federal court," said Thomas Perez, assistant attorney general for civil rights.

The lawsuit comes after the department began questioning the legality of the state's so-called voter purge program, which would remove names from Florida's voter rolls months before the 2012 presidential election, when Florida will play a key role as a battleground state with a large chunk of electoral votes.

Florida's move to eliminate non-eligible voters from its lists began after the state's Republican governor, Rick Scott, pressed the state to identify non-U.S. citizens who had registered to vote illegally.

Using information from Florida's Department of Highway Safety and Motor Vehicles, the state identified more than 100,000 names of non-eligible voters that could potentially be on the lists illegally.

JUST WATCHED Voter purge? Fla. respectfully disagrees Replay More Videos ... MUST WATCH Voter purge? Fla. respectfully disagrees 05:37

JUST WATCHED Scott wants voter purge done 'right' Replay More Videos ... MUST WATCH Scott wants voter purge done 'right' 02:41

JUST WATCHED Scott: Voter purge is non-partisan Replay More Videos ... MUST WATCH Scott: Voter purge is non-partisan 01:29

Critics say the plan unfairly targets minorities and paint it as an attempt to dissuade typically Democratic voters from going to the polls.

While the Justice Department notes that states can legally remove non-eligible voters from their lists, the letter Monday argues that the Florida program does not comply with legal standards, has "critical imperfections, which lead to errors that harm and confuse eligible voters."

The letter states that the Florida program improperly uses the information collected from old driver's license applications.

"The information is often going to be outdated, as a number of persons will subsequently have become citizens and lawfully registered to vote," Perez wrote.

It also suggests that the removal process has been going on during a "90-day quiet period" prior to the August 14 primary election.

"Please immediately cease this unlawful conduct," the letter states.

Florida Gov. Rick Scott, a Republican, told Fox News Monday that the state plans to sue the Department of Homeland Security to get access to the federal citizenship database so that it can better determine who is voting legally.

"I have a job to do to defend legitimate voters of our state," Scott said on Fox News.

Three of the state's largest counties -- Miami-Dade, Broward and Palm Beach -- agreed last week to end the removal of the names. The legal counsel for Florida's county election officials recommended halting the purge of names until the state responded to the federal government's legal assessment.

Florida's Republican-appointed secretary of state, Ken Detzner, has previously defended the state's practice, slamming the federal government for restricting access to federal citizenship databases and saying such a constraint was illegal.

"It is an unfortunate but now undeniable fact that Florida's voter rolls include individuals who are not citizens of the United States. The Florida Department of State has a solemn obligation to ensure the integrity of elections in this State," Detzner wrote in the letter last week addressed to T. Christian Herren, head of the Justice Department's elections unit.