Washington (CNN) The Georgia Secretary of State this week removed hundreds of thousands of registered voters it classified as "inactive" from its voting rolls, amid a legal challenge from a voting rights group to block the purge of a portion of the names.

The removal comes as part of a new state provision signed into law earlier this year. Under the provision, the state must remove registration records from the voter rolls that have been deemed "inactive" for more than three years. A voter is categorized as "inactive" if they don't vote in two general elections and have had no contact with board of elections in that time, according to Secretary of State Brad Raffensperger's office

About 313,000 voters were removed from the list, or about 4% of all registered voters in the state, according to the Secretary of State. The "inactive" voters were marked for removal after failing to respond to a pre-addressed, postage paid confirmation card within 30 days; the card asked voters to confirm or update their information.

State officials mailed out notices to the last known address of voters and posted the list of people online.

Fair Fight Action, a voting rights organization founded by Democrat Stacey Abrams, filed an emergency motion in federal court on Monday challenging the removal of a portion of the list -- 120,000 voters who were removed due to inactivity since the 2012 presidential election.

The group is in the midst of a federal case against Raffensperger over the so-called "use it or lose it" policy, which allows registrations to be canceled after voters fail to participate in elections for several years.

"In our view, it is a First Amendment right not to vote, and it is unconstitutional to take away a Georgian's right to vote simply because they have not expressed that right in recent elections," Fair Fight Action spokesman Seth Bringman said. "309,000 is the total purge. This also includes people who had returned mail, passed away, or informed the state that they moved. In our view, these are appropriate reasons for a Georgia voter to be removed from the rolls, but 'use it or lose it' is not."

The group went before US District Court Judge Steve C. Jones to fight the move on Monday afternoon. The judge allowed the removal to go forward and will hear arguments from the state and Fair Fight Action on Thursday.

The policy is meant to keep an accurate and fair count during elections and also help with planning of polling site equipment during election cycles, according to Georgia state officials.

According to Fair Fight Action's Bringman, the Secretary of State said it did not have the technical capacity to stop the purge from taking place, but that it will reinstate the voters if ordered to do so.

Jones echoed this policy, telling CNN that though the list has been revised, the information of "inactive" voters is still in the state database and could be made "active" through re-registration or an update of address on a state drivers license.

This isn't the first time the state has faced controversy surrounding voter list updates. In July 2017, more than half a million registered voters, or 8%, were removed from the rolls; of that total, more than 107,000 were removed for inactivity in recent elections, according to a report from APM

The state also came under fire in October 2018 when then-Secretary of State Brian Kemp -- then the Republican nominee for Georgia governor -- enforced a policy of "exact match." Under the policy, the most minor discrepancy, like a typo or missing letter, between a voter's registration and their drivers license, Social Security or state ID cards was flagged, leading to more than 53,000 voter applications being put on hold. The majority, seven out of 10, belonged to African-Amercans.

Ahead of the recent removal, Georgia had roughly 7.4 million registered voters. Under the new roll count, there are about 7.08 million registered voters. According to Jones, an estimated 460,000 new registered voters were added within the 2018 election cycle, which included the tight race for governor in which Abrams lost to Kemp by a mere 68,000 votes

Correction: This story has been updated to accurately reflect the number and percentage of voters who were removed from Georgia's voter rolls.