(CNN) The Supreme Court struggled on Wednesday with a dispute in a voting rights case concerning whether Ohio's method of removing names from its rolls violates federal law.

Voting rights groups have charged it is a veiled attempt at voter suppression.

At oral arguments, several of the justices seemed to side with an attorney for the battleground state who argued the system is necessary to ensure the accuracy of voter rolls.

Liberal Justice Stephen Breyer, as well as Chief Justice John Roberts and Justices Samuel Alito and Anthony Kennedy, pushed the lawyer for the challengers.

"What should a state do?" Breyer asked. "Every year a certain number of people die and every year a certain number move."

The case comes as President Donald Trump was forced to dissolve a controversial commission that had been launched to combat what the President claims is a problem of voter fraud, though many election law experts say there is no widespread evidence that such a problem exists. It also comes between elections, but the court's decision could impact close races down the road.

In Ohio, voters who have not engaged in voter activity for two years are sent address confirmation notices. If a voter returns the notice through prepaid mail or responds through the Internet, his or her information is updated. If the notice is ignored and the voter fails to update a registration over the next four years, the registration is canceled.

Larry Harmon, a voter in the state, challenged the process, arguing that he had been removed from the rolls even though he had not moved. He had simply opted not to vote in 2009 and 2010. When he showed up at the polls in 2015, he was told his registration had been canceled. He claimed no recollection of receiving a confirmation notice from the state and he later sued, along with two public interest groups, the Northeast Ohio Coalition for the Homeless and the A. Philip Randolph Institute.

In September 2016, a federal appeals court ruled that Ohio's so-called "supplemental process" violates a federal law called the National Voter Registration Act. The law says a state cannot remove a voter from the rolls "by reason of the person's failure to vote." The panel of the Sixth US Circuit Court of Appeals ruled that 7,515 ballots that had been struck could be cast in the last election.

Ohio Attorney General Mike DeWine appealed the decision to the Supreme Court, defending the process by noting that it targets people who have failed to respond to a notice, not those who have failed to vote.

"The Sixth Circuit's decision makes it harder for states to conduct what all can agree is a critical activity -- removing ineligible voters from registration lists -- by eliminating one method for doing so," Ohio State Solicitor Eric E. Murphy argued in court papers.

That argument appealed to the conservative justices at the Supreme Court on Wednesday as well as Breyer, who said he thought that Congress would not have passed a statute that would "prevent" the states from updating their rolls.

Kennedy said the reason that states purge voters is "they want to protect the voter roll from people that have moved and they're voting in the wrong district. That's the reason. What we're talking about are the best tools to implement that reason."

Alito agreed the issue was "important" and "sensitive" but said the court's job was to interpret the statute at hand. He was the most vocal critic of the challengers' position and told their lawyer, Paul M. Smith, that Ohio's system does not say that the failure to vote is the only ground for removal, instead saying, "moving out of the district is a ground for removal." Roberts noted that Ohio's process is not just triggered by a voter's failure to vote -- it is coupled with the notification process which serves the mandates of the federal law.

But the liberal justices fired back hard, grilling Murphy. Justices Elena Kagan and Sonia Sotomayor asked so many questions that Roberts gave the lawyer more time. Kagan said Ohio's process "explicitly" relies on non-voting. Sotomayor questioned whether sending an address notification is a "reasonable" way of determining that someone has moved and she asked if the process "might result in disenfranchising disproportionately certain cities where large groups of minorities live."

"I don't understand how you can say that the failure to vote can be used as the sole basis for sending out notices," she said.

Sotomayor asked why the state couldn't use other methods to determine whether someone had moved, such as DMV records or post office notices.

And Kagan and Sotomayor were very critical of a lawyer for the government, Solicitor General Noel Francisco, who argued in support of Ohio's position. Sotomayor noted that Francisco had switched sides from the position taken by prior Justice Departments.

"There's a 24-year history of solicitors general of both political parties under both -- presidents of both political parties who have taken a position contrary to yours," she said, adding it "seems quite unusual that your office would change its position so dramatically."

Francisco acknowledged the "prior" position and argued in briefs that the "NVRA does not prohibit a state from using nonvoting" as the basis to send an address-verification notice. He said "that conclusion is supported by the NVRA's text, context and history."

The court's ruling could directly affect similar policies in six states, and 17 states have filed a brief in support of Ohio.

Lawyers for the states argue that another provision of the voter registration act requires each state to conduct a program that "makes a reasonable effort" to remove people who have moved or passed away, and they need the Supreme Court to clarify how states can accomplish the goal. They interpret the federal law as allowing Ohio's system.

In a conference call last week, voting rights groups urged the justices to uphold the lower court's opinion.

They argued that a "key tenet" of the NVRA is that voting "is not a use-it-or-lose-it" right and said the notices sent out were based on a voter's failure to vote.

"Only seven states do what Ohio is doing here," said Dale Ho, director of the Voting Rights Project at the American Civil Liberties Union. He said the vast majority of other states find ways to make sure their voter lists are up to date by using tax records, returned mail or DMV change-of-address forms to determine whether someone may have moved.

"People who only vote in presidential elections will be purged if they miss a single election," said Stuart Naifeh, senior counsel at Demos.

He said Ohio's process targets lower income voters and those of color and penalizes "voters who may already face obstacles in exercising the right to vote," and that "it disproportionately harms the most vulnerable voters."