Washington (CNN) A deeply divided Supreme Court took up a partisan gerrymander case on Tuesday that could change the way state legislators draw district lines and realign modern day politics.

The key vote, Justice Anthony Kennedy, gave little indication which way he'll side during the often lively session that saw a familiar partisan split.

At one point during arguments, he expressed concern with whether the challengers in the case had the legal right, or "standing," to bring their case to the court. But Kennedy, a conservative who has often sided with the four liberals on the bench, also seemed sympathetic to the idea that an extreme partisan gerrymander might violate First Amendment rights.

While the Supreme Court has a standard limiting the overreliance on race in map drawing except under the most limited circumstances, it has never been successful in developing a test concerning an overreliance on politics.

At issue are maps drawn in Wisconsin after the last census that Democrats say were drawn unconstitutionally to benefit Republicans. They argue the maps represent extreme partisan gerrymander and that they prevent fair and effective representation by diluting voters' influence and penalizing voters based on their political beliefs.

Lawyers for the state defend the maps, saying they are legitimate because legislators considered political implications as only one of several factors. They also argue it's not the job of the courts to write district lines, arguing that the challengers don't have the legal right to be in court and that the justices should decline to decide the issue and instead leave it to the political branches to decide.

A lower court ruled in favor of the challengers, setting out a standard to judge when partisan politics goes too far in map drawing.

It was clear that the liberal justices -- Elena Kagan, Ruth Bader Ginsburg, Stephen Breyer and Sonia Sotomayor -- thought such a challenge should go forward and a workable standard was possible.

At one point, Ginsburg asked, "what's really behind all of this?" and worried about the "precious right to vote." She continued, "if you can stack a legislature in this way, what incentive is there for a voter to exercise his vote?" Sotomayor asked how political gerrymander "helps our system of government." Kagan was deeply critical of the Wisconsin maps, saying they go "over pretty much every line you can name."

Chief Justice John Roberts led the charge for conservatives, including Justice Samuel Alito and Neil Gorsuch, arguing that the courts should not intervene in an issue that should be handled by the political branches.

Roberts worried about the reputation of the court and that if it were to get involved in issues considering partisan gerrymander it would make the court look political. "That is going to cause very serious harm to the status and the integrity of the decisions of this court in the eyes of the country."

He said the "main problem" for him was that if the court stepped in, other claims from across the country, "every one," would come to the court. And he questioned the social science relied upon by the challengers, calling it "gobbledygook." He narrowed in on the threshold issue of standing, saying he thought it was "arresting" that the challengers could bring a statewide challenge instead of district-by-district.

Gorsuch pushed on what standard the court could use to decide the cases -- "what is the formula?" he asked. Alito also chimed in on whether the court should take over from the political branches. "Is this the time," he asked, for us to "jump in?" Justice Clarence Thomas asked no questions, but he is expected to side with the conservatives.

When they were asked to take up the case last June, Wisconsin asked them to put on hold the lower court opinion that went against the state. The court agreed, but the liberal justices went out of their way to note that they would have denied the application for a stay. Over the summer, Ginsburg told an audience at a Duke Law school event that the Gill v. Whitford case was perhaps "the most important grant so far."

Photos: Supreme Court Justice Anthony Kennedy Anthony Kennedy, the longest-serving member of the current Supreme Court, has announced that he will be retiring at the end of July. Kennedy, 81, was appointed by President Ronald Reagan in 1988. He is a conservative justice but has provided crucial swing votes in many cases. Hide Caption 1 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy was born in Sacramento, California, on July 23, 1936. In this photo, circa 1939, he sits between his mother, Gladys, and his sister, Nancy. Hide Caption 2 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy wears his Cub Scout uniform as he poses with his brother, Tim, circa 1946. Hide Caption 3 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy, third from right in the front row, stands with other Cub Scouts in the 1940s. Hide Caption 4 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy, right, spent time with the California Army National Guard after finishing law school in 1961. The man on the left, John J. Hamlyn Jr., also became a lawyer like Kennedy. Hide Caption 5 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy, right, and Hamlyn pose for a photo after basic training. Hide Caption 6 of 38 Photos: Supreme Court Justice Anthony Kennedy After more than a decade as a lawyer, Kennedy became a judge on the US Court of Appeals in 1975. He was nominated by President Gerald Ford on the recommendation of California Gov. Ronald Reagan. Hide Caption 7 of 38 Photos: Supreme Court Justice Anthony Kennedy This courtroom photo of Kennedy was taken in 1976. Hide Caption 8 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy has breakfast with his wife, Mary, and his son Gregory in 1984. Hide Caption 9 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy and his wife walk together in Sacramento, California, in 1987. Hide Caption 10 of 38 Photos: Supreme Court Justice Anthony Kennedy From 1965 to 1988, Kennedy was also a professor of constitutional law at the University of the Pacific's McGeorge School of Law. Hide Caption 11 of 38 Photos: Supreme Court Justice Anthony Kennedy In 1987, Kennedy was nominated by President Reagan to fill the Supreme Court seat vacated by Lewis Powell's retirement. The nomination came after the confirmation failures of nominees Robert Bork and Douglas Ginsburg. Hide Caption 12 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy, center, talks with US Sens. Ted Kennedy, left, and Joe Biden before a confirmation hearing in Washington. The two Kennedys are not related. Hide Caption 13 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy meets with President Reagan in the Oval Office. Hide Caption 14 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy is joined by his wife as he is sworn in by Chief Justice William Rehnquist on February 18, 1988. Reagan is on the right. Hide Caption 15 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy, top right, appears in a formal Supreme Court portrait in April 1988. In the front row, from left, are Thurgood Marshall, William Brennan Jr., Chief Justice William Rehnquist, Byron White and and Harry Blackmun. In the back row, from left, are Antonin Scalia, John Paul Stevens, Sandra Day O'Connor and Kennedy. Hide Caption 16 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy speaks at the McGeorge School of Law in 1991. He delivered the inaugural address in a lecture series named for the late Archie Hefner, whose portrait is behind Kennedy. Hefner was a prominent Sacramento attorney active in numerous civic and charitable groups. He died in 1988. Hide Caption 17 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy is on the far right in this Supreme Court portrait from 1998. In the front row, from left, are Antonin Scalia, John Paul Stevens, Chief Justice William Rehnquist, Sandra Day O'Connor and Kennedy. In the back row, from left, are Ruth Bader Ginsburg, David Souter, Clarence Thomas and Stephen Breyer. Hide Caption 18 of 38 Photos: Supreme Court Justice Anthony Kennedy In 2004, Kennedy speaks to high school students at the Ronald Reagan Presidential Library in Simi Valley, California. Hide Caption 19 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy speaks during a Senate subcommittee hearing in 2002. Hide Caption 20 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy discusses the court's budget requests with a House committee in April 2005. Hide Caption 21 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy participates in a panel discussion in Washington in November 2005. Hide Caption 22 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy receives an honorary degree at New York University in May 2006. Hide Caption 23 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy delivers the commencement address at New York University. Hide Caption 24 of 38 Photos: Supreme Court Justice Anthony Kennedy In February 2007, Kennedy testifies at a Senate committee hearing on judicial security and independence. Hide Caption 25 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy testifies before a House subcommittee in March 2007. He and fellow Justice Clarence Thomas spoke about concerns with the ongoing remodeling of the court building, the reduction of paperwork due to electronic media, and the disparity of pay between federal judges and lawyers working in the private sector. Hide Caption 26 of 38 Photos: Supreme Court Justice Anthony Kennedy The Supreme Court meets with President Barack Obama and Vice President Joe Biden in September 2009. From left are Samuel Alito, Ruth Bader Ginsburg, Kennedy, John Paul Stevens, Chief Justice John Roberts, Obama, Sonia Sotomayor, Biden, Antonin Scalia, Clarence Thomas, Stephen Breyer and retired Justice David Souter. Hide Caption 27 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy leaves after a Catholic Mass in Washington in October 2009. Hide Caption 28 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy joins the President and other officials at a memorial for the victims of a shooting in Tucson, Arizona, in 2011. Hide Caption 29 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy smiles as he is introduced to faculty members at the University of Pennsylvania Law School in October 2013. Kennedy was teaching there for a week. Hide Caption 30 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy is saluted by sailors as he tours the USS John C. Stennis in 2015. Hide Caption 31 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy testifies about a Supreme Court budget request during a House subcommittee meeting in 2015. Hide Caption 32 of 38 Photos: Supreme Court Justice Anthony Kennedy President Obama greets Kennedy and other Supreme Court justices before his final State of the Union address in January 2016. Hide Caption 33 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy, second from left, joins other Supreme Court justices in February 2017 during President Donald Trump's first address to a joint session of Congress. Hide Caption 34 of 38 Photos: Supreme Court Justice Anthony Kennedy As President Trump looks on, Kennedy administers the judicial oath to new Justice Neil Gorsuch in April 2017. Hide Caption 35 of 38 Photos: Supreme Court Justice Anthony Kennedy Kennedy and Trump walk together after Gorsuch's swearing-in ceremony. Hide Caption 36 of 38 Photos: Supreme Court Justice Anthony Kennedy Trump stands with the Supreme Court at Gorsuch's formal investiture ceremony in June 2017. From left are Elena Kagan, Samuel Alito, Ruth Bader Ginsburg, Kennedy, Chief Justice John Roberts, Trump, Gorsuch, Clarence Thomas, Stephen Breyer and Sonia Sotomayor. Hide Caption 37 of 38 Photos: Supreme Court Justice Anthony Kennedy Several members of the Supreme Court pose for a portrait before taking part in a procession to mark Harvard Law School's bicentennial in October 2017. On the top row, from left, are Kennedy, Roberts, Breyer and Gorsuch. In front of them are Kagan and retired Supreme Court Justice David Souter. Hide Caption 38 of 38

Although Kennedy did not tip his hand, he had no questions for a lawyer challenging Wisconsin.

"It's the most important case involving the structure of American politics in a generation," said Jeffrey Rosen, president of the National Constitution Center, at a recent event hosted by George Washington University Law School. The question: Are extreme partisan gerrymanders violations of either the equal protection clause or possibly the First Amendment?

In a 2004 partisan gerrymander case, conservatives on the court said they thought the issue should be handled by the political branches. But Kennedy was unwilling to bar all such future claims because he thought a workable standard to measure when there is an overreliance on politics might arise down the road.

"Partisan gerrymanders have become more common, more severe and more durable in their effects," Paul Smith, a lawyer for the Campaign Legal Center who is representing the challengers, argued in briefs. He said that partisan gerrymanders result in a violation of the "Equal Protection Clause by discriminating against the targeted party's voters" and preventing them from "fair and effective representation."

Part of the problem, he argued, is the fact that modern day technology has allowed better map-drawing that has "wrested power from voters." Using mapping software, census data and voting algorithms, he alleged the Republican map drawers diluted democratic votes by packing them into districts.

Erin Murphy, a lawyer at Bancroft who has filed an amicus brief on behalf of the Wisconsin State Senate and Wisconsin Assembly in support of the state, said that if the courts step into the battle it will transfer to judges a power that belongs with the political branches.

"Allowing claims like plaintiffs' to proceed would therefore wrest control over the districting process away from the state legislators to whom state constitutions assign the task, and hand it to federal judges, opportunistic plaintiffs and social scientists seeking to convert academic theories into constraints on the democratic process," she said.

A lower court ruled in favor of the challengers after conducting a trial in 2016. The court relied upon a three-part legal standard based on discriminatory intent, discriminatory effect and the justification for the law. The court concluded that the evidence established that one of the purposes of the redistricting plan -- Act 43 -- "was to secure Republican control of the Assembly under any likely future electoral scenario for the remainder of the decade, in other words to entrench the Republican Party in power."

Former President Barack Obama and former Attorney General Eric Holder have both taken up the cause. In a speech last July in Raleigh, North Carolina, Holder vowed to "ensure that after the next census in 2020 lines are drawn in a fair way -- not a partisan way -- to ensure that people's wishes are reflected in the candidates who are elected."

But the issue is not just one for Democrats. Arnold Schwarzenegger, the former Republican governor of California who sat in on Tuesday's arguments, called gerrymandering an "incumbent protection program" in a recent conference call.

A similar case is brewing in Maryland, but it was brought by Republicans.

And in a brief signed by Arizona Republican Sen. John McCain, lawyer Mark W. Moser asked the court to step in an adopt a standard that would block extreme partisan gerrymandering and stop special interest groups from infecting the map drawing process.

"Special interest groups, fueled by hidden funders with deep pockets and skin in the political game, are now focused on influencing redistricting," Moser wrote. "The payoff for these groups is obvious: By shaping the decennial redistricting process, special interest groups can affect the outcome of every congressional race in a state for the next decade."

JUST WATCHED Schwarzenegger on the Gerrymandering problem Replay More Videos ... MUST WATCH Schwarzenegger on the Gerrymandering problem 06:28

CORRECTION: A previous version of this story misstated the results of the 2014 Wisconsin legislative election.