Washington (CNN) During heated Supreme Court arguments Tuesday over whether the justices should take up challenges to politically gerrymandered voting districts, Chief Justice John Roberts confessed what matters to him.

It comes down to how things look and what people think.

What would the public think, he asked, if the justices began hearing challenges to partisan gerrymandering and ruled for Democrats, or conversely, Republicans?

"If you're the intelligent man on the street," Roberts said, "and the court issues a decision, and let's say the Democrats win, and ... that person will say, 'Well, why did the Democrats win? ... It must be because the Supreme Court preferred the Democrats over the Republicans.'"

"And that," the chief justice asserted, "is going to cause very serious harm to the status and integrity of the decisions of this court in the eyes of the county."

During an hour of arguments over redistricting and voting rights in a case that could have a dramatic political impact for elections nationwide, Roberts put special emphasis on how the court itself would be regarded if it began scrutinizing state legislators and their voting districts.

In public speeches, Roberts consistently insists that the justices -- five Republican-appointed conservatives and four Democratic-appointed liberals -- are not political players. But his remarks in the middle of an intense hearing over gerrymandered districts in Wisconsin offered the sharpest statement to date of his thinking -- and perhaps motivations -- regarding the institution that informally bears his name.

It is questionable how effective he might be, in these highly polarized times and with the Roberts Court's own record of 5-to-4 votes in politically incendiary cases. His concern about the "court in the eyes of the country" may fall on deaf ears outside the marble columned building, and possibly even within.

His remarks suggest, in fact, that he might in upcoming months continue to distance himself from newest Justice Neil Gorsuch, who, since his April confirmation, has not shied from flashing his conservatism in written opinions and off-bench activities.

A threshold question in Tuesday's dispute over a 2011 Wisconsin redistricting plan that a lower court deemed an unconstitutional partisan gerrymander is whether federal judges should even review partisan claims arising from the inherently political process.

Democratic state challengers say the sophisticated computer and data analytics used by the Republican majority in the state capitol gave the GOP a disproportionate advantage and diluted Democratic electoral influence throughout the state. Challengers say the partisan gerrymander was so extreme that it violated the Fourteenth Amendment equality guarantee and First Amendment rights related to party affiliation and views.

State officials said they relied on traditional redistricting principles and contend that courts should stay out of the politically charged map-drawing that follows each 10-year census.

The justices' comments from the bench suggest they would be tightly divided, with swing-vote Justice Anthony Kennedy poised to cast the crucial vote. Kennedy appeared open to First Amendment arguments related to the harm to voters distributed throughout districts based on political views.

The four liberal justices -- Ruth Bader Ginsburg, Stephen Breyer, Sonia Sotomayor and Elena Kagan -- appeared ready to allow legal challenges to gerrymanders that disproportionately favor one party over another. The three justices on the far right, Gorsuch, Clarence Thomas and Samuel Alito, seemed ready to block such lawsuits.

Chief Justice Roberts fell in with the latter group but partly for reasons related to the court's public image that only he voiced. A 2005 appointee of GOP President George W. Bush, Roberts has voted along conservative lines. He opposed a constitutional right to gay marriage, for example, and has voted against affirmative action and other racial remedies. In 2012, however, he surprised some observers when he cast the deciding vote to uphold President Barack Obama's health care overhaul.

He was blasted by many Republicans, including New York businessman Donald Trump, then four years from his presidential bid. Political complaints about the court have ramped up as Washington has become increasingly politicized.

Roberts tells audiences that the court is above such politics. "We in the judiciary do not do our business in a partisan, ideological manner," he said at an appearance earlier this year.

During arguments in Gill v. Whitford on Tuesday, he described perceptions about the court's role as his "main problem" in the case.

"If the claim is allowed to proceed, there will naturally be a lot of these claims raised around the country," Roberts told lawyer Paul Smith, arguing on behalf of the challengers. "And every one of them will come here for a decision on the merits. ... We will have to decide in every case whether the Democrats win or the Republicans win."

Roberts scoffed at the idea that people would believe the court relied on the challengers' proposed formula tied to "partisan asymmetry" in Democratic and Republican election results.

"The intelligent man on the street is going to say that's a bunch of baloney," Roberts said. And, for the court, he added, such a dilemma would recur "as these cases are brought in every state."

Smith said the risk would be greater if the court fails to become involved.

"It may be that you can protect the court from seeming political," he said, "but the country is going to lose faith in democracy big time because voters are going to be like ... voters in Wisconsin (saying) ... 'It really doesn't matter whether I vote.'"