Washington (CNN) The Supreme Court on Monday appeared sharply divided in a case pitting organized labor against conservative groups in a case that could deal a potentially crippling blow to public sector unions -- and hurt their Democratic Party allies.

The court's four liberals seemed to back the unions' position. Conservatives, including Chief Justice John Roberts, Anthony Kennedy and Samuel Alito, appeared ready to rule against the unions and maybe even overturn the court's 41-year-old precedent. Justice Clarence Thomas didn't speak but he is expected to rule against the unions.

But conservative Justice Neil Gorsuch did not ask one question.

At the center of the debate is a 1977 Supreme Court opinion known as Abood v. Detroit Board of Education that says while non-members of public sector unions cannot be required to pay fees for a union's political activities, they can be required to pay so-called "fair share" fees pertaining to issues such as employee grievances, physical safety and training.

The Abood decision was a careful compromise when it came down 41 years ago, but in recent years, critics of unions have pushed for it to be overturned.

Today, 22 states have laws on the books that allow broad fair share fees for public employees.

Mark Janus, an Illinois public sector employee, is challenging the fees. He says that because he is a government employee, issues germane to collective bargaining are inherently political. He argues that the First Amendment protects him from having to support such political expression.

Justice Elena Kagan led the charge Monday in favor of the public unions' position expressing concern that if the Supreme Court were to take the rare step and overturn precedent, thousands of contracts impacting millions of employees could be in jeopardy.

"Those contracts probably cover millions, maybe up to over 10 million workers," the liberal justice said. "When have we ever done something like that?"

She was joined by Justice Ruth Bader Ginsburg who worried that overruling Abood would "drain" the public sector union's resources and even trigger members to stop paying for something others are getting for free. She said the impact could create "free riders" who could benefit from a union's representation but not have to pay for it.

Justice Sonia Sotomayor said that the challengers were basically arguing to "do away" with unions.

Gorsuch silent

Back in 2016, the justices heard a similar challenge and seemed poised to overturn Abood, but then-Justice Antonin Scalia died after oral arguments and the court announced a 4-4 split. On Monday, all eyes were on Scalia's replacement, Gorsuch, who could cast the deciding vote, but he stayed silent.

Janus is represented by groups such as the National Right to Work Legal Defense Foundation and the Liberty Justice Center, who argue that approximately 5 million public sector employees are required -- as a condition of their employment -- to "subsidize the speech of a third-party that they may not support, namely the government-appointed exclusive representative."

In court, lawyers for Janus appeared to draw the support of the conservative bloc. Alito was the most vociferous and seemed deeply concerned about the First Amendment implications of requiring employees like Janus to pay the fees.

"When you compel somebody to speak, don't you infringe that person's dignity?" he asked.

Kennedy expressed concern that even fees that go to collective bargaining for issues such as higher wages and working conditions could be highly political in nature.

"Doesn't it blink reality to deny that is what's happening here?" he said.

Jacob H. Huebert, one of Janus' lawyers, said his client specifically does not support the union's advocacy for increased spending, especially given Illinois' current fiscal condition.

"A lot of people in the private sector in Illinois are struggling with an increased tax burden and a stagnant economy and Mark Janus doesn't think it's fair to put more demands on his neighbors at a time like this," Huebert said.

Lee Saunders, president of the American Federation of State, County and Municipal Employees, believes that the case boils down to corporate interests attacking the rights of workers.

"When working people are able to join strong unions, they have the strength in numbers they need to fight for the freedoms they deserve, like access to quality health care, retirement security and time off to work for a loved one," he said in a statement.

The unions point out that they are required by law to represent all employees regardless if they are members and that no one is required to join the union. They say that if non-members don't have any obligation to pay fair share fees for the collective bargaining obligations, they would become free riders, benefiting from the representation without sharing the costs. In addition, the coffers of public sector unions would suffer if non-members were able to get services for free.

They say the 40-year-old Supreme Court decision has left the decision in the hands of the states to decide how to proceed.

It's an argument supported by some Republican state lawmakers who have filed a brief supporting the unions and arguing the decision should be left to the states.

"As things stand, states can determine for themselves which rules work best for their work force and their communities, the court should not use the First Amendment to impose a single rule across the country that harms the ability of public sector workers to bargain together as unions," said Elizabeth Wydra, president of the Constitutional Accountability Center, which represents the lawmakers.