Most everyone at the Supreme Court on Tuesday agreed that the First Amendment protects a public employee in a non-political job from being fired or demoted for supporting a political candidate.

But in a case alternatively described as “bizarre” and more “like a law school hypothetical,” the question was what happens when the boss retaliates against an employee on the mistaken belief that the employee has asserted such support.

By the end of the hour-long argument, it seemed clear that Paterson, N.J., police officer Jeffrey Heffernan would have had a better case if he had endorsed his friend running for mayor rather than maintained his neutrality.

“The First Amendment talks about abridging freedom of speech, and I thought the case came to us on the proposition that he wasn’t engaging in speech at all,” Chief Justice John G. Roberts Jr. said, adding: “I’m not sure how he can say his freedom of speech has been abridged.”

On the other hand, Justice Elena Kagan warned that such a reading would be a threat to “every couch potato out there” — politically agnostic or uninterested government workers who might have no constitutional protection if their bosses set out to replace them with workers with more aligned political views.

“If somebody had come in to me before today’s argument and just said ‘Does the First Amendment prevent the government from punishing a person because that person does not share the government’s views?,’ I would have said, ‘Why, yes, of course the First Amendment protects that,’ ” Kagan told lawyer Thomas C. Goldstein, who was representing the city of Paterson.

“Now you’re telling me, no, the First Amendment does not prevent the government from punishing a person because that person doesn’t share the government’s views, unless that person is actively opposed to the government’s views.”

Even Goldstein acknowledged that Heffernan presented a “very sympathetic claim.”

He was a detective in the police department, assigned to a division headed by the police chief. His boss supported the incumbent mayor in the 2006 election; Heffernan was a close friend of former police chief Lawrence Spagnola, who was the challenger.

One day, while on his own time, Heffernan was observed by another officer picking up a yard sign at the Spagnola campaign headquarters. The sign was not for him, Heffernan explained later, but for his bed­ridden mother (Heffernan didn’t even live in Paterson and was ineligible to vote.)

Nevertheless, Heffernan was demoted to patrol officer the next day and told it was because of his “overt” involvement in a political campaign.

The case has bounced around lower courts for years — Heffernan won once, only to have the judgment overturned. Finally, the U.S. Court of Appeals for the 3rd Circuit ruled that Heffernan could not receive damages for a First Amendment violation when he had not actually exercised First Amendment rights.

The court’s conservatives seemed to agree that Heffernan might have a legitimate beef but that he could not make a constitutional case out of it.

Justice Antonin Scalia was among the most outspoken. “The First Amendment guarantees the right to freedom of speech and freedom of association,” Scalia said. “Your client was neither speaking nor associating. So how could he possibly have a cause of action under the First Amendment?”

There might be statutes that protect against unfair retribution, Scalia said, “but there’s no constitutional right not to be fired for the wrong reason, and that’s what happened here.” (Actually, Heffernan was only demoted.)

The justices across the board expressed some irritation with Heffernan’s attorney, Mark Frost of Philadelphia, when they asked whether there were New Jersey statutes that Heffernan could use to challenge his demotion.

But Frost insisted that Heffernan was right to assert his constitutional protections, saying “the fact that he actually was not engaged in any political activity should make no difference.”

The Obama administration agreed with Heffernan, and Assistant Solicitor General Ginger D. Anders defined the officer’s rights this way: He “has a First Amendment right not to have adverse action taken against him by his employer for the unconstitutional purpose of suppressing disfavored political beliefs.”

Anders acknowledged that would be an expansion of what the court has previously recognized, but said it would not lead to a flood of new lawsuits. (The case did not touch on the Hatch Act, which limits the political activity of federal executive branch employees.)

Justice Anthony M. Kennedy seemed to have a different view of the issue the case presented.

“Would it be fair to the proposition that you are putting before the court to say that you’re asserting the right to be free from government inquiry into an oversight of your views?” Kennedy asked, in one form or another, of all three lawyers.

Goldstein encouraged the justices to focus on the case in front of them: “He may have a state law right; he does have a collective-bargaining-agreement right, but he doesn’t have a First Amendment right because he’s not engaging in First Amendment-protected activity.”

But Justice Stephen G. Breyer said he was concerned that the demotion would chill the speech of others who might want to take a position. Justice Ruth Bader Ginsburg said it defied common sense that a person who spoke out would be protected from demotion and one who did not would have no case.

Justice Samuel A. Alito Jr. said that Heffernan’s case might be what the court likes to call a “one-off.”

“How often will it be the case that an employee will be unable to allege any expression or any association that is protected by the First Amendment?” he asked. “It seems to me quite rare.”

The case is Heffernan v. City of Paterson.