Back in 2004, Justice Antonin Scalia was participating in a panel discussion at NYU Law School when the subject of his famously scathing dissent in Lawrence v. Texas came up. One year earlier, the Supreme Court had ruled in Lawrence that the Constitution prohibits criminalizing sodomy between consenting adults, overturning recent precedent and saying, effectively, that gay men and lesbians couldn’t have their sex lives treated as criminal activity.

As he often is when the subject relates to gay rights, Scalia was furious, writing: “so imbued is the Court with the law profession’s anti-anti-homosexual culture, that it is seemingly unaware that the attitudes of that culture are not obviously ‘mainstream’; that in most States what the Court calls ‘discrimination’ against those who engage in homosexual acts is perfectly legal.”

When the time came for audience questions, a student named Eric Berndt stood up in the packed auditorium and pressed Scalia to explain his dissent, particularly his opinion as to whether it was constitutional for the government to peer into the bedrooms of consenting adults and punish them for what goes on in there. When Scalia did not answer to the student’s satisfaction, Berndt asked him, “Do you sodomize your wife?”

It was a breathtaking moment (Scalia simply glowered at Berndt without responding), and it triggered days of news coverage and weeks of hot-blooded recriminations at law schools across the country. After the shock of the question had worn off, some criticized Berndt for being so disrespectful of an august public official, but a larger number of students spoke out in his defense.

In retrospect, that exchange, and the response to it, revealed a key weakness in Scalia’s continued stance regarding gay rights in America.

The justice has pushed that stance to the forefront again in recent days. His latest intemperate remarks—comparing laws against homosexuality to laws against murder—have whipped up the predictable firestorm, but he’s made comments like this so often that some are asking whether he ought to recuse himself from participating in the Supreme Court’s upcoming decisions on the matter.

The short answer is no. The longer answer is more complicated.

As a Supreme Court justice, Scalia isn’t legally bound by the rules of judicial conduct that apply to all other federal judges, so the fact that he can’t even pretend to maintain an open mind on the issue of gay rights doesn’t matter, and we’re left with nothing more than a plea to Scalia that he recognize he’s no longer capable of behaving judiciously on this topic. (If you think that plea has any chance of success I have a duck blind I’d like to sell you.)

And what about those rules of conduct? They require judges to avoid the appearance of impropriety, defined as “when reasonable minds, with knowledge of all the relevant circumstances disclosed by a reasonable inquiry, would conclude that the judge’s … impartiality … is impaired.” It’s a legalistic way of saying that if a judge repeatedly spouts off in public forums about his or her views on an issue before the court, he or she cannot be relied on to be impartial when confronted with a case involving that issue.

You could say this describes Scalia and gay sex/marriage to a T, but this whole kerfuffle over his apparent biases is a red herring. All judges are human and have biases, of course, and anyone who demands that judges maintain the appearance of impartiality is in effect demanding that judges appear to be something that they’re not, and indeed could not possibly be.

Scalia’s real problem isn’t that he’s biased—as the exchange with Berndt revealed, it’s that he’s increasingly out of touch with the basic moral sentiments of the society over which he passes his increasingly cranky moral judgments. It’s obvious from polls (not to mention voting trends) that in another generation or so laws against same-sex marriage will be considered, by the vast majority of Americans, to be as bizarre and unjust as laws against, say, interracial marriage are considered today.

Consider that in 1958, when the Gallup Poll first asked the question, 4 percent of Americans approved of interracial marriage. Last year 86 percent did. Similarly, over the past 15 years the percentage of Americans who approve of same-sex marriage has nearly doubled, from 27 percent to 53 percent. Far more ominous for the foes of marriage equality, among adults under 30, its support stands at 73 percent.

Now, of course, Scalia would claim that for the Supreme Court to pay attention to public opinion polls is a crime against jurisprudential nature. This may well be—but like many other so-called crimes against nature, it is an exceedingly common one. For many decades the American right wing has made a veritable fetish out of the idea that the court is in the hands of a decadent cultural elite imposing its taste for various perversities on the public at large.

This fantasy bears little relation to the banal reality, which is that the court both reflects and is shaped by public opinion, although in a somewhat chaotic and highly inefficient way. For example, it’s hardly a coincidence that the court began to take an interest in same-sex marriage only after that concept was well on its way to being institutionalized by the broader political process—not to mention publicly endorsed by the president of the United States.

In this light, Scalia’s tactless fulminations are, at bottom, a reminder of why life tenure for Supremes is a bad idea, the badness of which increases in direct proportion to our average life expectancy. Put another way, someone who was in law school at a time when 96 percent of the public disapproved of interracial marriage should be considered too old to sit on the Supreme Court.

While in theory it might be preferable to have a political system in which unelected judges don’t decide fundamental public-policy issues, that’s never been the American system, and it never will be in the foreseeable future. And since we are fated to be ruled, in part, by the political and legal biases of Supreme Court justices, we should at least do what we can to ensure that not all of those justices are cranky old men.