If the Supreme Court’s landmark decision on same-sex marriage is a slippery slope, where does it lead?

The most common—and often ridiculed—hypothetical that critics of the ruling cite is polygamy. If a marriage between two men or two women is suddenly a constitutional right, why can’t a group of people decide to wed? Yet to Justice Samuel Alito, the consequences of the high court decision go far beyond marriage. In a lengthy and revelatory interview with Bill Kristol of the Weekly Standard published Monday morning, Alito suggested that the concept of “liberty” envisioned in Justice Anthony Kennedy’s majority opinion could lead to a parade of potential horribles, and some of them might be just as concerning to progressives as they are to conservatives:

And there are many other conceptions. The Court’s conception, I said in this opinion and I believe to be true, is a very postmodern idea; it’s the freedom to define your understanding of the meaning of life. Your—it’s the right to self-expression. So if all of this is on the table now, where are the legal limits on it? If a libertarian is appointed to the Supreme Court, is it then proper for the libertarian to say, “Well, I think that there is a right to work less than the minimum wage? I think there is a right to work as many hours as I want without being limited by the government. I think I have the right to build whatever I want on my property irrespective the zoning laws and so forth.” If a socialist is appointed to the Supreme Court, can the socialist say, “I think that liberty and the 14th Amendment means that everyone should have a guaranteed annual income or that all education through college should be absolutely free,” or whatever. There’s no limit.

The entire interview runs more than an hour, but it’s worth watching for Alito’s extended commentary on the marriage case and the rare light it sheds on a justice who generates significantly less discussion than his conservative colleagues on the bench (particularly Chief Justice Roberts, Antonin Scalia, and Kennedy, the frequent swing vote). More predictable in his voting than Roberts or Kennedy, and less colorful than Scalia, Alito has sat on the Court for nearly a decade but is perhaps best known to the general public for shaking his head and mouthing “Not true” when President Obama criticized the Citizens United decision during his State of the Union address in 2010. (One noteworthy tidbit from the interview is that Alito remembers his father personally drawing up new legislative districts for New Jersey after a 1964 Supreme Court ruling created the standard of “one person, one vote”; the Court will consider that principle anew next term.)

Alito expands significantly on his dissent in the marriage case (one of four total), which touched on his slippery-slope concerns but focused more on the traditional definition of marriage and fears that the ruling would be used to “vilify” those who support it. If there’s a theme to Alito’s voting record on the court, it’s that he’s no libertarian. As he explained to Kristol, he was on the opposite side of the slippery-slope argument in a pair of earlier First Amendment cases, when he dissented from the high court’s expansive reading of the freedom of speech. In one case, Alito was the lone dissenter in a decision striking down a federal law banning the sale of so-called “crush” videos depicting animal cruelty. And he was again alone in opposing the Supreme Court’s 8-1 ruling in favor of Westboro Baptist Church, the anti-gay group that protests at military funerals. In the interview, Alito explained that those restrictions were “peripheral” to the core “political speech” principle of the First Amendment—unlike, say, the question in Citizens United of whether financial contributions constituted political speech, a decision that he emphatically defended.