During the recent Supreme Court hearing on same-sex marriage, Justice Samuel Alito questioned if finding for the plaintiffs would open the door to marriages of multiple partners. Most considered the rejoinder more quip than question, a reduction ad absurdum of the marriage equality position.

But why can’t four (or more) consenting adults form a family together—perhaps not as a marriage, but as a family unit recognized by the state? Such cases have been brought for decades, especially in the 1970s heyday of communes, and have almost always lost. But a case currently working its way through the Connecticut court system may reopen the issue.

For months now, the city of Hartford, Connecticut, has been in a legal scuffle with a self-described “intentional family” who live together in a large, nine-bedroom home in the West End, a historically wealthy area of the city. The family is composed of three couples, three children, and two single adults, but they are quick to emphasize that they are one family unit.

Maureen Welch is a therapist, part of one of the couples, and a founding member of the group. She says that living in a big, mutually supportive chosen family has been a “life-long dream” for her—one that they’ve been peacefully living out for the last five years at a different address in Hartford.

After another member died two years ago, the “Scarborough family” (as they have come to be called in the papers) decided to leave their old home. “Emotionally we were just ready to move on,” Welch said. (It didn’t help that the old house had only one bathroom for all of them.) When they found the dilapidated mansion at 68 Scarborough Street, it seemed like a dream come true. Not only was the home spacious and well laid out, it had been on the market for over four years, meaning they wouldn’t be displacing existing tenants and they’d be helping to repair an eyesore on the block.

At first, things seemed fine. One neighbor left a gift basket welcoming them, while others invited them over for cocktails and dinners. In fact, the first inkling the Scarborough family had that trouble might be afoot was a letter written to the neighborhood civic association demanding a public hearing about their presence on the block. The letter writers (and others) saw them as a group of unrelated individuals whose presence in the neighborhood—zoned for single-family homes—would open the floodgates to frat houses, rooming places, and other dense and potentially disruptive new neighbors.

Or maybe the problem was something else.

“They think it’s somehow degenerate,” said Josh Blanchfield, another member of the Scarborough family, explaining why some neighbors objected.

Degenerate and unstable, even. “The Zoning Board of Appeals were convinced we were going to fall apart,” added Welch.

Of course, in a country where more than half of all marriages now end in divorce, it’s obvious that permanence does not define family. But this is an easy way to discredit any families that make some people uncomfortable, and similar charges have been laid at the feet of same-sex relationships for decades.

Last fall, the city issued a cease-and-desist order requiring the Scarborough family to leave their home. After mounting and losing an administrative appeal earlier this year, the Scarboroughs have now filed a countersuit in federal court, challenging the city’s definition of family.

What, then, are the reasons for limiting state-recognized families to those based on singles and couples?

It can’t be neighborhood density. The Scarborough family pointed out that the zoning laws allow for an infinite number of children and live-in domestic servants. In part, this points to an economic basis for some neighbors’ negative opinions. Servants? Yes. Socialist-seeming communes? No.

“There’s this sense that we’ve gotten something we don’t deserve,” said Blanchfield.

Nor can it be that non-traditional families are just a smokescreen for roommates to get tax benefits. There are many ways to distinguish between the Scarboroughs and such situations: contracts, commitments, evidence of emotional connection, and the same indicia the state regularly uses when it determines whether a marriage is real or a sham (as in immigration cases, for example).

“We have systems in place to ensure that we are functioning not just as a house but as a collective relationship,” said Blanchfield. Among other things, for the Scarborough family, this includes daily check-ins, cooking dinner together, making life decisions based on each other, and pooling monetary resources to support one another—all the normal stuff families do, just with more people involved.

Although they acknowledge that their situation is rare, the Scarboroughs see the suit against them as “a threat to anyone who’s had people in their home who are not blood but who they care for and love no matter what,” said Blanchfield.

Our legal system has a long history of wrangling around what—other than love—makes a family. Some states allow first cousins to marry, while others do not. Polygamy, although illegal, is practiced by an estimated 100,000 people in the United States, and was written into the theology of the Church of Latter-day Saints. The United States outlawed interracial families right up until the 1960s, and our attitudes toward single mothers (especially those of color) are still highly conflicted.

To many Americans, these prohibitions now seem outdated and discriminatory. Our definition of family has become more of a functional one, based on how the people involved interact, and less of a genetic, procreative, traditional, or otherwise prescriptive one.

“Family is the people with whom we choose to share lives and responsibilities,” said Ricci J. Levy, executive director of the Woodhull Sexual Freedom Alliance, an organization dedicated to “eliminating discrimination based on family structure and relationship choices.”

Levy pointed out that the Universal Declaration of Human Rights (the foundational document for the U.N., which the United States signed and ratified in 1948) states that “the family is the natural and fundamental group unit of society, and is entitled to protection by society and the State.” However, what constitutes a family is not defined in the document, and the U.N. has so far resisted doing so in any subsequent declarations, despite growing conservative pressure (led by Russia) to do so.

The recent gains by the same-sex marriage movement have simultaneously helped and hindered advocates working to change the legal definition of family more broadly. On the one hand, they’ve expanded protections and benefits to an untold number of same-sex couples, and arguably have shifted the conceptual basis for what constitutes a family in the first place. On the other hand, same-sex marriage has affirmed a system that still doles out privileges (like tax breaks, immigration status, and so on) on the basis of a narrow type of relationships.

To some advocates, this leaves a central question unanswered: Why does it make a difference to the government whether there are two people in a relationship or three?

“The state still defines the family,” said Sarah Wright, the chair of the Board of Directors for Unmarried Equality, an organization that fights for the rights of single people, and those in relationships other than monogamous, two-person marriages.

Wright herself has experienced the ways in which the gains of same-sex marriage can get twisted into losses for others. When her partner got a new job in Massachusetts, Wright could not get coverage under her insurance because in a “marriage equality” state, her partner’s employer saw no reason to offer benefits to domestic partners or other unmarried couples.

Historically, intentional families, polygamists, and other non-normative family structures have fared poorly in our justice system. However, in 2013, the stars of the TLC show Sister Wives challenged Utah’s anti-polygamy laws in the case of Brown v. Buhman. While Federal Judge Clark Waddoups ultimately ruled in favor of Utah’s ban on multiple marriages, he did strike down anti-polygamy laws that prohibited married individuals from cohabiting with multiple people.

Given legal precedent, it’s unlikely the Scarborough family will win their case. But they won’t be the last non-traditional family to run up against a system of connecting rights to certain kinds of sexual relationships. Our government has an obvious interest in granting benefits to people taking care of one another or raising children together, because that care can help keep vulnerable people off of the streets, out of the emergency rooms, and generally less dependent on the government.

In effect, giving material support to families is a roundabout way for the state to prevent more costly interventions later. But apart from “traditional” religious reasons (i.e., the same ones that aren’t admissible in the case of same-sex marriage), why should it matter if that family is a married couple or 11 people who care about each other just as much and make similarly binding commitments?

Removing the connection between civil benefits and marriage might go a long way towards strengthening the traditional religious ideal of marriage as a permanent union between two loved ones, by removing all the other reasons people get and stay married.

Regardless of legal recognition, the Scarborough family says they’re here to stay, and even hope to make friends with the neighbors if they’re allowed to remain in their home. If they have to move, they’ll find somewhere else to live, but they hope the other residents of their area can come to accept them as the family they are.

“I wish I could give to my clients this amount of support, this amount of trust, this amount of just day-to-day goodness,” said Welch.