A three-judge federal appeals court panel on Monday handed a setback to the burgeoning polyamorous rights movement, reversing a lower court ruling that decriminalized polygamous cohabitation in Utah.

The case was brought by the five-spouse Brown family of "Sister Wives" reality TV fame after local authorities openly investigated them for violating a state law against multispouse living arrangements. The family intends to appeal the latest ruling, their attorney Jonathan Turley said in a statement.

The Browns are fundamentalist Mormons who argue the U.S. Constitution allows them to live according to the teachings of their faith. They fled Lehi, Utah, in January 2011 after a deputy Utah County attorney quipped the family "made it easier for us by admitting to felonies on national TV."

In 2013, the family won a first-of-its-kind ruling from U.S. District Judge Clark Waddoups, who ruled the First Amendment protected the Browns from such a ban and that – in light of the 2003 Supreme Court decision shielding consensual same-sex sodomy from state laws – such a prohibition also violates the Constitution's Due Process Clause.

The law challenged by the Browns says "a person is guilty of bigamy when, knowing he has a husband or wife or knowing the other person has a husband or wife, the person purports to marry another person or cohabits with another person." Waddoups ordered "or cohabits with another person" be deleted and narrowed the meaning of "purports to marry" but allowed a ban on multiple marriage licenses.

Like most other polygamists, Kody Brown only is legally married to his first wife, Meri, though he has children and lives with each of the four women. The family currently lives in Nevada.

On appeal, state officials argued the Browns had no right to sue, as the Utah County Attorney's Office had adopted a policy mooting the case in 2012, after the lawsuit was filed but before Waddoup's ruling. The prosecutor's office policy allows for prosecution of bigamy only under two conditions: when someone remarries without dissolving their first marriage or when bigamous couples or unwedded cohabitators are "also engaged in some type of abuse, violence or fraud."

Though Waddoups dismissed the policy as an attempt to avoid a ruling on the Browns' claims, the appeals court judges found the policy did moot the case and overturned Waddoups' ruling without consideration of the consitutional issues.

"Of course a future county attorney could change the UCAO Policy, but that possibility does not breathe life into an otherwise moot case," the 10th Circuit appeals panel ruled.

Utah federal solicitor Parker Douglas, who defended the state law. says the Utah County policy is, to the best of his knowledge, maintained by every county in the state as well as by the state attorney general's office. He doubts a sudden boom in prosecutions.

The Monday ruling, he says, "means that the law is what the law was before the lawsuit was ever filed. It is a crime under the Utah bigamy statute for someone who is legally married to one person to cohabitate with others who are not their legal spouse, or purport to be married to them."

Douglas says it's important that the law remain on the books because there's often an evidentiary problem against polygamy-practicing groups that engage in secretive practices such as child marriage and rape.

"To think it's just a matter of consenting adults is ignorant and naive as to how polygamy is sometimes practiced in Utah," he says. "I grew up in California. … I understand the draw of the simple, 'Aren't we all consenting adults here' … but when you get into the weeds of what consent means in many of these relationships, it's a much more sophisticated and complex question, [especially] when you throw in abuse that can happen over generations."

Douglas says the majority of people living in polyamorous or polygamyst relationships probably do not abuse children or their partners, but that many do, pointing to the Fundamentalist Church of Jesus Christ of Latter-Day Saints led by imprisoned child rapist Warren Jeffs. That fundamentalist Mormon offshoot is much more secretive than the Browns' Apostolic United Brethren church.

"We really have to separate the issues of polygamy form the issues of abuse," counters Robyn Trask, executive director of the Colorado-based polyamorist advocacy group Loving More.

"It's actually the abuse that's the problem," says Trask, who lives with her husband but has two other long-term partners. "Abuses don't just happen in polygamy, they happen in monogamy as well."

Polyamory practitioners motivated by religion, like the Browns, or by secular reasons, like Trask, generally don't mix, though they have a clear overlapping interest in decriminalizing cohabitation, which Trask notes also is banned by towns around the country through laws intended to keep homes from being packed to the brim with immigrants or rowdy college students.

Trask is hosting a conference with dozens of polyamorous people later this month but says she's not sure anyone from next-door Utah is coming.

The socially ascendant secular polyamorous movement generally has more liberal views of the role of women in relationships, a more open stance toward same-sex relations and an embrace of more fluid relationship statuses.

Douglas says he disagrees that the legal trajectory for polyamory will mirror with a decade or so lag the gay rights cases that decriminalized sexual relations in 2003 and allowed for same-sex marriage last year.

"We're not talking about private sexual conduct," he says about comparisons of the Brown case to the 2003 sodomy ruling.

Brushing off a potential analogy to the push for same-sex marriage, he adds: "There just aren't allegations and actual cases where you can say abuse happens at a higher rate in same-sex marriage situations as you can in polygamist and bigamist settings. We have a historical record that's completely different. The issue isn't one of the social institution of marriage, it's one of safe living conditions."

Turley, the Browns' lawyer and a George Washington University law professor, said in a post on his blog the family is considering its next step.

"This has been a long struggle for the Brown family but they have never wavered in their commitment to defending the important principles of religious freedom in this case," he said. "The decision today only deepens their resolve to fight for those same rights."