SALT LAKE CITY -- A federal judge heard legal arguments that could potentially decriminalize polygamy in Utah, as a reality TV show polygamist and his four wives sought to overturn the state's ban on plural marriage.

Kody Brown and his wives, Meri, Janelle, Christine, and Robyn, are suing the state in a challenge to Utah's anti-polygamy statutes. They argue Utah laws criminalize their relationship -- even though they are consenting adults.

"What the Browns are seeking is what most families take for granted," their attorney, Jonathan Turley, told reporters outside the courthouse on Thursday.

The family came under investigation by Utah County authorities by starring in the reality TV show "Sister Wives." They were never charged, but argue that they were placed under a "cloud of suspicion" by Lehi police. The Browns argue that Utah's anti-polygamy statutes violate their free speech and free association rights.

"There are tens of thousands of families in this state that live under this condemnation," Turley said. "That they are felons because of how they live."

Lawyers for the state of Utah argued they have a right to protect marriage.

"There's no court in this country that has held that the practice of polygamy is a fundamental right," argued Assistant Utah Attorney General Jerrold Jensen.

But Jensen faced a blistering line of questioning from U.S. District Judge Clark Waddoups, who cited U.S. Supreme Court ruling Lawrence v. Texas, which decriminalized sexual conduct amongst consenting adults. Jensen appeared to draw a line between a man with multiple mistresses and someone who purports to be married, like polygamists do.

"How does this protect marriage?" Waddoups asked. "How do you believe consenting adults making a commitment to each other is damaging to the institution of marriage?"

Jensen said there are abuses within polygamy and the state has a right to step in. Waddoups countered that there are laws dealing with abuses associated with polygamy.

"The anti-polygamy statute deals with more than just consenting adults," Jensen told reporters outside of court. "Anytime you have a polygamous union, you tend to have children. Children become a part of the equation. They're very much a factor in this and most of the horror stories involving polygamy tend to involve children."

There are an estimated 30,000 polygamists in and around Utah, mostly fundamentalist Mormons who believe polygamy to be a religious principle.

Turley said the Browns were not seeking marriage licenses or a legalization of their relationships, but to be left alone by the state. He noted that Utah's anti-polygamy law is unique because other states' bigamy laws deal with marriage licenses only. Utah's, dating back to abandoning polygamy as a condition of statehood, criminalizes relationships.

"They're talking about my life," Valerie Darger, a plural wife, told FOX 13. "The thing that's different about what we're asking for is the right to exist and the right to be left alone. We're not seeking marriage licenses, and so, as far as legal marriage goes, it doesn't really pertain."

Darger and her sister-wife, Vicki, attended Thursday's hearing. So did anti-polygamy activists like Kristyn Decker, an ex-polygamous wife who left a relationship 11 years ago. She founded a group opposed to polygamy and led a protest to call attention to abuses and child-bride marriages within some polygamous communities.

"We feel like if they decriminalize polygamy, the human rights violations that have gone on for so long will just continue," Decker said.

Judge Waddoups took the arguments under advisement and did not say when he would issue a ruling. Both sides have said they would likely appeal any ruling to the 10th U.S. Circuit Court of Appeals in Denver. Legal observers tell FOX 13 the case would likely go all the way to the U.S. Supreme Court for consideration.

Late Thursday, Kody Brown issued a statement through his attorney, saying:

“On behalf of the entire Brown family, I want to thank Judge Waddoups for this opportunity to argue the merits of our case. We understand that this is a historic moment for all plural families and we are honored and humbled to serve as the plaintiffs in this action. We are especially grateful for the support of our counsel, Professor Jonathan Turley, Adam Alba, the students at George Washington University, and the many supporters who have stood by us over the course of this litigation. This has been a difficult road for us and we are relieved to see the case coming to the final arguments. We remained committed to this civil rights cause and the struggle of plural families, both religious and non-religious, in the state. We hope that Utahans can understand that our family – like tens of thousands in this state – are seeking only to be allowed to live according to our beliefs and not be declared felons simply because we are different.”