The 416 children taken from a ranch run by a polygamous sect will stay in state custody and, along with their parents, be subject to genetic testing to determine their relationships, a judge has ruled.

Child welfare officers said that after this adult mothers with children aged four and under, who had been allowed to stay together, would be separated from their children. Only mothers aged under 18 will be permitted to remain with their offspring.

The rulings came after Judge Barbara Walther at the district court in San Angelo, Texas, heard 21 hours of testimony over two days in one of the largest and most convoluted custody cases in US history. The genetic testing was ordered after child welfare officials told the court they were having difficulty determining how the children and adults were related because of evasive or changing answers. A mobile genetic lab will take DNA samples tomorrow at the main shelter where children are being kept, and parents will be able to submit samples on Tuesday in Eldorado, closer to the Yearning for Zion Ranch in west Texas.

Download the new Independent Premium app Sharing the full story, not just the headlines

The ranch was raided on 3 April after someone identifying herself as a 16-year-old girl with the Fundamentalist Church of Jesus Christ of Latter Day Saints (FLDS), which operates the ranch, claimed her 50-year-old husband beat and raped her. Hundreds of children were taken from the the Mormon splinter group's compound. The judge summed up the hearing in one line: "The issue before the court is – can I give them back?"

The scale of the case meant defence lawyers queued up in the court's aisles to take turns asking their questions. But the judge was able to exercise more control over the mass custody hearing than on Thursday, when the case descended into chaos, mainly due to hundreds of lawyers competing for clients.

Experts for the state had told the court that the girls entered into under-age marriages without resistance because they were indoctrinated from birth to believe disobedience would lead to damnation. A psychiatrist, Dr Bruce Perry, an authority on children in cults, said the sect's belief system was "abusive". "The culture is authoritarian," he said.

But under cross-examination, he admitted that the sect mothers were loving parents and that there were no signs of abuse among younger girls or any of the boys. Dr Perry stated that the girls he interviewed said they freely chose to marry young, but he added that those choices were based on lessons drilled into them from birth. "Obedience is a very important element of their belief system. "Compliance is being godly; it's part of their honouring God." Dr Perry was applauded by dozens of FLDS parents when he admitted that the children would suffer if placed in traditional foster care.

John Walsh, a witness for the parents, told the court that a bed in the retreat's white temple was not used to consummate the marriages of under-age girls to much older men. "There is no sexual activity in the temple," he said. Instead, he said, it was used for naps during the sect's long services.

Mr Walsh also denied that the young girls had no say in who they would marry. "They're into matchmaking," he said of the sect. Girls who refused matches had not been expelled. "I believe the girls are given a real choice. Girls have successfully said 'No, this is not a good match for me', and they remained in good standing," he said.

The children were seized in the raid on the desert compound because of evidence of physical and sexual abuse, including the forcing of under-age girls into marriage and childbearing, the judge was told. Only a few of the children were teenage girls, but about 20 women or more gave birth when they were minors, some as young as 13, authorities said.

The Child Protective Services agency argued that the teachings of the FLDS – to marry shortly after puberty, have as many children as possible and obey their fathers or their prophet, imprisoned leader Warren Jeffs – amounted to abuse. Mr Jeffs is in prison for being an accomplice to rape and was convicted in Utah last year of forcing a 14-year-old into marrying an older man. Mr Walsh told the court that the sect did not promote under-age marriages until Jeffs took over as its "prophet".