Grand Marais is a quiet Minnesota town on the Lake Superior coast. Seth Jeffs is the brother of notorious cult leader Warren Jeffs, and the leader of his own secretive sect. The only thing separating them is a dispute over local wetlands.

Seth Jeffs was a prominent figure in the Fundamentalist Church of the Latter Day Saints, a polygamist sex cult that married off underage girls to adult men. After leader Warren Jeffs was sentenced to life in prison for crimes including raping his 12-year-old “wife,” Seth Jeffs and other FLDS members flitted from state to state establishing new religious compounds and dodging legal action. Now Seth Jeffs is build a new compound in Grand Marais, all while facing a lawsuit from a woman who says he and the FLDS subjected her to ritual sex abuse at 8 years old.

Right now, Grand Marais’ best hope to stop Jeffs is a regulation about construction in the area’s wetlands.

On May 18, approximately 100 Grand Marais residents, nearly a tenth of the town, gathered for a community meeting about their new neighbors. In December, Jeffs secured a permit to build a 5,760-square-foot structure on 40 acres of local land. (Jeffs was unreachable for comment.)

“We wouldn’t be able to tell if anything is going on there,” one local worried during the meeting, according to KARE 11.

They had real reason to worry. Jeffs has a long criminal history with the FLDS, starting with its notorious stronghold in Texas in the early 2000s. There, Jeffs’ brother Warren oversaw a fundamentalist cult rooted in polygamy and the marriage of young girls to adults. Warren was reported to have more than 80 wives, more than a dozen of whom were previously married to his father Rulon. Under Warren’s leadership, high-ranking church men were assigned multiple wives, while “reassigning” the wives and children of men who fell out of favor. Two of Warren’s children accused him of sexual abuse, including his son Roy, who killed himself on Friday.

In 2005, Warren was charged with sexual assault of a minor, as well as a charge for arranging the marriage of a 14-year-old to her cousin. He spent a year on the run before police captured him in a Cadillac full of disguises and burner cell phones. Seth Jeffs was also arrested and convicted for harboring Warren and concealing his whereabouts.

Warren went on to serve life in prison but Seth Jeffs returned to the FLDS, which said it no longer arranged child marriages. In a 2009 interview in a Texas compound, Jeffs said he had seven wives and 19 children. Jeffs presented his polygamous family as a godly arrangement. But behind the scenes he and other church elders were conspiring to break the law. In 2016, they were convicted of an elaborate scheme to game the SNAP food program by claiming benefits for the compound’s sprawling families, then hoarding the benefits for themselves and the church.

When Texas seized the FLDS compound, Jeffs and many followers of his church moved to a new facility outside rural Pringle, South Dakota. Like their eventual Grand Marais neighbors, Pringle locals didn’t take kindly to the new compound. The sprawling, secretive compound hosted more church members than Pringle had residents. At least one Pringle resident moved 65 miles away to escape the church and its constant construction.

A former compound resident claimed at least two dozen children had been born within its walls, an expected development for a church that promotes large families. But none of the births (or any deaths) was recorded with the state’s department of health. In response, South Dakota lawmakers in February voted in favor of a new law that would make it a misdemeanor not to report a new birth or death within a year.

But by then, Jeffs was already on the move again. In 2017, a former FLDS member filed a lawsuit accusing the church of subjecting her to “systemic sex abuse” from ages eight to 14. When she turned 14, the woman alleged, the church forced her to become a “scribe” recording elders’ abuse of other underage girls. She accused Jeffs and his brother Warren of watching and participating in the abuse.

The woman’s lawyer, Alan Mortensen, tried serving Jeffs with court papers, but couldn’t locate him for more than a year. The legal team finally got a tip-off when a company called “Emerald Industries” bought 40 acres of land outside Grand Marais and filed a permit for one of the large buildings characteristic of FLDS compounds. Jeffs was listed as the company’s manager. Mortensen’s team served Jeffs with papers, although Jeffs does not appear to have responded to the suit.

Grand Marais residents were concerned. But their options were frustratingly limited. At the May 18 community meeting, an investigator who wrote a book on the FLDS informed locals of their rights

“You can’t infringe on other people’s rights. Then you turn into the bad guy,” the author, Sam Brower told the group. “But you can be diligent, you can watch, you can look out for the signs. You can look out for child labor or young girls with babies, things like that.”

The county sheriff echoed him, stating that police couldn’t intervene unless they found the group breaking the law.

Late last week, county officials said they found sect breaking local rules against developing in wetlands. The county issued a temporary order to stop construction. The order won’t permanently block the settlement, however.

“He’s planning on doing some work,” Brower said of Jeffs during the community meeting. “He’s planning on staying.”