A federal jury this week found former New York Gov. George Pataki and two other former state officials “not liable” for illegally confining six sex offenders to state mental institutions after they had completed their sentences.

On one hand, Wednesday's decision was not surprising. American politics and society have long struggled with the demand to continue to keep locked up criminals who have committed the most vile crimes but who, according to law, are allowed to go free. Yet that tendency threatens the integrity of the judicial process, with few willing to stand up for people who have done such reprehensible things, some legal experts say.

Such cases are emotionally fraught and constitute a delicate legal balance between public safety and due process, and the jury's verdict in the New York sex offenders' civil suit points to the difficulty of striking that balance.

“It’s an ominous development,” says Robert Burt, a law professor at Yale University in New Haven, Conn. These kinds of legal efforts seek “to turn confinement [of sexual offenders] away from the ordinary criminal justice system, into a mental health issue, and then to offer treatment." But the offer is a "fraud" that's not followed up on, he says.

The case centered on a 2005 Pataki administration initiative that called for the psychiatric evaluation and continued confinement of potentially dangerous prisoners slated to be released. Twenty states, including California, Florida, and Illinois, have enacted laws permitting the civil commitment of sexual offenders as of 2010. The Adam Walsh Child Protection and Safety Act of 2006 also authorizes the federal government to commit and treat federal sex offenders. But these include legal safeguards that have passed constitutional muster, including judicial review. The Pataki initiative was simply an administrative policy.

In 2006, a federal judge found the Pataki administration's Sexually Violent Predator Initiative to be unconstitutional, and most of those confined under civil commitments were released. The offenders in the civil suit, each convicted of horrendous sex crimes, sought $10 million in damages.

The jury found Mr. Pataki; Glenn Goord, former correctional services commissioner; and Eileen Consilvio, the former executive director of the Manhattan Psychiatric Center, the hospital where the plaintiffs were held, not liable. The jury did find the former commissioner of the State Office of Mental Health, Sharon Carpinello, liable for their involuntary confinement. It awarded the former prisoners $1 each in damages.

In his instructions to the jury, federal district Judge Jed Rakoff explained that it was “undisputed” that the procedures of the Sexually Violent Predator Initiative violated constitutional due process. So the issue was, he said, whether this "violation of a plaintiff's rights was done intentionally, recklessly, wantonly, maliciously or the like, or was done, by contrast, in good faith."

For all but Ms. Carpinello, the jury found that they acted in good faith.

“I think the fact they found one defendant liable for $1 sends a clear message that the state acted properly,” said Pataki, “and that these plaintiffs were not entitled to any real damages based on a technical mistake.”

American prisons and state-run mental institutions have a long history of abridging the due-process rights for social outcasts and those deemed dangerous social pariahs. Earlier this month in California, for example, investigators discovered that administrators in California prisons were sterilizing dozens of female prisoners against their will and without the required approval of a state medical committee – a haunting reminder for a state with a history of eugenics.

In the early 20th century, the Colorado state legislature had either rejected, or the governor had vetoed, a eugenic sterilization law. But the superintendent of the Colorado State Hospital in Pueblo was sterilizing patients anyway – and these were often young women he thought were too promiscuous to be released, and who would then have out-of-wedlock, socially undesirable children.

The 2005 Pataki initiative caused concern for a different reason. He launched his initiative partly because he could not get it passed by the New York Legislature. On five separate occasions, the state Senate would overwhelmingly pass a bill to confine dangerous sex offenders to mental institutions after they served their full sentences. State Assembly leaders, however, would not even bring to measure to the floor for a vote.

So the governor instituted an executive initiative to keep these offenders in state custody. The decision followed a pattern throughout US history: executive actions have been a common way to institutionalize – and in the case of eugenics, sterilize – social outcasts without due process of law.

“Our clients were not people who should have been committed,” says Jeffrey Rothman, one of the attorneys for the former prisoners. “They were people who committed crimes and served the prison sentence, and they were entitled – entitled – to be released.... They were such social outcasts, they were easily tarred and feathered by those in power and made into something that could have political value. They instituted this initiative and robbed them of their liberty.”

Yet the deep disgust citizens feel for sexual predators often influences their notion of what "justice" means. Most of the men in the case had been convicted of sexual crimes against children; one was convicted of molesting his daughter.

“What the Constitution is supposed to do is to protect the highest and lowest among us equally, and it’s supposed to protect the most despised in society against the tyranny of the majority,” says Mr. Rothman. “And that’s what the Constitution does in providing individual rights – it prevents the majority from just, you know, lynching them. It provides them with basic rights to prevent them from being torn asunder by the anger of the mob.”