In 2004, Justice Stephen Breyer cast the decisive vote in Hamdi v. Rumsfeld, allowing the government to detain alleged terrorists indefinitely without trial. Hamdi interpreted the Authorization for Use of Military Force as a boundless grant of power to the president, permitting the executive to imprison anyone deemed an “enemy combatant” without bona fide due process. The Supreme Court’s sweeping decision allowed Presidents George W. Bush, Barack Obama, and Donald Trump to keep these “combatants” locked up at Guantanamo Bay, deprived of fundamental constitutional safeguards. Trump is eager to hold them there for years to come.

On Monday, Breyer announced his desire to “confront the difficult question left open by Hamdi”—and, implicitly, his growing discomfort with the decision’s impact. But it’s almost certainly too late. The Supreme Court has lost its swing votes and conservative skeptics of executive power. Until Congress and the president decide otherwise, the courts will permit this suspension of due process with no end in sight.

Congress passed the AUMF in the week after 9/11, a hasty attempt to arm Bush with the necessary tools to protect the nation. It authorized the executive to “use all necessary and appropriate force against those nations, organizations, or persons” that “planned, authorized, committed, or aided the terrorist attacks” or “harbored such organizations or persons.” Crucially, the law allows the president to “determin[e]” who is subject to the AUMF.

The Bush administration construed the law to permit, among other things, the detention of anyone the president labeled an enemy combatant—citizens and aliens alike. Hamdi involved a challenge to this alleged power. Yaser Esam Hamdi was a U.S. citizen captured in Afghanistan, taken to Guantanamo, then transferred to a naval brig in the United States. Accused of joining the Taliban, Hamdi was deemed an enemy combatant, and the government asserted a right to hold him indefinitely without even bringing formal charges, let alone giving him a trial.

The evidence against Hamdi rested entirely on a declaration by Michael Mobbs, the special adviser to the undersecretary of defense for policy. Mobbs stated that, upon his “review of relevant records and reports,” he believed Hamdi was associated with the Taliban. But the government refused to show these “records and reports” to the court. Instead, it asserted that because the president had “determin[ed]” that Hamdi was an enemy combatant, the military could use “appropriate force” against him—that is, detain him indefinitely—and the courts had no right to intervene.

In a split decision, the Supreme Court handed the government a compromise. A plurality opinion by Justice Sandra Day O’Connor, joined by Chief Justice William Rehnquist, Justice Anthony Kennedy, and Breyer, endorsed a key part of the Bush administration’s AUMF theory: It agreed that the law allowed the president to detain those he determines to be an enemy combatant. The plurality also agreed that these individuals are not owed full due process, even if they’re U.S. citizens.

O’Connor, however, deployed a balancing test that let Hamdi “receive notice of the factual basis for his classification” and “rebut the Government’s factual assertions before a neutral decisionmaker.” She then stacked the deck against him, permitting hearsay (like Mobbs’ declaration) as evidence, granting “a presumption in favor of the Government’s evidence,” and allowing the hearing to take place before a military tribunal. In other words, Hamdi wouldn’t get a fair trial, as prescribed by the Constitution; he would receive scraps of due process, “a factfinding process” rigged against him. (Eventually, Hamdi renounced his U.S. citizenship and was deported to Saudi Arabia.)

Justice Clarence Thomas would’ve let the government hold Hamdi indefinitely, without even a semblance of due process. On the other end of the spectrum, Justice Antonin Scalia, joined by the liberal Justice John Paul Stevens, railed against the plurality’s “Mr. Fix-it Mentality.” He noted that Congress had failed to suspend the writ of habeas corpus, which grants individuals the right to challenge their detention. Unless and until Congress suspends this right—which it can only do “in cases of rebellion or invasion the public safety”—the president cannot revoke citizens’ due process. Moreover, Scalia noted, the AUMF does not actually confer the broad detention powers the government claimed. Thus, Hamdi should either have been given a fair trial or released.

Scalia was right in Hamdi, and his dissent grew more prescient with time. He was, by no means, a champion of justice for enemy combatants; later, in the Guantanamo cases, he clarified that he believed noncitizens held abroad have no right to habeas corpus. But his dissent from the plurality’s radical interpretation of the AUMF has stood the test of time. By equating the power to use “necessary and appropriate force” with the power of indefinite detention, the Hamdi court handed the executive branch a blank check to keep alleged terrorists imprisoned without trial forever. And that is exactly what they have done. Today, there are about 40 men at Guantanamo, and the federal courts keep rejecting their efforts to leave. Why? Because of Hamdi.

Now Breyer seems to have his regrets. In 2014, he raised the possibility that the government had stretched the AUMF too thin by applying the law to a succession of conflicts untethered from its original purpose. And on Monday, when the court refused to hear a new case involving another combatant, he voiced an even stronger objection to the AUMF’s continued rein. O’Connor’s Hamdi opinion, he wrote, warned that if “the practical circumstances of a given conflict are entirely unlike those of the conflicts that informed” the AUMF, the court’s “understanding” of the law’s reach “may unravel.” Monday, Breyer wrote, it is “past time” to decide whether the basis of Hamdi had unraveled: After all, the government “does not state that any end is in sight” to the war on terror.

Scalia was right in Hamdi, and his dissent grew more prescient with time.

To drive home his point, Breyer noted that the detainee challenging his confinement in the case at hand, Moath Hamza Ahmed al-Alwi, has been held in Guantanamo for 17 years. “Al-Alwi,” he concluded, “faces the real prospect that he will spend the rest of his life in detention based on his status as an enemy combatant a generation ago, even though today’s conflict may differ substantially from the one Congress anticipated when it passed the AUMF.”

Curiously, Breyer voted with his colleagues not to take al-Alwi’s appeal, instead preferring to wait for “an appropriate case” to revisit Hamdi. (Why isn’t al-Alwi’s case appropriate? Breyer didn’t say.) But if such a case ever arises, Breyer is unlikely to reverse Hamdi’s damage. Scalia and Stevens, the strange bedfellows who defended habeas corpus, are gone. The court’s Guantanamo swing votes, O’Connor and Kennedy, have resigned, replaced by Justices Samuel Alito and Brett Kavanaugh. Both staunchly oppose rights for Guantanamo detainees. Indeed, while serving on the U.S. Court of Appeals for the District of Columbia Circuit, Kavanaugh played a leading role hobbling Kennedy’s efforts to bring constitutional protections to Guantanamo.

Since Hamdi, in fact, the Supreme Court has allowed an expansion of indefinite detention beyond so-called enemy combatants. In two recent cases, the conservative majority has allowed the Trump administration to detain certain immigrants indefinitely—a development that Breyer decried as betrayal of “our basic values.” With the court’s assent, Immigration and Customs Enforcement is creating an archipelago of Guantanamos where due process has been suspended for immigrants. “I would find it alarming,” Breyer wrote in a 2018 dissent, “to believe that Congress” intended this result.

There is a lesson to be gleaned from these awful cases: The power of indefinite detention is a weapon that presidents can wield recklessly and brutally. In Hamdi, the Supreme Court turned over this weapon to the executive branch, which continues to use it in a predictably appalling and arbitrary manner. Congress must repeal the AUMF, as well as any statute—particularly immigration laws—that can arguably be read to allow endless imprisonment. The Supreme Court can no longer be trusted to enforce due process, despite Breyer’s pleas. If Congress doesn’t put a stop to this unconstitutional cruelty, no one will.