NEW YORK -- Justice Department lawyers faced tough questioning from federal judges Thursday over whether former high-level officials like John Ashcroft and Robert Mueller III can be held accountable for abuses of immigration detainees in the immediate aftermath of the Sept. 11 attacks.

"Tell me now who did it," Judge Richard Wesley, of the U.S. Court of Appeals for the 2nd Circuit, asked. "Someone had to say it's the FBI's call."

But the Justice Department lawyers would not say who decided to place the fate of more than 700 Arabs and Muslims swept up on immigration violations under the FBI's control. Ashcroft was then the attorney general, Mueller the director of the FBI.

For more than a decade since Ibrahim Turkmen and other detainees filed their lawsuit in 2002, government lawyers and civil liberties advocates have been fighting a complicated battle over who can be held responsible for the harsh treatment they experienced.

After the FBI designated them all as persons of interest in the 9/11 attacks, often on the flimsiest of evidence, the Arabs and Muslims were held in jails for months under conditions as harsh as those at the federal "supermax" prison in Florence, Colorado. They were banned from talking to lawyers, prevented from reading the Quran and, in some cases, slammed against walls.

Two withering reports from the Justice Department's inspector general blasted those jailhouse conditions, but no high-level official from the George W. Bush administration has ever been held accountable. The class action lawsuit against Ashcroft, Mueller and others has instead faltered, hobbled by the 2009 Supreme Court ruling in a similar case, Ashcroft v. Iqbal.

In the Iqbal case, the court ruled that wronged immigrants could only proceed with a lawsuit against Ashcroft and Mueller if they had a "plausible" claim that those officials knew about or directed unconstitutional discrimination. Because those plaintiffs had been denied discovery -- the legal process by which they might have been able to probe the government's documents for more evidence -- their case was fatally hamstrung.

In Turkmen v. Ashcroft, the government is similarly attempting to shut out the plaintiffs. It argues there is no plausible basis to conclude that an FBI directive to hold the plaintiffs under the tough conditions accorded to suspected terrorists -- instead of those generally accorded to immigration violators, which they by and large were -- led to the resulting abuses in Federal Bureau of Prisons custody.

"The key point here is, somebody made a decision to hold them in segregation -- but Mr. Ashcroft and Mr. Mueller did not make that decision," said Justice Department attorney H. Thomas Byron III.

Judge Rosemary Pooler asked whether that meant the warden at the Metropolitan Detention Center in Manhattan -- just blocks from where the World Trade Center towers once stood and the site of some of the worst abuses -- had led a "rogue organization."

In January 2013, a U.S. district judge agreed with the Justice Department, throwing out the part of the lawsuit aimed at Ashcroft, Mueller and James Ziglar, former director of the Immigration and Naturalization Service. But lawyers at the Center for Constitutional Rights are appealing to have the case against those officials reinstated, as well as pursuing other claims in the same lawsuit against Bureau of Prisons officials who directly oversaw the prisons where their clients were abused.

Intense questioning from the circuit judges on Thursday suggested that they may be searching to find a way to keep Ashcroft and Mueller in the case, despite the Supreme Court's ruling in Iqbal.