WASHINGTON (AP) -- The Supreme Court ruled Monday that Muslim men detained after the Sept. 11 attacks can't sue top U.S. law enforcement officials.

The justices by a 4-2 vote ended a long-running lawsuit against former Attorney General John Ashcroft, former FBI Director Robert Mueller and other top Bush administration officials. The suit was filed by Muslim men who were detained for months in harsh conditions in a Brooklyn jail after the 2001 attacks on the World Trade Center and the Pentagon.

They were seeking damages against Ashcroft, Mueller, former immigration chief James Ziglar and the man who ran the federal jail. A lower court still may re-examine claims against the jailer.

In his majority opinion, Justice Anthony Kennedy noted that the case arose in the aftermath of the deadly attacks, when U.S. officials wanted to be sure that there were no other potential attackers in the country. The men who were arrested all were in the United States illegally, many having stayed past the expiration of a visa.

He also acknowledged that the men were treated badly.

"There is therefore a balance to be struck, in situations like this one, between deterring constitutional violations and freeing high officials to make the lawful decisions necessary to protect the Nation in times of great peril," Kennedy wrote. But Congress, not the courts, should set when top officials can be sued, Kennedy said.

The decision was the third in which the court has intervened and ruled for Ashcroft in lawsuits against him and others from Muslims who were arrested in the U.S. following the 2001 attacks.

In a dissent joined by Justice Ruth Bader Ginsburg, Justice Stephen Breyer said the ability to hold officials accountable is especially important in a time of war or national security emergency. "History warns of the risk to liberty in times of national crisis," Breyer said in a dissent he summarized aloud from the bench Monday.

Only six of the nine justices were eligible to take part in the case. Justices Elena Kagan and Sonia Sotomayor had been involved in earlier stages of the lawsuit, before they were on the court. Justice Neil Gorsuch joined the court nearly three months after the case was argued.