LOS ANGELES (Reuters) - A federal judge on Tuesday dismissed a lawsuit charging that the FBI violated civil liberties by sending an informant into several California mosques to spy on U.S. Muslims, ruling that allowing the case to proceed could risk disclosure of government secrets.

U.S. District Judge Cormac Carney wrote in a 36-page order that he was reluctant to toss out the case before it could be litigated but was forced to weigh national security against individual liberties and an open judicial process.

In the decision, Carney compared himself to the fictional Greek hero Odysseus, who while sailing home from the Trojan War faced navigating his ship between a six-headed monster on one side and a dangerous whirlpool on the other.

"Odysseus opted to pass by the monster and risk a few of his individual sailors, rather than hazard the loss of his entire ship to the sucking whirlpool," he wrote. "Similarly, the proper application of the state secrets privilege may unfortunately mean the sacrifice of individual liberties for the sake of national security."

But the judge, ruling after a hearing in U.S. District Court in Santa Ana, allowed the case to go forward against five current or former FBI agents who the plaintiffs claim violated the Foreign Intelligence Surveillance Act.

The lawsuit, which was filed last year in U.S. District Court in Santa Ana, said the FBI sent a paid undercover informant named Craig Monteilh into Orange County mosques to collect personal information on hundreds, or possibly thousands, of Muslims.

According to the suit, Monteilh took hundreds of hours of surreptitious video and audio recordings of religious lectures, classes, cultural events and other meetings in 2006 and 2007 as part of a counterterrorism investigation, known as "Operation Flex," that did not produce a single conviction.

APPEAL PLANNED

Plaintiffs' attorney Ahilan Arulanantham said the American Civil Liberties Union and the Council on American-Islamic Relations, which filed the lawsuit on behalf of three Orange County Muslims and were seeking class-action status, intended to appeal the dismissal.

"It's deeply unfair because now hundreds of law-abiding Muslim citizens will never know whether the government violated their most basic civil rights," Arulanantham, from the ACLU of Southern California, told Reuters.

"That's why it's so important that this case be reviewed on appeal because the state secrets privilege is absolute when the government successfully invokes it," he said.

The FBI has acknowledged that Monteilh was used as a confidential informant during the operation but denies engaging in any unconstitutional practices, saying it took reasonable measures to investigate credible evidence of potential militant activity.

Government attorneys said that details about Monteilh's activities remain highly sensitive information.

A U.S. Department of Justice spokesman declined to comment and a lawyer representing three of the FBI agents named as individual defendants could not be reached by Reuters.

Carney wrote that he did not reach the decision lightly and conducted a careful review of classified government filings with "a skeptical eye" before ruling.

(Reporting by Dan Whitcomb; Editing by Cynthia Johnston and Eric Beech)