In this May 6, 2004, file photo, Larry Klayman speaks in Melbourne, Fla.

WASHINGTON -- Three federal appeals court judges struggled Tuesday over whether the National Security Agency's phone data surveillance program is an intelligence-gathering tool that makes the nation safer or an intrusive threat that endangers privacy.

The judges --?? all appointed by Republican presidents --?? expressed uncertainty about where to draw the line between legal surveillance and violations of constitutional rights in the age of terrorism.

Since 2006, the FBI has obtained orders from the secret Foreign Intelligence Surveillance Court directing phone companies to produce telephone "metadata" --?? outgoing phone numbers dialed and numbers from incoming calls -- to the government.

The NSA consolidates the records into a searchable database in the hunt for terror suspects.

During the hour-and-a-half hearing, Judge David Sentelle questioned whether it is an invasion of privacy if the NSA simply collects the data, stopping short of using it.

Is it not an invasion "with mere collection?" asked Sentelle.

It is not, replied Justice Department lawyer H. Thomas Byron.

Arguing against the NSA program, activist attorney Larry Klayman disputed Byron, telling Judge Janice Rogers Brown that "collection is enough" to justify pursuing the lawsuit against the government.

It is Klayman's legal burden to show that he has been harmed by the program --?? an issue that threatens to derail his lawsuit and which clearly frustrated him during the court proceeding. The issue comes down to Klayman having to prove that his phone company was a participant in the NSA surveillance. Klayman is a Verizon Wireless customer, but the government has acknowledged only that a separate Verizon company turned over records to the government.

"This is not an insignificant matter" with the "government standing over our shoulder. I can't talk on my phone with my clients," Klayman lamented.

Isn't the sweep of phone numbers in the program "virtually universal?" appeals judge Stephen Williams asked Byron, the Justice Department lawyer. "And if that's the case, is it safe to assume that Verizon Wireless numbers are brought in?"

"The government doesn't and never has acquired all or nearly all of the telephone call data records," Byron replied.

Klayman won the first round in December when U.S. District Judge Richard Leon, a Republican appointee, ruled that the NSA's surveillance program likely runs afoul of the Constitution's ban on unreasonable searches. The government is seeking to have Leon's ruling thrown out. It could take months for the appeals court judges to rule.

Former NSA systems analyst Edward Snowden revealed the phone data collection effort a year and a half ago, triggering a debate over privacy rights and surveillance.

In New York, the U.S. Court of Appeals for the 2nd Circuit recently heard arguments in an appeal of a judge's opinion that found the surveillance program legal.

The three appeals judges in the Washington case have generally come down on the government's side on national security issues.

Sentelle permitted the George W. Bush administration to withhold names and other details about hundreds of foreigners detained in the months after the Sept. 11, 2001, terrorist attacks. Williams upheld the military tribunals set up by the Bush administration to try terrorism suspects for war crimes. Brown ruled that four British citizens had no right to sue Pentagon officials over accusations that the detainees were tortured and their religious rights violated while held at the U.S. detention center at Guantanamo Bay, Cuba.

Sentelle and Williams were nominees of President Ronald Reagan. Brown was nominated by President George W. Bush.

The government is using a 35-year-old Supreme Court ruling as the legal underpinning for the NSA's phone data surveillance program. In Smith v. Maryland, police asked a phone company to install a device to monitor numbers that a robbery suspect had dialed. The court ruled that the suspect had no expectation of privacy for the dialed phone numbers.

The Smith case doesn't fit today's reality, Klayman wrote in recently filed court papers. The bulk collection effort covers "every aspect of our lives," he wrote.

Klayman is using a recent Supreme Court opinion, Riley v. California, in his challenge to the surveillance program.

In that unanimous Supreme Court ruling in June, Chief Justice John Roberts said police generally may not search the cellphones of people they arrest without first getting search warrants.