US president Barack Obama forcefully defended the embattled National Security Agency on Friday in a speech that outlined a series of surveillance reforms but stopped well short of demanding an end to the bulk collection of American phone data.

In his widely anticipated address at the Justice Department on the future course of US surveillance policy, Obama said the government should no longer hold databases of every call record made in the United States, citing the “potential for abuse”.

But Obama did not say what should replace the databases and made it clear the intelligence agencies should still be able to access call records information in some unspecified way, signalling a new round in the battle between privacy advocates and the NSA’s allies.

Mounting a forceful defence of the NSA, Obama said: “They’re not abusing authorities in order to listen to your private phone calls, or read your emails.” He did not mention that judges on the secret surveillance court have found NSA has repeatedly and “systematically” overstepped its bounds. Instead, he counselled strongly against any steps that would undermine US national security. “We cannot unilaterally disarm our intelligence agencies,” he said.

Obama’s remarks were bound to give the beleaguered NSA a boost of confidence, while disappointing civil libertarians who wanted to hear the president defend the privacy of American citizens more emphatically.

In the key points of his speech, Obama said:

• The government will no longer store the phone call information of millions of Americans. But he did not say who should maintain the information, instead giving the intelligence community 60 days to come up with options.

• Intelligence agencies must, with immediate effect, apply to the secret Fisa court for judicial approval to access Americans' phone records.

• The secret Fisa court should be reformed to include a panel of independent advocates to provide a voice in "significant cases".

• The NSA will not spy on the heads of state and governments of allies, and said some further protections would be given to foreign citizens whose communications were caught up in the agency's dragnet.

• The US government had to be held to a "higher standard" than private corporations that store user data or foreign governments that undertake their own surveillance.

Obama said a balance had to be struck between competing demands. “We have to make some important decisions about how to protect ourselves and sustain our leadership in the world, while upholding the civil liberties and privacy protections that our ideals and our constitution require,” he said.

More reviews

The president called for further reviews of the thorny questions surrounding the scope of NSA’s power. “The challenge is getting the details right, and that is not simple.”

Obama has directed attorney general Eric Holder, NSA director Keith Alexander and the rest of the intelligence community to present proposals over the next several weeks for what private entity should hold Americans’ telephone metadata, casting it as an issue that requires further deliberation before any decision that could have major implications for both privacy and security.

Facebook Twitter Pinterest CIA director John Brennan left, talks with the director of national intelligence James Clapper before Barack Obama outlined his NSA reforms. Photograph: Jim WatsonAFP/Getty Images

He made it clear that when he previously reviewed US intelligence gathering operations, he concluded that the NSA needed the capabilities offered by the controversial programs and did not intentionally abuse them.

“What I did not do is stop these programs wholesale, not only because I felt that they made us more secure, but also because nothing in that initial review, and nothing that I have learned since, indicated that our intelligence community has sought to violate the law or is cavalier about the civil liberties of their fellow citizens,” Obama said.

Obama said that ahead of transitioning the database out of government hands, the NSA will now have to receive approval from the secret surveillance court before searching through the data trove for connections to terrorist groups. Those searches would be restricted to two “hops”, meaning that a number “reasonably” suspected can have all the numbers it called and from which it received calls reviewed, and all of those numbers can also have their connections examined.

“I believe it is important that the capability that this program is designed to meet is preserved,” Obama said.

Cautious welcome

That call fell short of expectations from privacy advocates in and outside of Congress, who want the government collection of Americans’ metadata without specific connections to terrorism, approved by a judge, to stop. "We must all remember that the very act of bulk collection of private data undermines Americans’ constitutional rights," said Senator Ron Wyden, a prominent critic, on Twitter after the speech.

But Wyden and another Democrat on the Senate intelligence committee, Mark Udall, gave the speech as a whole a cautious welcome. “Make no mistake, this is a major milestone in our longstanding efforts to reform the National Security Agency’s bulk collection program,” they said a joint statement.

“The president has listened to some of the advice of his independent panel of experts and endorsed some of the reforms we have long advocated. The fight to protect liberty and increase security is far from over.”

Patrick Leahy, a Democrat on the Senate justice committee, welcomed the reforms to the bulk collection, but said Congress had to act to put them into place. "The president has ordered some significant changes, but more are needed," he said in a statement.

Republican critics in Congress were more blunt in their assessment of the speech's shortcomings. “It's not about who holds it, I don't want them collecting Americans' information,” senator Rand Paul of Kentucky told CNN.

“Paul Revere was warning us that the British were coming. He wasn't warning us that the Americans were coming,” he added, taking issue with Obama's reference to colonial revolutionaries in defence of NSA surveillance.

In his speech, Obama was more definitive about assuring allied foreign leaders that he would not spy on their communications, an issue that became a diplomatic row straining relations with Germany, Brazil and other countries.

“The leaders of our close friends and allies deserve to know that if I want to learn what they think about an issue, I will pick up the phone and call them, rather than turning to surveillance,” Obama said.

But Obama did not announce any new protections for non-Americans abroad, instead punting the issue to his top officials for further consideration. Nor did he address the NSA’s secret weakening of encryption standards, first reported by the Guardian, which have greatly concerned US businesses.

The Guardian reported on Thursday, based on documents leaked by NSA whistleblower Edward Snowden, that the NSA collects an average of 200m texts messages abroad every day. The collection of “untargeted and unwarranted” communications, through a program codenamed Dishfire, is one of several NSA activities that have drawn global concern since their disclosure.

Similarly, Obama defended a program, now conducted under Section 702 of the Fisa Amendments Act, that allows the NSA to hold the internet communications of foreigners abroad. He directed Holder and James Clapper, the director of national intelligence, to review the program to add protections for Americans whose information has been caught up in the dragnet.

But he stopped he short of ordering NSA not to query that database for Americans’ identifying information, an authority first revealed by the Guardian and which senator Ron Wyden dubbed the “backdoor search loophole”. Any restriction on that authority would apparently follow the review.

Edward Snowden, whose disclosures prompted the reforms. Photograph: Sunshinepress/Getty Images

Obama said he wanted additional safeguards for the secret surveillance court, known as the Fisa court, which currently only the government can petition. Obama urged Congress to set up “an independent voice” before the court, along with new technological consultants, who can argue for the privacy interests of US citizens and help the court better navigate the vast, complex and changing technological capabilities of the NSA. Declassified rulings show this has vexed judges on the court for years.

The president also rejected a recommendation from his handpicked surveillance panel to place judicial oversight over the FBI’s practice of issuing a nonjudicial subpoena known as a National Security Letter for the records of US businesses. But Obama proposed allowing businesses greater, but as yet unspecified, ability to discuss those letters, which are currently under a gag order.

Much of the substance of Obama’s proposals remain undefined. The telephone companies have resisted having to store customer data for additional periods of time on behalf of the NSA, and any new third-party private storehouse of metadata would have to be created from scratch.

The lack of clarity places increasing pressure on Congress to ultimately resolve many of the complexities of surveillance – creating effectively a new round of jockeying on Capitol Hill between privacy advocates and the NSA’s allies, who fear losing what Obama described as a valuable tool for determining domestic connections to terrorism.

"Now it's up to the courts, Congress, and the public to ensure that real reform happens, including stopping all bulk surveillance – not just telephone records collection,” said Cindy Cohn, legal director of the Electronic Freedom Foundation.

Obama gave less attention to the man whose disclosures prompted his speech: Edward Snowden. “I’m not going to dwell on Mr Snowden’s actions or motivations,” Obama said. “I will say that our nation’s defence depends in part on the fidelity of those entrusted with our nation’s secrets.”