The U.S. National Security Agency has lost the legal authority to collect telephone records after the Senate failed to agree Sunday on an extension of several provisions put in place following the September 11, 2001 terror attacks.

The once-secret program that collected information such as the phone numbers, date and duration of calls to try to thwart suspected terrorists drew criticism from Americans who said it violated their privacy.

A new law, already passed by the House of Representatives, would modify the program to allow the NSA to request that data from phone companies on a case-by-case basis, but not collect it on its own. The Senate did not approve the measure before the surveillance program expired at midnight Sunday, but could still pass the bill later this week.

White House spokesman Josh Earnest said late Sunday that the Senate needs to act quickly to address what he called an "irresponsible lapse." He said the new program is a "reasonable compromise balancing security and privacy."

Without the new bill, the NSA has also been forced to at least temporarily shut down two other programs, including a "lone wolf" tracking provision that has never been used and a roaming wiretap program.

Earlier Sunday, Central Intelligence Director John Brennan urged the Senate to pass the new authorization, warning that terrorists were looking to take advantage of any gap in the spying effort.

"Terrorist elements have watched very carefully what has happened here in the United States," Brennan said in an appearance on CBS's Face the Nation. "Whether or not it’s disclosures of classified information, or whether it’s changes in the law and policies, they’re looking for the seams to operate within. This is something that we can’t afford to deal with right now, because if you look at the horrific terrorist attacks and violence that’s being perpetrated around the globe, we need to keep our country safe.”

Senator Rand Paul, who is running for president in 2016, led the opposition against the surveillance program, using the Senate's rules to delay a final vote for several days and thus allow the provisions to expire.

But other lawmakers accused him of exploiting the issue for political gain, including Senator John McCain, who said Paul places a "higher priority on his fundraising and his ambitions than on the security of the nation.

Obama urges Senate to approve USA Freedom Act

President Barack Obama strongly urged the Senate to approve the House-passed USA Freedom Act, which he said was being blocked by "a handful of senators."

WATCH: President Barack Obama discusses USA Freedom Act in weekly address

"I don't want us to be in a situation in which, for a certain period of time, those authorities go away and suddenly we're dark," Obama said. "And, heaven forbid we've got a problem where we could have prevented a terrorist attack or apprehended someone who was engaged in dangerous activity but we didn't do so simply because of inaction in the Senate."

Surveillance Provisions to Expire Sunday

Unusual alliances

The debate over whether to renew some of the NSA surveillance powers is cutting across the usual political divides between Democrats and Republicans, creating rare alliances.

On Friday, the American Civil Liberties Union joined with the Tea Party Patriots conservative group to call for the Senate to let the provisions expire, saying the government had overreached with its data collection and had violated Americans’ right to privacy.

VOA's Cindy Saine contributed to this report from Capitol Hill.

WATCH: Debate over surveillance measures comes down to the wire