Former CIA Director David Petraeus stoked the controversy over the Obama administration's handling of the Libya terror attack, testifying Friday that references to "Al Qaeda involvement" were stripped from his agency's original talking points -- while other intelligence officials were unable to say who changed the memo, according to a top lawmaker who was briefed.

Rep. Peter King, R-N.Y., told Fox News that intelligence officials who testified in a closed-door hearing a day earlier, including Director of National Intelligence James Clapper and Acting CIA Director Mike Morell, said they did not know who changed the talking points. He said they went out to multiple departments, including the State Department, National Security Council, Justice Department and White House.

"To me the question right now is who changed those talking points and why. ... I'd say it was somebody in the administration had to have taken it out," King told Fox News. "That, to me, has to be pursued."

Petraeus left Capitol Hill around noon, after testifying in private hearings before the House and Senate intelligence committees. In his wake, Republicans and Democrats battled over whether his testimony should raise more suspicions about the administration's handling of the attack.

King and other Republicans indicated they still have plenty of questions about the aftermath of the strike.

"No one knows yet exactly who came up with the final version of the talking points," he said.

Petraeus' testimony both challenges the Obama administration's repeated claims that the attack was a "spontaneous" protest over an anti-Islam video, and according to King conflicts with his own briefing to lawmakers on Sept. 14. Sources have said Petraeus, in that briefing, also described the attack as a protest that spun out of control.

"His testimony today was that from the start, he had told us that this was a terrorist attack," King said, adding that he told Petraeus he had a "different recollection."

Still, the claim that the CIA's original talking points were changed is sure to stoke controversy on the Hill.

"The original talking points were much more specific about Al Qaeda involvement. And yet the final ones just said indications of extremists," King said, adding that the final version was the product of a vague "inter-agency process."

King said a CIA analyst specifically told lawmakers that the Al Qaeda affiliates line "was taken out."

A congressional source familiar with this week's testimony also told Fox News that the language in the CIA talking points about Benghazi was changed from "Al Qaeda-affiliated individuals to extremist organizations" -- which had the effect of minimizing the role of terrorists in the attack.

"It really changed the whole tone of it," King told Fox News.

Democrats, though, suggested Republicans were taking the whole issue out of context.

Rep. Adam Smith, D-Wash., said claims the talking points were changed are "completely wrong." Besides, he said, the affiliation of Ansar al-Sharia, the militant group suspected in the attack, to Al Qaeda is still being examined.

Sen. Kent Conrad, D-N.D., said the discrepancy can be attributed to the classified talking points that some saw versus the unclassified version that others, like U.S. Ambassador to the U.N. Susan Rice, used.

Lawmakers are focusing on the talking points in the first place because of concern over the account Rice gave on five Sunday shows on Sept. 16, when she repeatedly claimed the attack was spontaneous -- Rice's defenders have since insisted she was merely basing her statements on the intelligence at the time.

But a source said Rice had access to both classified and unclassified information on Benghazi. King said the administration has "hidden behind" the claim that Rice was only using the intelligence community's best assessment. But he said Petraeus' testimony suggests their best assessment conflicted with what Rice said on Sept. 16.

One source told Fox News that Petraeus "has no idea what was provided" to Rice or who was the author of the talking points she used.

"He had no idea she was going on talk shows" until the White House announced it one or two days before, the source said.

Obama in his first post-election press conference Wednesday, called the criticism against Rice "outrageous" and told those lawmakers to "go after me" instead.

California Democratic Rep. Adam Schiff also came to Rice's defense Thursday, saying after a House intelligence committee hearing that Rice was given the intelligence community's "best assessment" at the time.

"Those who have suggested that Ambassador Rice was politicizing the intelligence or misrepresenting what the intelligence community was putting forward as its best assessment are either unfamiliar with the facts, or willfully disregarding them," he said.

Fox News' Catherine Herridge contributed to this report.