U.S. military sources serving in North Africa are challenging the latest White House claim that the administration is applying "all the resources" at its disposal to bring the Benghazi attackers to justice, charging instead that the Obama administration knows who is responsible but is not acting.

"They have let it slip by because of politics, and now we've taken all the correlation we had and dropped the ball because of risk (aversion) -- and now the security in Libya is more fragile than ever," one U.S. special operator told Fox News. The source, speaking on condition of anonymity, confirms that U.S. forces have tracked the alleged attackers since October but have since lost the trail of some of them, as no one up the chain of command would authorize them to capture or kill the targeted militia members.

Sources who have worked in and around Benghazi since last October spoke out after White House Press Secretary Jay Carney repeatedly said at a briefing more than a week ago that the administration was going after the suspects in the Sept. 11 terror attack. "From the beginning, the president has committed all the resources of this administration, of this government, to finding out who was responsible and to bringing them to justice," Carney said, as he faced a barrage of critical questions from the press on the heels of reports that challenged the administration's Benghazi narrative.

"Carney just said they want to bring those responsible to justice -- that's a big ole negative," said one special operator who watched the press conference with part of his team and disputed Carney's characterization of the administration's efforts in the wake of the attack.

According to well-placed sources, the administration has known where some of the perpetrators are, based on information given to the Pentagon back in January, but no action has been taken to capture or kill them.

More On This...

Further, sources said they are being restricted from any reconnaissance or advanced force operations to go after those responsible in the eastern part of Libya.

"We know exactly where the mastermind lives," one U.S. official said.

U.S. intelligence sources claim the "mastermind" and other suspects are on video that night at the U.S. compound, and that investigators have other evidence. Fox News reported last fall that investigators have visually identified several attackers at the compound, including one who was caught in Turkey, deported to Tunisia and eventually released by the Tunisian government due to a "lack of evidence."

The information on potential targets, sources said, went to the military's AFRICOM command and the Defense Secretary's office in January, after more evidence had been collected.

Representatives from the Pentagon did not return requests for comment.

The State Department would need to be on board for any operation to take place inside Libya.

The Justice Department referred FoxNews.com to Attorney General Eric Holder's testimony on Capitol Hill May 15.

Asked at the time for an update on the FBI investigation on the attack, Holder said: "I can't be definitive other than to say the investigation is ongoing, that we are at a point where we have taken steps that I would say are definitive, concrete, and we will be prepared shortly I think to reveal all that we have done."

Sources who spoke to Fox News, who also fought in Iraq and Afghanistan, said the consequences of inaction in tracking the Libya suspects are very dangerous.

"The place is becoming a safe haven for Al Qaeda, and terrorists are being indoctrinated and trained. There are also known foreigners who are now traveling to Libya in the eastern mountainous regions, that are training to conduct attacks in Europe and throughout the region," one source said. "If nothing is done you will see fallout of terrorist attacks on westerners on a scale unlike we've never seen."

Sources said a strike on the suspect or suspects does not have to be unilateral, but could be carried out with other partner nations.

As part of its investigation, the FBI recently posted images of three people wanted for questioning regarding the Benghazi terror attack. The FBI released the grainy images on its website, claiming the individuals pictured were at the U.S. compound when it was attacked on Sept. 11.

"We are seeking information about three individuals who were on the grounds of the U.S. Special Mission when it was attacked," the FBI said in a statement. "These individuals may be able to provide information to help in the investigation."

The Associated Press also reported Tuesday that, according to unnamed officials, the U.S. has identified five men who might be behind the attack. The AP reported that the U.S. has enough evidence to justify using military force to seize them as suspected terrorists - but there is not enough proof to try them in civilian court.

The men are still at large, though. Sources told Fox News that any operation now to go after those responsible is much more dangerous because they have lost track of some of the targets due to the fact they've been held back.

The FBI is encouraging anyone with information to text or email BenghaziTips@ic.fbi.gov or submit information confidentially by going to https://forms.fbi.gov/benghazi.

Carney was asked about the FBI photos at the May 10 press briefing, questioned on why the FBI "just got around" to releasing the images.

Carney argued that that "is probably not a characterization that reflects the very hard work that the FBI is engaged in."

The White House has repeatedly asked the American people for patience in bringing the perpetrators to justice, invoking the successful mission to kill Usama bin Laden.

Carney said: "You can believe -- and I think this president has a record to prove it -- that he will keep focused on this until those who are responsible are brought to justice. And again, I think this president has a record that backs that up."

The concern of some who have been on the ground in Benghazi in recent months is that some of the information obtained after the Sept. 11 attack was time-sensitive and may have expired.