WASHINGTON (CNN) -- The United States is keeping a close eye on Pakistan after this week's Taliban surge into the Buner district brought them just 60 miles from the capital, Islamabad.

A Taliban fighter in the district of Buner, which is only 96 kilometers from Islamabad.

A Pakistani government official said Friday that the insurgents had completely withdrawn from the district by the end of the week, but a human rights group said people in Buner were reporting that local Taliban remained in the district.

And senior U.S. officials cautioned that any withdrawal by the Taliban was likely meaningless and that the fundamentalist group now holds large areas of the country with the government seemingly unable to stop them.

"We're certainly moving closer to the tipping point," Adm. Mike Mullen, chairman of the U.S. Joint Chiefs of Staff said on NBC's "Today" show Friday.

In the interview from Afghanistan, Mullen said he was "extremely concerned" about indications the Taliban is moving closer to Pakistan's capital of Islamabad.

Syed Mohammed Javed, commissioner of the Malakand Division that includes Buner, said the Taliban withdrew on Friday without any conditions.

Earlier in the day, Taliban spokesman Muslim Khan told CNN that the militants would pull back from the district.

Pakistani Express TV showed live footage of armed and masked Taliban militants in Buner, loading pickup trucks and driving away.

Sufi Muhammed, an Islamist fundamentalist leader who has been negotiating on behalf of the Taliban, was on scene overseeing the withdrawal, police said.

U.S. military commander Gen. David Petraeus credited about 300 Pakistan's Frontier Corps with driving Taliban militants out of Buner.

The U.S. Central Command chief spoke to CNN on Friday after testifying before a Congressional panel about the need for the United States to boost its support for Pakistani counterinsurgency troops such as the Frontier Corps.

But Amnesty International's regional chief said people in Buner are reporting a different situation.

"What we're hearing from people in Buner ... is that the Taliban that have moved out are the non-local ones," Sam Zarifi, Amnesty's Asia Pacific director, told CNN. "So the local branch of the Taliban are still in place in Buner."

Amnesty International is concerned that those local Taliban will continue to enforce the Taliban's "abusive and repressive" control of Buner.

Girls over the age of 7 are forced to wear a burka, a head-to-toe covering that the Taliban say is required of Muslims under its radical interpretation of Islamic law, or sharia.

"I think we're going to see those harsh edicts stay," Zarifi said.

The Taliban installed strict Islamic law when it took over Afghanistan after a two-year fight with warlords, many of whom surrendered without a fight. They fought with a coalition of some of those warlords known as the Northern Alliance from 1996 until U.S. forces, seeking the leadership of al Qaeda after the September 11, 2001, terrorist attacks, added bombing raids to Northern Alliance ground operations and successfully drove the Taliban out of most of Afghanistan. iReport: Should the U.S. intercede in Pakistan?

But the Taliban regrouped in 2004, launching a guerrilla war against the Afghan government while operating from the tribal areas of that country and Pakistan.

Meanwhile, senior U.S. officials said that the "retreat" was likely meaningless. Control was the impetus behind the Taliban move into Buner, and the fundamentalist group now controls large areas of Pakistan, they said.

The officials said, however, that they did not believe the Taliban's goal was to take over the government of Pakistan but rather to create instability by taking advantage of Pakistan President Asif Ali Zardari's apparent inability to cope with the situation.

Zardari, they said, does not understand the gravity of the situation, remains distracted by domestic politics and appears unable to make critical decisions to deploy the army to stabilize the country.

State Department envoy Richard Holbrooke has been on the phone "nonstop" with officials in Islamabad and Washington, the officials said, providing frequent information to Secretary of State Hillary Clinton and President Obama.

Holbrooke's read on the situation, officials familiar with the conversation said, is that the Pakistani government does not know how to handle the situation and is looking to the United States for direction.

A senior military official, however, presented a more dire case. Pakistan is "rapidly deteriorating," he said. He said that he could not rule out the possibility of a military takeover in Islamabad, although he added that such a prospect was not in the immediate future.

As for Pakistan's nuclear arsenal, he said the weapons "are safe for now" but that the United States is monitoring the safety systems in place.

Administration officials said that the Pakistani military had assured the United States it would not act without an express order from the civilian leadership.

Earlier this week, Clinton warned that nuclear-armed Pakistan was in danger of falling into terrorist hands.

Before the Taliban's apparent withdrawal Friday, a local Pakistani official expressed doubt about whether the militants would leave, as they pledged to local elders on Thursday.

"Nobody can trust them," Sardar Hussain Babik, the provincial education minister, said by phone from Buner.

The Taliban have broken promises before and probably would do so again, he said.

Militants who swarmed into Buner subsequently locked up courthouses, seized court documents and battled Pakistani troops who were sent to protect residents.

The militants said they took control of the district to ensure that Islamic law was properly imposed. The Pakistani government called the land grab a breach of a recent peace agreement.

CNN's Barbara Starr, Elise Labott, Ivan Watson, Tricia Escobedo and Samson Desta contributed to this report.

All About Pakistan • The Taliban • U.S. Central Command • Asif Ali Zardari • Nuclear Weapons