Story highlights U.S. defense secretary, Afghan president talk by phone about controversy

The United States handed over a prison at Bagram Air Base, a spokesman says

The U.S. military is temporarily halting the handover of a number of detainees

"We have paused the transfer of the remaining detainees," a spokesman says

The United States declined Monday to surrender an undisclosed number of detainees during the handover of a prison to Afghan control, a spokesman for the American-led coalition in Afghanistan said.

The handover of Parwan prison at Bagram Air Base was the linchpin in a larger agreement that outlines the gradual transfer of power from U.S. forces to Afghanistan as American troops prepare to withdraw.

A coalition official told CNN the United States is holding on to several Afghan detainees because of concerns about whether Afghan authorities will properly handle their cases and under what circumstances they might be released. The U.S. also is keeping several prisoners of other nationalities who were not part of the agreement, the source said.

U.S. Secretary of Defense Leon Panetta and Afghan President Hamid Karzai spoke by telephone Monday about the brewing controversy. Pentagon spokesman George Little said the two leaders "expressed a shared commitment to implement the terms of the memorandum of understanding on detention operations in Afghanistan."

JUST WATCHED A daunting transition in Afghanistan Replay More Videos ... MUST WATCH A daunting transition in Afghanistan 03:11

Roughly 99% of the detainees in custody before the agreement was signed in March have been turned over to Afghan authorities, Jamie Graybeal, a spokesman for coalition, said by telephone.

"We have paused the transfer of the remaining detainees until our concerns are met," he said.

Graybeal declined to release the number of detainees still in U.S. custody, citing operational security.

The detainees in question are believed to be high-value Taliban and Haqqani network militants. The U.S. military has not publicly identified them.

The U.S. military also is temporarily halting the handover of any detainees taken into custody after the agreement was signed in March.

Graybeal did not outline what the concerns were or what steps needed to be taken before the U.S. military agreed to turnover the detainees.

While the handover ceremony at Bagram occurred Monday despite the U.S. military's refusal to surrender the detainees, the issue will likely be a sticking point between Afghan President Hamid Karzai and U.S. military leaders.

Karzai has been adamant that all prisoners be under the control of Afghan authorities.

As late as last week, there was an indication of a possible disagreement between Karzai and U.S. Marine Gen. John Allen, commander of NATO troops.

In November, a loyal jirga, or grand assembly of Afghan elders, endorsed the continued presence of U.S. forces following the end of combat operations in 2014, though only if the two countries could agree on the transfer of prisoners, the end to night raids and lifting immunity for U.S. troops accused of committing crimes.

Karzai more recently released a statement following a meeting with Allen in which he said that under the agreement, "the prison is now supposed to be transferred to Afghan government and thus any delay in its handover is considered a breach of Afghan national sovereignty."