The officials said Mr. Gates and Ms. Rice expressed their concerns about Guantánamo in conversations with Mr. Bush and others, including Mr. Gonzales, beginning in January and onward. One widely discussed alternative would move the prisoners to military brigs in the United States, where they would remain in the custody of the Pentagon and would be subject to trial under military proceedings. There is widespread agreement, however, that moving any detainees or legal proceedings to American territory could bring significant complications.

Some administration lawyers are deeply reluctant to move terrorism suspects to American soil because it could increase their constitutional and statutory rights — and invite an explosion of civil litigation. Guantánamo was chosen because it was an American military facility but not on American soil.

Placing the detainees in military brigs on United States territory might fend off some of those challenges. The solution may eventually require a new act of Congress establishing legal standing for the detainees and new rules for their trial and incarceration if brought to the United States.

Mr. Gates’s criticism of Guantánamo marks a sharply different approach than the one taken by his predecessor, Donald H. Rumsfeld. It also demonstrated a new dynamic in the administration, in which Mr. Gates was teaming up with Ms. Rice, who often was at loggerheads with Mr. Rumsfeld. The State Department has long been concerned about the adverse foreign-policy impact of housing prisoners at Guantánamo.

In the end, Mr. Gates did succeed in killing plans to build a $100 million courthouse and detention complex at Guantánamo, after he argued that the large and expensive project would leave the impression of a long-lasting American detainee operation there and that the money could be more effectively spent elsewhere by the Pentagon. Mr. Gates approved a far more modest facility at one-tenth of the cost.

Image Camp Delta, the fenced-in core of the Guantánamo prison. Credit... Angel Franco/The New York Times

The setback in his effort to close Guantánamo was described by senior Pentagon officials as Mr. Gates’s only significant failure during an effort in his first three months in office to shift course from policies pursued by Mr. Rumsfeld. The outcome suggests that Mr. Bush, Mr. Cheney and Mr. Gonzales remain committed to a detention plan that has become one of the most controversial elements of the administration’s counterterrorism program.