''But this was a subterfuge,'' said Mr. Costello. ''To award Rochefort the medal would amount to an admission that Washington had committed an intelligence blunder.''

'He Was Yanked Out of There'

The Navy would not comment on the earlier decisions not to award the medal. But Arthur Davidson Baker 3d, an official adviser to Navy Secretary John Lehman Jr., said this was the first time such a proposal had come before Mr. Lehman, who has held the Navy post since 1981.

However, he added, ''Somebody sure as hell did Rochefort in, because he was yanked out of there.''

Admiral Nimitz's recommendation of the medal for Captain Rochefort was turned down by Adm. Ernest J. King, chief of naval of operations, who is now dead.

Admiral Layton said two people who worked to undermine Captain Rochefort, and strongly advised Admiral King against singling him out for the award, were two brothers, Capt. John R. Redman and Comdr. Joseph R. Redman. Captain Redman had been officer in charge of the research desk at naval intelligence in Washington; Commander Redman was deputy director of naval communications. Both are now dead.

A Rival Team

For Captain Rochefort to have received the medal, Admiral Layton writes, would have been a tacit admission that the principal intelligence breakthrough had been made by his team. ''Worse, it would not have squared with the Redmans' claims that their Negat team was responsible for the crucial cryptanalytic success.''

Capt. Wesley A. Wright, a leading cryptanalyst of World War II who served in Pearl Harbor and Washington, recently said of the Redman brothers, ''Their specialty was the Office of Naval Communications, and they were very good at it. They felt strongly that naval intelligence that was derived from intercepted enemy messages should be controlled by their office in Washington.''

Admiral King accepted his chief of staff's recommendation that the medal be disapproved, in part because the intelligence work done by Washington had been ''of as high an order as that done in Honolulu.''