Located just a seven-minute walk from the White House, this palace-like Russian consular property in the heart of Washington, D.C. has hosted more than a fair share of spies, defectors, and even atomic bomb scares throughout the 20th century.

While the Cold War public zeitgeist focused on the threat of Soviet planes and later missiles, government leaders may have been kept awake at night by a threat much closer to home. In the 1950s and 1960s, U.S. war planners reportedly assumed that a low-yield atomic bomb was sitting in the attic of the Soviet Embassy at this site.

This insane-if-true historical nugget was related by Hugh Sidey, the well-sourced White House correspondent for Time Magazine who covered every POTUS from Eisenhower to G. W. Bush. In 1961, after returning from the Vienna Summit with John F. Kennedy, Sidey sat down at the Palm Beach “Winter White House” to discuss the trip over daiquiris and fish-in-a-bag.

As President Kennedy recounted the pushback he had encountered on the topic of West Berlin—a thorny NATO outpost in the heart of the Soviet Bloc—Sidey joked that there was an analogous face-off to be found back home in Washington. “We have a bustling communist enclave just four blocks from the White House,” Sidey joked, alluding to the Soviet Embassy on 16th Street. Kennedy paused, fork in the air—“You know” he shot back in a deadpan Massachusetts accent, “they have an atom bomb on the third floor of the embassy.”

As Kennedy told it, the Soviets had smuggled nuclear material and bomb components into the country using diplomatic pouches that have international protections against inspections or x-ray. Most plausibly a low-yield, semi-portable weapon (later nicknamed a “suitcase nuke” or “backpack nuke”), such a weapon had the capability of evading first strike warning systems and kicking off World War III with an unstoppable attack.

Sidey was skeptical of this bombshell and brushed it off as a “Sure, why not?” exaggeration. And while he didn’t file a news story at the time, Sidey could still recall Kennedy’s response decades later. “That’s what I’m told,” Kennedy quipped, fork and fish still in hand, “Do you know something that I don’t?”

The smuggled atom bomb story seems a little too incredible to believe today, but the fact that President Kennedy and others took it seriously at the time is an illuminating piece of historical information on its own.