Joseph Stalin was supposedly all-powerful, but an ordinary Soviet office worker might have received better care than he did after a stroke on the night of 1 March 1953. He had the best personal security protection, but none of it “could prevent him lying for hours in his own urine, paralysed, and without the ability to scream”. Servants and guards were worried about his non-appearance but afraid to disturb him. When his colleagues were finally called, they hovered uncertainly before finally calling a doctor – not Stalin’s personal physician, since he and a number of his colleagues had recently been arrested as traitors in the Doctors’ Plot. The terrified man recommended “absolute quiet; the application of eight medical leeches to his ears; a cold compress on his head; an enema of milk of magnesia; and the removal of his false teeth”. Stalin died without regaining consciousness on the evening of 5 March.

His closest colleagues did not kill him, despite much speculation that Joshua Rubenstein rightly deflates. But neither did they strive officiously to keep him alive. Stalin, whose health had been poor and ability to work diminished for some years, had become extremely suspicious, not just in general but specifically of them. He had launched an antisemitic campaign about which they were very dubious. He may have been gearing up for a new round of purging and deportations. Thanks to ruthless repression and belt-tightening, the country was back on its feet economically after the huge losses of the second world war. But Stalin was in no mood for introducing measures to improve living standards that many felt were overdue.

Stalin’s closest colleagues did not kill him, despite much speculation that Joshua Rubenstein rightly deflates

His last days make a dramatic story and Rubenstein tells it well – with particular attention to the anti-semitic campaign on which, as author of Stalin’s Secret Pogrom (2001), he is an expert. But it’s what happened next that is really interesting. More than half of this new book is devoted to the year after Stalin’s death, when his old gang – the Politburo members, including Vyacheslav Molotov, Georgy Malenkov and Nikita Khrushchev, who had worked with him for many years – took over, and, to almost everyone’s astonishment, launched immediately into wide-ranging fundamental reforms, including attempts to improve relations with the west.

My own book, On Stalin’s Team: the Years of Living Dangerously in Soviet Politics, published last year, covers some of the same ground as Stalin’s Last Days, though neither of us knew that at the time of our respective writing. But whereas my focus is on domestic political developments, Rubenstein’s is on the foreign and international dimensions. It is a fascinating and at the same time disconcerting story. If you grew up, as I did, with John Foster Dulles as cold war villain and bogeyman of the left, you may subsequently have wondered if he could possibly have been as bad as he was painted. Judging by Rubenstein, who is fair-minded almost to a fault, the answer is yes.

The speed and energy with which the new leaders embarked on reform was breathtaking. Already, at Stalin’s funeral, the new head of government, Malenkov, was talking about détente and the new security boss, Lavrenty Beria, about legality and constitutional protections. Within weeks, the antisemitic campaign was called off and the (mainly Jewish) Kremlin physicians accused in the Doctors’ Plot exonerated and freed. In short order, a million prisoners were released from the Gulag under an amnesty, the powers of the security police curtailed, more consumer goods and public housing promised, cultural controls loosened, and overtures made to the west about improving relations.

Rubenstein gives a fair account of all this, if a slightly grudging one. “We can dismiss the possibility that it was an act of conscience,” he writes of the new leaders’ repudiation of the Doctors’ Plot, although in fact there is substantial evidence that Stalin’s close colleagues had disliked and passively resisted his antisemitic initiatives in 1952. Rubenstein takes a similar approach to the reform programme in general, ruling out the possibility that the new leaders actually thought change was needed and concluding that they must simply have been pursuing some “pragmatic advantage”, albeit one whose nature is unclear – perhaps a “signal to the west”?

Certainly the Soviets wanted to signal their interest in easing cold war tensions. In the US, however, this was not a message people wanted to hear. A new administration led by President Dwight Eisenhower and secretary of state Dulles had taken office in January 1953 on a programme of getting tough with the Soviets and “rolling back communism”. Soviet communism was the villain of the piece, and it was disconcerting, to say the least, if it stopped playing its role properly. Time magazine found the new developments “bewildering, welcome, sinister.” Dulles was adamant that, regardless of conciliatory Soviet overtures, the US must “keep our pressures on, psychological and otherwise”. For Dulles, in the opinion of New Yorker writer AJ Liebling, Soviet relaxation was in fact a threat to US policy aims, as it might “de-frighten Europeans”.

There were other more positive responses in the west to Soviet overtures. In Britain, the shrewd and well-informed Soviet expert Edward Crankshaw left Observer readers in no doubt that something momentous was happening. Winston Churchill, nearing the end of his political career, saw a historic moment that needed to be seized, and wrote to Eisenhower to tell him so. But Eisenhower was “horrified” when Churchill called in the House of Commons for a summit as an appropriate response to post-Stalin changes in the Soviet Union. Dulles was categorically opposed to any face-to-face meetings with the Soviets, either public or private. Critical observers discerned in him an “almost superstitious fear”. Perhaps he realised that, if you have successfully turned your enemy into a devil with horns, you don’t want to give him a chance to look human.

US President Dwight Eisenhower and Swiss president Max Petitpierre. Photograph: Erich Lessing/Magnum Photos

Rubenstein’s verdict is that, in the crucial months between Stalin’s death in March and the Berlin uprising in June of 1953, the US missed a great opportunity to meet the new Soviet leaders halfway. This was partly because of poor information and advice: experts told Eisenhower that Stalin’s successors were even more dangerous and aggressive than he had been. Eisenhower himself wanted to speak over the heads of Soviet leaders to the Soviet people, who might be encouraged to rise up and overthrow tyranny.

Communism was the villain of the piece, and it would be disconcerting to the US if it stopped playing its role properly

Rubenstein states in his introduction that “for complex reasons both Soviet and western governments could not overcome the decades of distrust that divided them”. But, when you reach the end of his clear and judicious analysis of the events of 1953, that isn’t quite the conclusion that suggests itself. A summation of “fault on both sides” is usually a safe bet, and it certainly works for the origins of the cold war in the second half of the 1940s.

It would no doubt have been fully appropriate in the hypothetical situation that the west had responded to Soviet overtures in 1953, only to have détente turn sour because of problems on both sides, including internal Soviet disagreements. But that is not what happened. By Rubenstein’s own account, the Soviets made the first and subsequent moves, and the US turned them down flat without even trying to find out how far they were prepared to go.

Dulles, with his extraordinary belief that the Soviet threat as of 8 May 1953 constituted “the most terrible and fundamental” in the last 1,000 years of western civilisation, was the prime mover of this rebuff. Thus the responsibility for the missed opportunity for some kind of détente after Stalin’s death appears to lie squarely with the US.

• To order The Last Days of Stalin for £20.50 (RRP £25) go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.

• This article was amended on 23 August 2016. In a previous version, the man in the photograph with Dwight Eisenhower was misidentified. He is Swiss president Max Petitpierre.