If Russia is a riddle wrapped in a mystery inside an enigma, the same holds true for its most famous living citizen, Mikhail Gorbachev. From March 1985 to December 1991 he was under an unrelenting national and international spotlight as the Soviet Union’s leader. He wrote several autobiographical books while in power and has written more since retirement. At least a dozen associates have published memoirs in which he features prominently. Yet in spite of all this scrutiny, key questions about the man who did more than any other to change Europe and the world in the last half of the 20th century remain without clear answers.

How did a secret reformist get chosen by deeply conservative elders to be their country’s next leader? Gorbachev felt his country needed fundamental change, so why did he not quickly develop a programme of political and economic action once he had secured the top job? Why did he fail to foresee the rise of nationalist unrest that eventually led to the Soviet Union’s fragmentation? Why did he consent to Germany’s reunification inside Nato without demanding anything in return, except cash to pay for Soviet troops’ rehousing?

William Taubman, a professor at Amherst College in Massachusetts, who won a Pulitzer prize for his biography of Khrushchev, has done a phenomenal amount of research into Gorbachev’s career, including interviews with the man himself. He relies heavily on accounts by the closest aides, in particular the sparky diary of Anatoly Chernyaev, which reveals Gorbachev’s bewildering volatility of mood as well as his intellectual contradictions. But Taubman concludes he has to leave many questions about Gorbachev unresolved. Here is a man who confided to Chernyaev in 1987 that “we’ve made a mess of socialism: nothing is left of it”, yet took time off during crises to reread Lenin’s speeches in the conviction that he could learn lessons from 70 years earlier on how to strengthen socialism in today’s context. Or take Gorbachev’s attitude to the Communist party when its cadres became increasingly vocal in resisting change by 1990. His aides were urging him to leave the party and found a social democratic alliance to compete and take its place. “I can’t let this lousy rabid dog off the leash. If I do, all this huge structure will be turned against me,” he told them, using the foul language to which he often descended in moments of anger or despair.

Facebook Twitter Pinterest Gorbachev and George Bush, Moscow, 1991. Photograph: Wojtek Laski/Getty Images

Gorbachev’s biggest contribution was to provide Russians with freedom of speech and a multi-party democracy. Internationally, it was not to have used force to retain control of eastern Europe when its ruling Communist parties started to lose their grip. Behind this strategy lay a kind of Russian isolationism that ran counter to decades of Soviet internationalism. One might expect such a dramatic shift to have been preceded by long debate. Yet Gorbachev and his colleagues hardly ever discussed it. On the day after the Berlin Wall fell, Gorbachev did not even call the politburo into session, although he found time to send messages to US President George Bush, Margaret Thatcher and François Mitterrand, saying the East German leaders had taken the right decision. He was equally relaxed about the fall of communism elsewhere in the region.

One reason was that Gorbachev was preoccupied and overwhelmed with domestic crises within the Soviet Union. What had started as a revolution from above had become a revolution from below, with street protests and revolts breaking out all over. This provoked massive resistance from conservatives in the Soviet Communist party. Gorbachev was buffeted by pressures on all sides. Taubman’s approach to this tumultuous story is chronological and Kremlin-oriented. While this means that his fast-paced narrative leaps about, accurately reflecting Gorbachev’s tactical zigzagging, it leaves insufficient space for describing the context of daily life for Soviet citizens and the mounting disillusionment with reform that led many Russians to view Gorbachev as an agent of destruction. It also means the book lacks an explanation for basic issues.

To name just a few: why did shops have such massive food shortages? How come the black market became so pervasive? What went wrong with the effort to allow the development of private enterprise and small business under the guise of co-operatives? The book jumps from crisis to crisis just as Gorbachev’s daily agenda did, but readers would have benefited from some thematic chapters looking at key topics with the advantage of scholarly hindsight, such as the role of the mass media in hastening change, or Gorbachev’s clumsy handling of Baltic nationalism, or why (a question of renewed significance since 2014) a majority of people in eastern Ukraine, and even in Crimea, voted for independence and a break from Moscow in 1991.

Facebook Twitter Pinterest Boris Yeltsin and Gorbachev, 1991. Photograph: The Asahi Shimbun via Getty Images

In a volume as comprehensive as this, it may seem cavalier to query points of emphasis. But Taubman’s section on the Afghan war is unnecessarily short. Gorbachev’s internationally negotiated withdrawal of Soviet troops while ensuring that Afghanistan’s ruling group under Mohammad Najibullah remained in power for a further three years was not a humiliation but a qualified success. It brought relief to thousands of Soviet families who feared their sons would be conscripted and killed but, like other Gorbachev accomplishments, won little gratitude from Soviet citizens. The chapter on the hardliners’ coup that put Gorbachev under arrest in his Crimean villa in August 1991 gets a mere 17 pages in a very long book. Yet it includes a surprisingly large number of paragraphs of speculation on whether Gorbachev was somehow complicit and only pretended to be detained (a claim that the plotters produced in their defence once the coup had failed).

As one of the few foreign Moscow-based reporters who had not left Russia on holiday that August, I had the extra good luck of flying on the official plane that brought Gorbachev the news of his liberation, all recounted in the Guardian and in my subsequent book. The coup was one of the most dramatic turning points in Gorbachev’s career and deserves more ample treatment than Taubman provides. No one who spoke to the just freed Gorbachev, as we did, and witnessed his euphoria could have doubted that he had genuinely been detained. He gave us a breathless account of how he threatened suicide when his senior ministers, the coup organisers, pressed him to resign.

Important too, though not covered by Taubman, was the atmosphere on the streets of Moscow during the three days the coup lasted. Although Boris Yeltsin, the other leading reformer, resisted the coup and several thousand people surrounded his headquarters in the Russian parliament to prevent his arrest, passivity and fatalism were the dominant reaction in the rest of Moscow and in other cities. Yeltsin’s call for a general strike went unheeded. But Taubman is right to devote space to Gorbachev’s missteps on his return to Moscow and ask why the president failed to go to the parliament and thank Yeltsin and the demonstrators for resisting the coup. It is another of the many questions to which Taubman replies with the phrase: it’s hard to understand.

Jonathan Steele’s is the author of Eternal Russia: Yeltsin, Gorbachev and the Mirage of Democracy.

• Gorbachev: His Life and Times is published by Simon & Schuster. To order a copy for £21.25 (RRP £25) go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.