When the Arab spring spread across the Middle East eight years ago, many observers were filled with optimism. Years of sclerotic dictatorship were over, repressive regimes would fall, a wave of progressive politics would sweep across the region. But this was always unlikely, as the more astute commentators made clear at the time. Many of the authoritarian states remained strong, buttressed by patronage networks, vested interests and regional or international support. Opposition movements were undermined by ideological disagreements, ethnic or other divides and determined, brutal repression. Brave teenagers with inspiring slogans proved no match for the teargas and tanks of regimes, nor for the calculations of distant powers.

One key factor often missed was that many of the people in states as diverse as Tunisia, Egypt, Syria, Saudi Arabia and Iran did not admire the values, systems and lifestyles of the west as much as some hoped. To be anti-regime did not necessarily mean to be pro-western. The relationship of many liberals in the Middle East with democracy, individualism, western ideas and institutions was complicated and conflicted, freighted with historical grievance as much as aspiration.

If many of those grievances go back centuries, the decade before the Arab spring added new ones. Emma Sky is uniquely qualified to consider them. In her first book, The Unravelling, published to justified critical acclaim, she described her time in charge of a province in northern Iraq and then as a senior adviser to the top US general in the country. Sky, as someone who opposed the war but went on to work incredibly hard to mitigate its worst consequences, was, and remains, an important voice for anyone who witnessed those extraordinary events first-hand. Her account of the chaos and complacency of the US-led effort to turn Iraq into a pro-western democracy will be a significant work for historians seeking to understand what went wrong and why.

An Iraqi soldier at a checkpoint east of Baghdad in January 2014. Photograph: Ali Al-Saadi/AFP/Getty Images

I covered the invasion of 2003 and its aftermath and can still vividly remember the sights, sounds and smells of Baghdad, Basra, Ramadi, Kirkuk and Fallujah in those years when US troops fought through dusty towns on the Euphrates, communities took up arms against each other and the most virulent form of Islamic extremism seen in recent decades first evolved and then took hold. If it seems distant now, Sky makes very clear in her new book that we are very much living with the consequences.

To do so she travels across the Middle East as a simple observer, reporter and traveller, albeit one with a spectacular range of contacts. She stays with ambassadors and dines with tribal sheikhs but also wanders off on her own – into markets, mosques, ancient madressas and mountain villages. This book is not for everyone. Sky’s clear, unadorned and unpretentious style is a reflection of her intellectual and emotional commitment to a region that she evidently loves and understands, but readers looking for lyrical, literary prose will be disappointed. Accounts of the complexities of the parts of the region she knows best are riveting; stories of pony trekking in Kyrgyzstan, kayaking in Sudan or, back in the UK, dining with ministers in the Houses of Parliament less so.

Many of the best passages deal with Iraq and Syria and the two individuals who emerge from these pages as especially blameworthy for the hideous mess in both countries are Nouri al-Maliki, the Shia prime minister of Iraq from 2006 to 2014; and the man who was ultimately responsible for US policy in the two countries: Barack Obama. Maliki ran an administration that was not only secretive, paranoiac, corrupt and incompetent but aggressively sectarian too; it pursued a series of policies that marginalised and penalised Iraq’s Sunni minority. This systematic discrimination contributed immensely to the rise of Islamic State in Iraq, which exploited the slide of Syria into chaos before seizing the northern city of Mosul in 2014. Within a year, it had launched major attacks in Europe. For Sky, Obama’s decision not to become involved in the Middle East may have seemed reasonable after the mistakes of his predecessor, but too often amounted to a dereliction of duty.

Nuri al-Maliki addressing delegates at a conference in central London, 2009. Photograph: Leon Neal/AFP/Getty Images

In a Time of Monsters – the title is taken from the Italian Marxist theorist Antonio Gramsci – also recounts a powerful personal story as Sky struggles with the banality of life after the intensity of her time in Iraq as well as the deep ambivalence she clearly feels about her work there. “All that effort and then to see it all unravel,” she said in a 2015 interview.

The book’s final pages speak of hope and humanity. We are all travellers on this voyage through life and are connected by our common search for meaning, significance and happiness, Sky says, and makes a heartfelt appeal for empathy, collaboration and cooperation among communities, peoples and states. Sky, who is British, now lives in the US and these lines may perhaps resonate better across the Atlantic where, even if the city on the hill shines a little less brightly these days, there is still greater tolerance for such sentiments than among readers in her jaded, battered, cynical native land. In her final paragraph, she pictures “the millennial generation striding forth determined to leave the world a better place than they found it”. Let’s hope so. In the Middle East at least, as Sky knows and describes so well, its predecessors have little to be proud of.

• In a Time of Monsters is published by Atlantic (£17.99). To order a copy go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £15, online orders only. Phone orders min p&p of £1.99.