Soon after the Libyan capital fell to the rebels in August 2011 I got to know a 32-year-old man called Ahmed Abdullah al-Ghadamsi. We met when he tried to evict me from my hotel room, which he said was needed for members of the National Transitional Council, in effect the provisional government of Libya. I wasn’t happy about being moved because the hotel, the Radisson Blu on Tripoli’s seafront, was full of journalists and there was nowhere else to stay. But Ahmed promised to find me another room, and he was as good as his word.

He was lending a hand to the provisional government, he said, because he was strongly opposed to Gaddafi – as was the rest of his family. He came from the Fornaj district of the city, and was contemptuous of the efforts of government spies to penetrate its network of extended families. He derided Gaddafi’s absurd personality cult and his fear of subversive ideas: ‘Books used to be more difficult to bring into the country than weapons. You had to leave them at the airport for two or three months so they could be checked.’ He had spent six years studying in Norway and spoke Norwegian as well as English; on returning to Libya he got a job on the staff of the Radisson Blu. One of Gaddafi’s sons, Al-Saadi, had a suite in the hotel, and he watched the ruling family and their friends doing business and enjoying themselves.

Ahmed was a self-confident man, not noticeably intimidated by the sporadic shooting which was keeping most people in Tripoli off the streets. I asked him if he would consider working for me as a guide and assistant and he agreed. Tripoli had run out of petrol but he quickly found some, along with a car and driver willing to risk the rebel checkpoints. He was adept at talking to the militiamen manning the barricades, and helped me get out of the city when the roads were blocked. After a few weeks I left Libya; I later heard that he was working for other journalists. Then in October I got a message saying that he was dead, shot through the head by a pro-Gaddafi sniper in the final round of fighting in Sirte on the coast far to the east of Tripoli. It turned out that there was a lot that Ahmed hadn’t told me.

When the protests started in Benghazi on 15 February he had been among the first to demonstrate in Fornaj, and he was arrested. His younger brother Mohammed told me that ‘he was jailed for two hours or less before his friends and the protesters broke into the police station and freed him.’ When Gaddafi’s forces regained control of Tripoli, Ahmed drove to the Nafusa Mountains a hundred miles south-west of the capital to try to join the rebels there, but they didn’t know or trust him so he had to return. He smuggled weapons and gelignite into Tripoli and became involved in a plot, never put into action, to blow up Al-Saadi Gaddafi’s suite in the Radisson. Mohammed said Ahmed felt bad that he’d spent much of the revolution making money and, despite his best efforts, had never actually fought. He went to Sirte, where Gaddafi’s forces were making a last stand, and joined a militia group from Misrata. He had no military experience, as far as I know, but he didn’t flinch during bombardments and was stoical when he was caught in an ambush and wounded by shrapnel from a mortar bomb, and the militiamen were impressed. On 8 October his commander told Ahmed to take a squad of five or six men to hunt for snipers who had killed a number of rebel fighters. He was shot dead by one of them a few hours later.

What would Ahmed think of the Libyan revolution now? An interim government is nominally in control but the streets of Tripoli and Benghazi have been full of militia checkpoints manned by some of the 225,000 registered militiamen whose loyalty is to their commanders rather than the state that pays them. When demonstrators appeared outside the headquarters of the Misrata militia in Tripoli on 15 November demanding that they go home, the militiamen opened fire with everything from Kalashnikovs to anti-aircraft guns, killing 43 protesters and wounding some four hundred others. This led to popular protests in which many militias were forced out of Tripoli, though it’s not clear whether this is permanent. Earlier the prime minister, Ali Zeidan, was kidnapped by militia gunmen without a shot being fired by his own guards to protect him. (He was released after a few hours.) Mutinying militias have closed the oil ports to exports and eastern Libya is threatening to secede. The Libyan state has collapsed, for the simple reason that the rebels were too weak to fill the vacuum left by the fall of the old regime. After all, it was Nato airstrikes, not rebel strength, that overthrew Gaddafi.

It’s a similar story elsewhere in the Middle East. The uprisings of the Arab Spring have so far produced anarchy in Libya, a civil war in Syria, greater autocracy in Bahrain and resumed dictatorial rule in Egypt. In Syria, the uprising began in March 2011 with demonstrations against the brutality of Assad’s regime. ‘Peace! Peace!’ protesters chanted. But ‘if there was a fair election in Syria today,’ one commentator said, ‘Assad would probably win it.’

It isn’t only the protesters and insurgents of 2011 whose aspirations are being frustrated or crushed. In March 2003 the majority of Iraqis from all sects and ethnic groups wanted to see the end of Saddam’s disastrous rule even if they didn’t necessarily support the US invasion. But the government now in power in Baghdad is as sectarian, corrupt and dysfunctional as Saddam’s ever was. There may be less state violence, but only because the state is weaker. Its methods are equally brutal: Iraqi prisons are full of people who have made false confessions under torture or the threat of it. An Iraqi intellectual who had planned to open a museum in Abu Ghraib prison so that Iraqis would never forget the barbarities of Saddam’s regime found that there was no space available because the cells were full of new inmates. Iraq is still an extraordinarily dangerous place. ‘I never imagined that ten years after the fall of Saddam you would still be able to get a man killed in Baghdad by paying $100,’ an Iraqi who’d been involved in the abortive museum project told me.

Why have oppositions in the Arab world and beyond failed so absolutely, and why have they repeated in power, or in pursuit of it, so many of the faults and crimes of the old regimes? The contrast between humanitarian principles expressed at the beginning of revolutions and the bloodbath at the end has many precedents, from the French Revolution on. But over the last twenty years in the Middle East, the Balkans and the Caucasus the rapid degradation of what started as mass uprisings has been particularly striking. I was in Moscow at the start of the second Russo-Chechen war in October 1999, and flew with a party of journalists to Chechnya to see the Chechen president, Aslan Maskhadov, in his headquarters in Grozny, where he was desperately trying – and failing – to avert the Russian assault by calling for a ceasefire. We were housed in a former barracks which seemed worryingly vulnerable to Russian air attack. But it soon became evident that the presidential guard’s greatest anxiety was that we would be abducted by Chechen kidnappers and held for ransom. The first Chechen revolt in 1994-96 was seen as a heroic popular struggle for independence. Three years later it had been succeeded by a movement that was highly sectarian, criminalised and dominated by warlords. The war became too dangerous to report and disappeared off the media map. ‘In the first Chechen war,’ one reporter told me, ‘I would have been fired by my agency if I had left Grozny. Now the risk of kidnapping is so great I would be fired for going there.’

The pattern set in Chechnya has been repeated elsewhere with depressing frequency. The extent of the failure of the uprisings of 2011 to establish better forms of governance has surprised opposition movements, their Western backers and what was once a highly sympathetic foreign media. The surprise is due, in part, to a misunderstanding of what the uprisings were about. Revolutions come into being because of an unpredictable coincidence of forces with different motives targeting a common enemy. The political, social and economic roots of the upsurges of 2011 go deep. That this wasn’t obvious to everyone at the time is partly a result of the way foreign commentators exaggerated the role of new information technology. Protesters, skilled in propaganda if nothing else, could see the advantage of presenting the uprisings to the West as unthreatening ‘velvet’ revolutions with English-speaking, well-educated bloggers and tweeters prominently in the vanguard. The purpose was to convey to Western publics that the new revolutionaries were comfortingly similar to themselves, that what was happening in the Middle East in 2011 was similar to the anti-communist and pro-Western uprisings in Eastern Europe after 1989.

Opposition demands were all about personal freedom: social and economic inequality were rarely declared to be issues, even when they were driving popular rage against the status quo. The centre of Damascus had recently been taken over by smart shops and restaurants, but the mass of Syrians saw their salaries stagnating while prices rose: farmers ruined by four years of drought were moving into shanty towns on the outskirts of the cities; the UN said that between two and three million Syrians were living in ‘extreme poverty’; small manufacturing companies were put out of business by cheap imports from Turkey and China; economic liberalisation, lauded in foreign capitals, concentrated wealth in the hands of a politically well-connected few. Even members of the Mukhabarat, the secret police, were trying to survive on $200 a month. ‘When it first came to power, the Assad regime embodied the neglected countryside, its peasants and neglected underclass,’ an International Crisis Group report says. ‘Today’s ruling elite has forgotten its roots. It has inherited power rather than fought for it … and mimicked the ways of the urban upper class.’ The same was true of the quasi-monarchical families and their associates operating in parallel fashion in Egypt, Libya and Iraq. Confident of their police-state powers, they ignored the hardships of the rest of the population, especially the underemployed, overeducated and very numerous youth, few of whom felt that they had any chance of improving their lives.

The inability of new governments across the Middle East to end the violence can be ascribed to a simple-minded delusion that most problems would vanish once democracies had replaced the old police states. Opposition movements, persecuted at home and often living a hand to mouth existence in exile, half-believed this and it was easy to sell to foreign sponsors. A great disadvantage of this way of seeing things was that Saddam, Assad and Gaddafi were so demonised it became difficult to engineer anything approaching a compromise or a peaceful transition from the old to a new regime. In Iraq in 2003 former members of the Baath Party were sacked, thus impoverishing a large part of the population, which had no alternative but to fight. The Syrian opposition refuses to attend peace talks in Geneva if Assad is allowed to play a role, even though the areas of Syria under his control are home to most of the population. In Libya the militias insisted on an official ban on employing anyone who had worked for Gaddafi’s regime, even those who had ended their involvement thirty years before. These exclusion policies were partly a way of guaranteeing jobs for the boys. But they deepen sectarian, ethnic and tribal divisions and provide the ingredients for civil war.

What is the glue that is meant to hold these new post-revolutionary states together? Nationalism isn’t much in favour in the West, where it is seen as a mask for racism or militarism, supposedly outmoded in an era of globalisation and humanitarian intervention. But intervention in Iraq in 2003 and Libya in 2011 turned out to be very similar to imperial takeover in the 19th century. There was absurd talk of ‘nation-building’ to be carried out or assisted by foreign powers, who clearly have their own interests in mind just as Britain did when Lloyd George orchestrated the carve-up of the Ottoman Empire. A justification for the Arab leaders who seized power in the late 1960s was that they would create powerful states capable, finally, of giving reality to national independence. They didn’t wholly fail: Gaddafi played a crucial role in raising the price of oil in 1973 and Hafez al-Assad created a state that could hold its own in a protracted struggle with Israel for predominance in Lebanon. But to opponents of these regimes nationalism was simply a propaganda ploy on the part of ruthless dictatorships concerned to justify their hold on power. But without nationalism – even where the unity of the nation is something of a historic fiction – states lack an ideology that would enable them to compete as a focus of loyalty with religious sects or ethnic groups.

It’s easy enough to criticise the rebels and reformers in the Arab world for failing to resolve the dilemmas they faced in overturning the status quo. Their actions seem confused and ineffective when compared to the Cuban revolution or the liberation struggle in Vietnam. But the political terrain in which they have had to operate over the last twenty years has been particularly tricky. The dissolution of the Soviet Union in 1991 meant that the endorsement or tolerance of the US – and the US alone – was crucial for a successful takeover of power. Nasser was able to turn to Moscow to assert Egyptian independence in the Suez crisis of 1956, but after the Soviet collapse smaller states could no longer find a place for themselves between Moscow and Washington. Saddam said in 1990 that one of the reasons he invaded Kuwait when he did was that in future such a venture would no longer be feasible as Iraq would be faced with unopposed American power. In the event, he got his diplomatic calculations spectacularly wrong, but his forecast was otherwise realistic – at least until perceptions of American military might were downgraded by Washington’s failure to achieve its aims in Afghanistan as well as Iraq.

So the insurgencies in the Middle East face immense difficulties, and they have faltered, stalled, been thrown on the defensive or apparently defeated. But without the rest of the world noticing, one national revolution in the region is moving from success to success. In 1990 the Kurds, left without a state after the fall of the Ottomans, were living in their tens of millions as persecuted and divided minorities in Turkey, Iran, Iraq and Syria. Rebellion in Iraq during the Iran-Iraq War of 1980-88 failed disastrously, with at least 180,000 killed by poison gas or executed in the final days of the conflict. In Turkey, guerrilla action by the Kurdistan Workers Party (PKK), who combined Marxism-Leninism with Kurdish nationalism, began in 1974 but by the end of the 1990s it had been crushed by the Turkish army; Kurds were driven into the cities; and three thousand of their villages were destroyed. In north-east Syria, Arab settlers were moved onto Kurdish land and many Kurds denied citizenship; in Iran, the government kept a tight grip on its Kurdish provinces.

All this has now changed. In Iraq the Kurdistan Regional Government (KRG), though it shares power with the central government in Baghdad, is close to becoming an oil-rich independent state, militarily and diplomatically more powerful than many members of the UN. Until recently the Turks would impound any freight sent to the KRG if the word ‘Kurdistan’ appeared in the address, but in November the KRG president, Massoud Barzani, gave a speech in the Turkish Kurd capital of Dyarbakir and talked of ‘the brotherhood of Turks and Kurds’. Standing with him was the Turkish prime minister, Recep Tayyip Erdogan, who spoke of ‘Kurdistan’ as if he’d forgotten that a few years ago the name had been enough to land anyone who uttered it in a Turkish jail. In Syria meanwhile, the PKK’s local branch has taken control of much of the north-east corner of the country, where two and a half million Kurds live.

The rebellion in the Kurdish heartlands has been ongoing for nearly half a century. In Iraq the two main Kurdish parties, Barzani’s Kurdistan Democratic Party and Jalal Talabani’s Patriotic Union of Kurdistan, were expert at manipulating foreign intelligence services – Iranian, Syrian, American and Turkish – without becoming their permanent puppets. They built up a cadre of well-educated and politically sophisticated leaders and established alliances with non-Kurdish opposition groups. They were lucky that their worst defeat was followed by Saddam’s self-destructive invasion of Kuwait, which enabled them to take control of an enclave protected by US airpower in 1991. At this point, despite having gained more independence than any previous Kurdish movement, the KDP and PUK embarked on a vicious civil war with the Iraqi state. But then they had another stroke of luck when 9/11 provided the US with the excuse to invade and overthrow Saddam. The Kurdish leaders positioned themselves carefully between the US and Iran without becoming dependent on either.

It isn’t yet clear how the bid of thirty million Kurds for some form of national self-determination will play out, but they have become too powerful to be easily suppressed. Their success has lessons for the movements of the Arab Spring, whose failure isn’t as inevitable as it may seem. The political, social and economic forces that led to the ruptures of 2011 are as powerful as ever. Had the Arab opposition movements played their cards as skilfully as the Kurds, the uprisings might not have foundered as they have done.

None of the religious parties that took power, whether in Iraq in 2005 or Egypt in 2012, has been able to consolidate its authority. Rebels everywhere look for support to the foreign enemies of the state they are trying to overthrow, but the Kurds are better at this than anyone else, having learned the lesson of 1975, when Iran betrayed them to Saddam by signing the Algiers Agreement, cutting off their supply of arms. The Syrian opposition, by contrast, can only reflect the policies and divisions of its sponsors. Resistance to the state was too rapidly militarised for opposition movements to develop an experienced national leadership and a political programme. The discrediting of nationalism and communism, combined with the need to say what the US wanted to hear, meant that they were at the mercy of events, lacking any vision of a non-authoritarian nation state capable of competing with the religious fanaticism of the Sunni militants of al-Qaida, and similar movements financed by the oil states of the Gulf. But the Middle East is entering a long period of ferment in which counter-revolution may prove as difficult to consolidate as revolution.