People in Baghdad are not passive victims of violence, but seek desperately to avoid their fate. In April 2004, I was almost killed by Shia militiamen of the Mehdi Army at a checkpoint at Kufa in southern Iraq. They said I was an American spy and were about to execute me and my driver, Bassim Abdul Rahman, when they decided at the last moment to check with their commander. "I believe," Bassim said afterwards, "that if Patrick had an American or an English passport [instead of an Irish one] they would have killed us all immediately."

In the following years, I saw Bassim less and less. He is a Sunni, aged about 40, from west Baghdad. After the battle for Baghdad between Shia and Sunni in 2006, he could hardly work as a driver as three-quarters of the capital was controlled by the Shia. There were few places where a Sunni could drive in safety outside a handful of enclaves.

What happened to Bassim was also to happen to millions of Iraqis who saw their lives ruined by successive calamities. As their world collapsed around them they were forced to take desperate measures to survive, obtain a job and make enough money to feed and educate their families.

Download the new Independent Premium app Sharing the full story, not just the headlines

In the US and Europe, the main measure of whether the war in Iraq is "going well" or "going badly" is the casualty figures. The number of American soldiers and Iraqi civilians being killed went down to 39 US soldiers and 599 Iraqi civilians in January. The White House is promoting the idea that the United States is finally on the road to success, if not victory, in Iraq.

On the back of this renewed optimism about the war, Senator John McCain, the premier hawk among the Republican candidates for the presidency, has been able to revive his foundering campaign and is set to be his party's nominee. Despite the scepticism of many US journalists permanently stationed in Iraq, television and newspaper newsrooms in New York and Washington (in London they are more sceptical) have largely bought into the idea that "the surge" – the wider deployment of 30,000 extra US troops since February 2006 – has succeeded.

But any true assessment of the happiness or misery of Iraqis must use a less crude index than the number of dead and injured. It must ask if people have been driven from their houses, and if they can return. It must say whether they have a job and, if they do not, whether they stand a chance of getting one. It has to explain why so few of the 3.2 million people who are refugees in Syria and Jordan, or inside Iraq, are coming back.

At the time we had our encounter with the Mehdi Army in Kufa, Bassim was living in a house in the mixed Sunni-Shia area of Jihad in south-west Baghdad. He loved the house, which had a sitting room and two bedrooms, because he had built it himself in 2001. "I didn't complete it because I didn't have enough money," he said. "But we were so happy to have our own home."

He was living there in the summer of 2006 with his wife Maha, 38, and his children Sarah, 13, Noor, eight, and Sama, three, when Shia militiamen took over Jihad. The struggle for the capital had begun on 22 February when Sunni insurgents blew up a revered Shia shrine in Samarra. Bassim fled to Syria with his family and, when he returned to Jihad three months later, he found pictures of Muqtada al-Sadr, the Shia nationalist cleric who heads the Mehdi Army, pasted to the gate of his house.

Neighbours told Bassim to get out as fast as he could before the Mehdi Army militiamen came back and killed him. He drove with his family to his father-in-law's house in the tough Sunni district of al-Khadra, where he and his wife and three children were to live in future in a single small room. He did not dare go back to his old home, but he heard about it in the summer of 2007 from a friendly Shia neighbour who said it had been taken over by militiamen. "They accused me," says Bassim, "of being a high-rank officer in the former intelligence service and because of that they got a permit [from al-Sadr's office] to take it over."

Two Shia families moved in for a couple of months and, when they left, they took all his remaining belongings. They left the house unlocked, and soon the wooden doors and other fittings were gone. The permanent loss of his home, his only possession of any value apart from his car, was a terrible blow to Bassim and his wife. "I have nothing else to lose aside from my house," he wrote to me in a sad letter in the autumn of 2007, "and because of what happened I had a heart attack. I worked as a taxi driver for a few days, but I couldn't do it any longer because of the dangerous situation and I had no other way of earning a living. Finally, I sold my car and my wife's few gold things and I will try to go to Sweden even if I have to go illegally."

Daily coronavirus briefing No hype, just the advice and analysis you need Enter your email address Continue Continue Please enter an email address Email address is invalid Fill out this field Email address is invalid Email already exists. Log in to update your newsletter preferences Register with your social account or click here to log in I would like to receive morning headlines Monday - Friday plus breaking news alerts by email Update newsletter preferences

I thought his plan to travel to Sweden was a terrible one, as Bassim spoke only Arabic and had not travelled outside Iraq, apart from a few trips to Syria and Jordan. But there was nothing I could do to dissuade him. I did not see or hear from him for six months, though I heard from his friends that his bid to reach Sweden had failed and that he was stuck in Kuala Lumpur.

Then, on 1 February, he appeared at the door of my hotel room in Baghdad, looking shrunken and miserable, and told me of his strange and disastrous odyssey.

I had originally hoped that his plan to travel illegally to Sweden was a fantasy he would never try to realise, but everything he had said in his letter turned out to be true. He had sold his car, his wife's gold jewellery and some furniture for $6,500 (about £3,300) and borrowed $1,500 from his sister and the same amount from friends. Of this, $6,900 was paid to Abu Mohammed, an Iraqi in Sweden, who provided Bassim and a friend called Ibrahim with Lithuanian passports (these turned out to be genuine, but one of Bassim's many fears over the next three months was that his passport was a fake and he would be thrown in jail). The two men went first to Damascus and then, instructed over the phone by Abu Mohammed in Sweden, they flew to Malaysia.

This would seem to be the wrong direction, but Malaysia has the great advantage of being one of the few countries to give Iraqis entry visas at the airport. Bassim and Ibrahim took rooms at the cheapest hotel they could find in Kuala Lumpur.

They were then told by Abu Mohammed to get a plane to Cambodia and take a bus to Vietnam. Though their money was fast dwindling, they did so. Somehow, still speaking only Arabic, they made their way from Phnom Penh to Ho Chi Minh City. The plan was to get a ticket to Sweden by way of France (Bassim now thinks that this was a mistake and it would have been better to travel first to Lithuania, posing as citizens returning home, but this would have left the two Iraqis with the problem of explaining to officials there why they did not speak Lithuanian).

In the check-in queue at the airport in Vietnam on 5 January this year, Bassim was desperately worried he would be detected. He had staked all his remaining money and his family's future on getting to Sweden. In fact, he and Ibrahim had little chance of being allowed on to the plane. Too many Iraqis, claiming to be citizens of small East European states, had tried this route before. Suspicious Vietnamese immigration officials took them to an investigation room where Bassim felt ill and asked for a glass of water, which was refused. He and Ibrahim continued to protest that they were Lithuanian citizens and demanded to be taken to the Lithuanian embassy, knowing full well that Lithuania is unrepresented in Vietnam.

It was all in vain. The officials guessed that they were Iraqis. They sent Bassim and Ibrahim back to Cambodia. Half-starved because he did not like the local food – "I was used to Iraqi bread," he recalled later – and with his money almost gone, Bassim made his way back to Kuala Lumpur by the end of January. He last saw his friend Ibrahim heading for Indonesia in a small boat.

Abu Mohammed in Sweden became elusive and, when finally contacted by phone after six days, admitted that "for Iraqis, all the ways from Asia to Sweden are shut". He did not offer to return Bassim's $6,900. Demoralised, and hearing that many Iraqi refugees trying to get to Europe through Indonesia simply disappeared, Bassim used his last few dollars to fly to Damascus and took a shared taxi across the desert to Baghdad. "The journey took three months but it felt like 10 years," he said. "I have lost everything."

Life in the Iraq to which Bassim has returned is said by foreigners and Iraqis alike to be getting better. Perky pieces in the foreign media breathlessly describe how Sunni children are once again playing football in al-Zahra park near the Green Zone, where they would have been murdered a year ago. "The problem," complained one American correspondent in Baghdad, "is that newsrooms back home have two mindsets – 'War Rages' and 'Peace Dawns' – and not a lot in between."

Previous claims of an improvement in security by the US or the Iraqi government had been wholly false. I remembered Paul Bremer, the US viceroy during the first year of the occupation, claiming that the Sunni insurgents were a doomed remnant battling against "the new Iraq". When Bremer left in 2004, he was shown on television clambering into one helicopter and then, when the cameras departed, scuttling on to a second aircraft in case those same insurgents might shoot him down.

In contrast to the spurious turning-points of the past, the most recent political changes in Iraq, which had led to the fall in American and Iraqi casualties, are quite real. But they differ significantly from the way in which they are portrayed in the outside world, and have less to do with al-Qa'ida and the US than the continuing struggle for power between Sunni and Shia in Iraq.

From the fall of Saddam Hussein in 2003 to the summer of 2006, the five million-strong Sunni community had battled the US and the Shia-Kurdish Iraqi government. Then, quite suddenly, last year many of the Sunni rebel groups switched sides and allied themselves with the Americans, formed the "al-Sahwa" or "Awakening" movement and declared war on al-Qa'ida.

Dramatic changes of side when enemies embrace each other are not unknown in Iraqi politics and may stem from its traditions of tribal warfare. I was in Iraqi Kurdistan in 1996 when the Kurdish leader Massoud Barzani, many of whose family and tribe had been murdered by Saddam Hussein, called in Saddam's tanks to capture the city of Arbil and to repulse an offensive by the rival Kurdish leader Jalal Talabani, now president of Iraq.

The US commander in Iraq, General David Petraeus, and the US ambassador, Ryan Crocker, are cautious about claiming too much success. But the White House and the Republicans have been quick to suggest that a turning point had been reached in the war. As in 2003, after the American overthrow of Saddam, both the Democrats and much of the American media could be easily intimidated by the fear that they were being unpatriotic or defeatist when military victory was in sight.

"The problem in Iraq is that the agenda is driven not by what is really happening, but by the perception in America of what is happening," Ahmad Chalabi, veteran of the opposition to Saddam and one of the most astute observers of the Iraqi scene, told me. A problem is that US politicians and commentators assume far greater American control of events in Iraq than is the case. The US is the most powerful player there, but it is by no means the only one.

The dramatic change of sides of Sunni guerrilla organisations such as the "1920 Revolution Brigades" and the "Islamic Army" came about for many reasons. In Anbar province west of Baghdad (perhaps one-third of Iraq by area), the Sunni tribes had become enraged by al-Qa'ida's attempt to establish total dominance, and to replace or murder traditional leaders and set up a Taliban-type state. But the Sunni community could also see that, although its guerrilla war was effective against the US, it was being defeated by the Shia who controlled the Iraqi government and armed forces after the elections of 2005.

The only source of money in Iraq is oil revenues, and the only jobs – four million, if those on a pension are included – are with the government. The Shia, in alliance with the Kurds, controlled both. "The Sunni people found that the only way to be protected from the Shia was to be allied to the Americans," said Kassim Ahmed Salman, a well-educated Sunni from west Baghdad. "Otherwise we were in a hopeless situation. For the last two years it has been possible for Sunni to be killed legally [by death squads covertly supported by the government] in Baghdad."

The "surge" – the 30,000 extra US troops implementing a new security plan in Baghdad – has helped to make Baghdad safer. In effect, they have frozen into place the Shia victory of 2006. The city is broken up into enclaves sealed off by concrete walls with only one entrance and exit.

Areas that were once mixed are not being reoccupied by whichever community was driven out. Bassim can no more reclaim, or even visit, his house in the Jihad district of Baghdad than he could a year ago. He can still work as a taxi driver only in Sunni areas. The US military and the Iraqi government are wary of even trying to reverse sectarian cleansing because this might break the present fragile truce.

"People say things are better than they were," says Zanab Jafar, a well-educated Shia woman living in al-Hamraa, west Baghdad, "but what they mean is that they are better than [during] the bloodbath of 2006. The situation is still terrible."

Baghdad still feels and looks like a city at war. There are checkpoints everywhere. "You seldom see young girls walking in the streets, or in restaurants," adds Zanab Jafar, "because their families are terrified they will be kidnapped, so they send private cars to pick them up directly from school." New shops open, but they are always in the heart of districts controlled by a single community because nobody wants to venture far from their home to shop.

For all the talk of Baghdad being safer, it remains an extraordinarily dangerous place. One Western security company is still asking $3,000 to pick a man up at the airport and drive him six miles to his hotel in central Baghdad. The number of dead bodies being picked up by the police every morning in the capital is down to three or four when once it was 50 or 60.

"People are being killed in the back streets and alleyways but not in the main roads as they were 12 months ago," says one Shia leader with a network of contacts throughout Baghdad. "About twice as many people are being killed as the government admits." This figure is still well below what it was 18 months ago, and is unlikely to return to its previous level as long as al-Qa'ida does not resume its suicide bombing campaign, using trucks loaded with a ton of explosives detonated in the middle of Shia markets or religious processions, killing and wounding hundreds. If the attacks on the two bird markets in Shia areas on 1 February, killing 99 people, are repeated, then Shia death squads will start a fresh cycle of tit-for-tat killings of Sunni.

The new element in Iraq is the development of the Awakening Council, or al-Sahwa, movement. Suddenly there is a Sunni militia, paid by the US, that has 80,000 men under arms. This re-empowers the Sunni community far more than any legislation passed by the Iraqi parliament. But it also deepens the divisions in Iraq because the leaders of the Awakening do not bother to hide their hatred and contempt for the Iraqi government.

At the end of January, I visited Abu Marouf, one of the leaders of the Awakening, in his headquarters near Khan Dari, halfway between Abu Ghraib and Fallujah. Asked about his attitude to the government of Iraqi Prime Minister Nouri al-Maliki, Abu Marouf, until recently a commander of the 1920 Revolution Brigades, said: "Maliki has got 13 divisions [in the army] most of whom are Shia, and half are from militias controlled by Iran."

In his State of the Union address, President Bush spoke of the 80,000 Awakening Council members – also labelled "concerned local citizens", as if they were respectable householders who have taken up arms against "terrorists".

The picture Bush evoked is similar to that often seen in Hollywood Westerns when outraged townsfolk and farmers, driven beyond endurance by the crimes of a corrupt sheriff or saloon owner and their bandit followers, rise in revolt. In reality, in Iraq the exact opposite has happened. The Awakening Council members of today are the "terrorists" of yesterday.

Even the police chief of Fallujah, Colonel Feisal, the brother of Abu Marouf, cheerfully explained that until he was promoted to his present post in December 2006 he was "fighting the Americans". Abu Marouf is threatening to go back to war or let al-Qa'ida return unless his 13,000 men receive long-term jobs in the Iraqi security services. The Iraqi government has no intention of allowing this because to do so would be to allow the Sunni and partisans of Saddam Hussein's regime to once again hold real power in the state.

Bizarrely, the US is still holding hundreds of men suspected of contacts with al-Qa'ida in Afghanistan and elsewhere, while in Iraq many of the Awakening members are past and, in many cases, probably current members of al-Qa'ida being paid by the US Army.

"I knew a young man, aged 17 or 18," says Kassim Ahmed Salman, "who was a friend of my brother and used to carry a PKC [a Russian light machine-gun] and fight for al-Qa'ida. I was astonished to see him a few days ago in al-Khadra where he is a lieutenant in al-Sahwa, standing together with Iraqi army officers."

The present state of Iraq is highly unstable, but nobody quite wants to go to war again. It reminds me of lulls in the Lebanese civil war during the 1970s and 1980s, when everybody in Beirut rightly predicted that nothing was solved and the fighting would start again. In Iraq the fighting has never stopped, but the present equilibrium might go on for some time.

All the Iraqi players are waiting to see at what rate the US will draw down its troop levels. The Mehdi Army is discussing ending its six-month ceasefire, but does not want to fight its Shia rivals if they are supported by American military power. Al-Qa'ida is wounded but by no means out of business. Four days after I had seen Abu Marouf, who was surrounded by bodyguards and maintains extreme secrecy about his movements, al-Qa'ida was able to detonate a bomb in a car close to his house and injure four of his guards.

Protestations of amity between Shia security men and Awakening members should be treated with scepticism. My friend, the intrepid French television reporter Lucas Menget, filmed a Shia policeman showering praise on the Awakening movement. He introduced two of its members, declaring enthusiastically to the camera: "You see, together we will defeat al-Qa'ida." Back in his police car, the policeman, lighting up a Davidoff cigarette and shaking his head wearily, explained: "I don't have a choice. I was asked to work with these killers."

Iraq remains a great sump of human degradation and poverty, unaffected by the "surge". It was not a government critic but the civilian spokesman for the Baghdad security plan, Tahseen Sheikhly, who pointed out this week that the city is drowning in sewage because of blocked and broken pipes and drains. In one part of the city, the sewage has formed a lake so large that it can be seen "as a big black spot on Google Earth".

In the coming weeks, we will see the fifth anniversary of the invasion of Iraq by American and British forces on 19 March, and the fall of Saddam Hussein on 9 April. There will be much rancorous debate in the Western media about the success or failure of the "surge" and the US war effort here.

But for millions of Iraqis like Bassim, the war has robbed them of their homes, their jobs and often their lives. It has brought them nothing but misery and ended their hopes of happiness. It has destroyed Iraq.