Why is the Arab world – let us speak with terrible sharpness – so backward? Why so many dictators, so few human rights, so much state security and torture, so terrible a literacy rate?

Why does this wretched place, so rich in oil, have to produce, even in the age of the computer, a population so poorly educated, so undernourished, so corrupt? Yes, I know the history of Western colonialism, the dark conspiracies of the West, the Arab argument that you cannot upset the sheikhs and the kings and the autocrats, the imams and the emirs when the "enemy is at the gates". There is some truth to that. But not enough truth.

Once more the United Nations Development Programme has popped up with yet one more, its fifth, report that catalogues – via Arab analysts and academics, mark you – the retarded state of much of the Middle East. It talks of "the fragility of the region's political, social, economic and environmental structures... its vulnerability to outside intervention". But does this account for desertification, for illiteracy – especially among women – and the Arab state which, as the report admits, is often turned "into a threat to human security, instead of its chief support"?

Download the new Independent Premium app Sharing the full story, not just the headlines

As Arab journalist Rami Khouri stated bleakly last week: "How we tackle the underlying causes of our mediocrity and bring about real change anchored in solid citizenship, productive economies and stable statehood, remains the riddle that has defied three generations of Arabs." Real GDP per capita in the region – one of the statistics which truly shocked Khouri – grew by only 6.4 per cent between 1980 and 2004. That's just 0.5 per cent annually, a rate which 198 of 217 countries analysed by the CIA World Factbook bettered in 2008. Yet the Arab population – which stood at 150 million in 1980 – will reach 400 million in 2015.

I notice much of this myself. When I first came to the Middle East in 1976, it was crowded enough. Cairo's steaming, fetid streets were already jam-packed, night and day, with up to a million homeless living in the great Ottoman cemeteries. Arab homes are spotlessly clean but their streets are often repulsive, dirt and ordure spilling on to the pavements. Even in beautiful Lebanon, where a kind of democracy does exist and whose people are among the most educated and cultured in the Middle East, you find a similar phenomenon. In the rough hill villages of the south, the same cleanliness exists in every home. But why are the streets and the hills so dirty?

I suspect that a real problem exists in the mind of Arabs; they do not feel that they own their countries. Constantly coaxed into effusions of enthusiasm for Arab or national "unity", I think they do not feel that sense of belonging which Westerners feel. Unable, for the most part, to elect real representatives – even in Lebanon, outside the tribal or sectarian context – they feel "ruled over". The street, the country as a physical entity, belongs to someone else. And of course, the moment a movement comes along and – even worse – becomes popular, emergency laws are introduced to make these movements illegal or "terrorist". Thus it is always someone else's responsibility to look after the gardens and the hills and the streets.

And those who work within the state system – who work directly for the state and its corrupt autarchies – also feel that their existence depends on the same corruption upon which the state itself thrives. The people become part of the corruption. I shall always remember an Arab landlord, many years ago, bemoaning an anti-corruption drive by his government. "In the old days, I paid bribes and we got the phone mended and the water pipes mended and the electricity restored," he complained. "But what can I do now, Mr, Robert? I can't bribe anyone – so nothing gets done!"

Even the first UNDP report, back in 2002, was deeply depressing. It identified three cardinal obstacles to human development in the Arab world: the widening "deficit" in freedom, women's rights and knowledge. George W Bush – he of enduring freedom, democracy, etc etc amid the slaughter of Iraq – drew attention to this. Understandably miffed at being lectured to by the man who gave "terror" a new name, even Hosni Mubarak of Egypt (he of the constantly more than 90 per cent electoral success rate), told Tony Blair in 2004 that modernisation had to stem from "the traditions and culture of the region".

Will a solution to the Arab-Israeli war resolve all this? Some of it, perhaps. Without the constant challenge of crisis, it would be much more difficult to constantly renew emergency laws, to avoid constitutionality, to distract populations who might otherwise demand overwhelming political change. Yet I sometimes fear that the problems have sunk too deep, that like a persistently leaking sewer, the ground beneath Arab feet has become too saturated to build on.

I was delighted some months ago, while speaking at Cairo University – yes, the same academy which Barack Obama used to play softball with the Muslim world – to find how bright its students were, how many female students crowded the classes and how, compared to previous visits, well-educated they were. Yet far too many wanted to move to the West. The Koran may be an invaluable document – but so is a Green Card. And who can blame them when Cairo is awash with PhD engineering graduates who have to drive taxis?

And on balance, yes, a serious peace between Palestinians and Israelis would help redress the appalling imbalances that plague Arab society. If you can no longer bellyache about the outrageous injustice that this war represents, then perhaps there are other injustices to be addressed. One of them is domestic violence, which – despite the evident love of family which all Arabs demonstrate – is far more prevalent in the Arab world than Westerners might realise (or Arabs want to admit).

But I also think that, militarily, we have got to abandon the Middle East. By all means, send the Arabs our teachers, our economists, our agronomists. But bring our soldiers home. They do not defend us. They spread the same chaos that breeds the injustice upon which the al-Qa'idas of this world feed. No, the Arabs – or, outside the Arab world, the Iranians or the Afghans – will not produce the eco-loving, gender-equal, happy-clappy democracies that we would like to see. But freed from "our" tutelage, they might develop their societies to the advantage of the people who live in them. Maybe the Arabs would even come to believe that they owned their own countries.