"The west is ill at ease with Islam", a BBC colleague remarked, long before 9/11. "Even communism was more familiar." Communism, after all, came from within the western intellectual tradition. Islam, in contrast, is alien as well as threatening.

Our mistake is to see Islam as monolithic. We think of the Saudi brand as the norm – as if cutting off hands, outlawing the building of churches and denying women the right to drive were the norm across the vast sweep of the Muslim world. After 30 years' experience travelling in the Muslim world – most of that time as a regional specialist with the BBC World Service – I'm still constantly startled by how many ways there are of being a Muslim in the modern world.

The failure to see Islam's extraordinary diversity has in turn hampered our efforts to understand Islamism – the notion that it is an ideology as well as a religion. Islamism is, at root, a reaction to western power. It is no accident that the archetypical Islamist movement, the Muslim Brotherhood, was born in British-ruled Egypt. Its founder, a young schoolteacher called Hasan al-Banna, believed Muslims were fighting a battle on two fronts – an internal battle to revive the faith and an external one to drive foreigners from Muslim lands. In his mind, the two were linked. "Eject imperialism from your souls", he declared, "and it will leave your lands".

Al-Banna saw Islamism as essentially a social movement. His successor as ideologue of the Brotherhood, Sayyid Qutb, saw it as a revolutionary struggle not just against the west, but against what he denounced as the apostate regimes of the Muslim world. Qutb was hanged in an Egyptian jail in 1966, becoming Islamism's first important martyr and a role model for Bin Laden and the global jihadists of today.

What turned Islamism into a truly global phenomenon was an accident of geopolitics. Two events came together in 1979 – the Khomeini revolution which overthrew the Shah of Iran, and the Soviet invasion of Afghanistan – which radicalised and internationalised Islamism in both its Sunni and Shia forms.

The paradox of Islamism is that it has captured the grassroots but for the most part failed to make the breakthrough to power. It has proved more effective as an instrument of protest than an instrument of governance.

Groups of the al-Qaida type, making skilful use of the internet, have developed a narrative of humiliation. Their message to Muslims is simple: look around the world, and you see countless conflicts in which your co-religionists are involved, invariably on the losing side. From this springs the idea that the umma – the worldwide community of the faithful – is under siege, and that every able-bodied Muslim has a duty to come to its defence.

In this narrative, the aggressive violence of the west must be met with the defensive violence of the mujahid, the holy warrior. The jihad is not only just; it washes away the stain of humiliation. The idea – put forward recently by Farid Zakaria in Newsweek – that the narrative has lost its force and al-Qaida has lost the ideological battle strikes me as wishful thinking.

Much of the talk about winning Muslim "hearts and minds" is shallow and misguided. The issue is often seen, especially in the United States, as a matter of public relations – as if America has an image problem in the Muslim world, and dollars can buy it a better one. Or it is seen, in a facile way, as a matter of bolstering "good Muslims" while clobbering "bad Muslims". Without a surer grasp of Islamism and its discontents, the battle for Muslim hearts and minds will be lost.