First it was Shamima Begum, Amira Abase and Kadiza Sultana, three schoolgirls from Tower Hamlets who smuggled themselves to Syria during their half-term holiday. Then it was “Jihadi John”, the Islamic State executioner who was unmasked by the Washington Post last week as the Kuwaiti-born Londoner Mohammed Emwazi.

The stories of the three schoolgirls and of Emwazi are very different. But the same questions are being asked of them. How did they get radicalised? And how can we stop it from happening again? These are questions being increasingly asked across Europe. A recent report by the International Centre for the Study of Radicalisation suggests that there are now 4,000 European fighters with Isis, a figure that has doubled over the past year.

What is it that draws thousands of young Europeans to a brutal, sadistic organisation such as Isis? “Radicalisation” is usually seen as a process through which extremist groups or “hate preachers” groom vulnerable Muslims for jihadism by indoctrinating them with extremist ideas. Some commentators blame western authorities for pushing young Muslims into the arms of the groomers. The advocacy group Cage UK claimed last week that Mohammed Emwazi had been driven to Syria by MI5 “harassment”. Others stress the “pull” factor in radicalisation. The problem, they claim, lies with Islam itself, a faith that, in their eyes, legitimises violence, terror and inhumanity.

Neither claim is credible. Whatever the facts of his relationship with MI5, Emwazi himself was responsible for joining Isis. No amount of “harassment” provides an explanation for chopping off people’s heads.

Nor is Islam an adequate explanation. Muslims have been in Europe in large numbers since the 1950s. It is only in the last 20 years that radical Islam has gained a foothold. Blaming it all on Islam does nothing to explain the changing character of Muslim communities and their beliefs.

The problem with both approaches is in the idea of “radicalisation”. Marc Sageman, a former CIA operations officer who worked with the Afghan mujahideen in the 1980s, is now a distinguished academic and a counter-terrorism consultant to the US and other governments.

He said: “The notion that there is any serious process called ‘radicalisation’ is a mistake. What you have is some young people acquiring some extreme ideas – but it’s a similar process to acquiring any type of ideas. It often begins with discussions with a friend.”

European recruits to Isis are certainly hostile to western foreign policy and devoted to their vision of Islam. Religion and politics form indispensable threads to their stories. And yet the “radicalisation” argument looks at the jihadis’ journey back to front.

It begins with the jihadis as they are at the end of their journey – enraged about the west, and with a black-and-white view of Islam – and assumes that these are the reasons they have come to be as they are. But for most jihadis, the first steps on their journeys to Syria were rarely taken for political or religious reasons.

What is striking about the stories of wannabe jihadis is their diversity. There is no “typical” recruit, no single path to jihadism.

Sahra Ali Mehenni is a schoolgirl from a middle-class family in the south of France. Her father, an industrial chemist, is a non-practising Muslim, her mother an atheist. “I never heard her talk about Syria, jihad,” said her mother. One day last March, to the shock of her family, she took not her usual train to school but a flight from Marseilles to Istanbul to join Isis. When she finally phoned home it was to say: “I’ve married Farid, a fighter from Tunisia.”

Kreshnik Berisha, a German born of Kosovan parents, played as a teenager for Makkabi Frankfurt, a Jewish football club and one of Germany’s top amateur teams. He went on to study engineering and in July 2013, boarded a bus to Istanbul and then to Syria. “I didn’t believe it,” said Alon Meyer, Makkabi Frankfurt’s coach. “This was a guy who used to play with Jewish players every week. He was comfortable there and he seemed happy.” Berisha later returned home to become the first German homegrown jihadi to face trial.

There are hundreds of stories such as these, from all over Europe. What they tell us is that, shocking though it may seem, there is nothing unusual in the story of the runaway Tower Hamlets schoolgirls. And that what Emwazi has in common with other European recruits is not so much his harassment as his college education.

The usual clichés about jihadis – that they are poor, uneducated, badly integrated – are rarely true. A survey of British jihadis by researchers at London’s Queen Mary College found no link to “social inequalities or poor education”; most were highly educated young people from comfortable families who spoke English at home. According to Le Monde, a quarter of French jihadis in Syria are from non-Muslim backgrounds.

What draws most wannabe jihadis to Syria is, to begin with, neither politics nor religion. It is a search for something a lot less definable: for identity, for meaning, for “belongingness”, for respect. Insofar as they are alienated, it is not because wannabe jihadis are poorly integrated, in the conventional way we think of integration. Theirs is a much more existential form of alienation.

There is, of course, nothing new in the youthful search for identity and meaning. What is different today is the social context in which this search takes place. We live in a more atomised society than in the past; an age in which many people feel peculiarly disengaged from mainstream social institutions and in which moral lines often seem blurred and identities distorted.

In the past, social disaffection may have led people to join movements for political change, from far-left groups to anti-racist campaigns. Today, such organisations often seem equally out of touch. What gives shape to contemporary disaffection is not progressive politics but the politics of identity.

Identity politics has, over the last three decades, encouraged people to define themselves in increasingly narrow ethnic or cultural terms. A generation ago, “radicalised” Muslims would probably have been far more secular in their outlook and their radicalism would have expressed itself through political organisations. Today, they see themselves as Muslim in an almost tribal sense, and give vent to their disaffection through a stark vision of Islam.

These developments have shaped not just Muslim self-perception but that of most social groups. Many within white working-class communities are often as disengaged as their Muslim peers, and similarly see their problems not in political terms but through the lens of cultural and ethnic identity. Hence the growing hostility to immigration and diversity and, for some, the seeming attraction of far-right groups.

Racist populism and radical Islamism are both, in their different ways, expressions of social disengagement in an era of identity politics. There is something distinctive about Islamist identity. Islam is a global religion, allowing Islamists to create an identity that is intensely parochial and seemingly universal, linking Muslims to struggles across the world, from Afghanistan to Palestine, and providing the illusion of being part of a global movement.

In an age in which traditional anti-imperialist movements have faded and belief in alternatives to capitalism dissolved, radical Islam provides the illusion of a struggle against an immoral present and for a utopian future.

However, most homegrown wannabe jihadis possess a peculiar relationship with Islam. They are as estranged from Muslim communities as they are from western societies. Most detest the mores and traditions of their parents, have little time for mainstream forms of Islam and cut themselves off from traditional community institutions. It is not through mosques or religious institutions but through the internet that most jihadis discover their faith and their virtual community. Disembedded from social norms, finding their identity within a small group, shaped by black-and-white ideas and values, driven by a sense that they must act on behalf of all Muslims and in opposition to all enemies of Islam, it becomes easier for wannabe jihadis to commit acts of horror and to view such acts as part of an existential struggle between Islam and the west.

Simplistic narratives about “radicalisation” miss the complex roots of homegrown terrorism. Proposed solutions, such as banning organisations, pre-censoring online hate speech, increasing state surveillance and so on, betray our liberties without addressing the issues that has made Islamism attractive to some in the first place.

Jihadis are responsible for the choices they make. However much we may deplore western policies, at home or abroad, they provide no reason for the grotesque acts of Isis.

And yet there is an uncomfortable question to be asked of society, too. Why is it that so many intelligent and resourceful young people find an ideology that espouses mass beheadings, slave labour and the denial of rights to women more appealing than anything else that is on offer?

Kenan Malik’s most recent book is The Quest for a Moral Compass. He is also the author of From Fatwa to Jihad

• Comments will be opened later this morning