We can understand a great deal more about new Islamic groups if we approach at least some of them as new religious movements. Their beliefs and practices can find appeal among some, predominantly young, people not only because they are Islamic but also because they are new and provide a basis for creating alternative space for social experimentation and efforts at social change.

These groups generate social tensions and even conflicts. But they do not do so on their own. The reaction of a wider society is a large part of the story. And while these tensions and conflicts can lead to violence, they don't necessarily do so.

My research on the New Muslims movement in the northern Caucasus illustrates these points. This "radical" movement appeared in the early 1990s in a small republic of Kabardino-Balkaria, a region beset by a near-collapsing economy and suffering the repercussions of armed conflicts in neighbouring Chechnya and tensions elsewhere in the northern Caucasus. Compared with non-members, however, the New Muslims tended to be better educated and aspiring to take advantage of the new social, economic and political freedoms of the post-Soviet 1990s – yet at the same time shackled by clan-based and highly restrictive social relationships replete with bribery and tight subordination, not least of younger people. The existing "old" Islamic institutions in the region held little appeal for the young.

Communities of the New Muslims – the jamaats – on the other hand, provided a range of opportunities for these young people: in business (all members were employed), in social status (holding positions of responsibility in the jamaats), and in personal life (jamaats were drug and alcohol free). What was new about the Islam of the New Muslims was its Salafi orientation. It was radical in the sense that it provided authoritative justifications for challenging the existing social order. In an apparent paradox, the New Muslims' "return" to what they saw as a pure, Salafi Islam was in many ways an attempt to modernise social practices and create an alternative social space for what they themselves would call "halal life" and what I would call "social experimentation".

An emergence of any new religious group means a construction of a new "us" versus an existing "them", and tensions are likely to occur. So the New Muslims refused to participate in many traditional activities that their parents and grandparents saw as central to their existence as a community. However, these tensions did not have to spill over into violence. Indeed, after 1998, the New Muslims' leaders became increasingly concerned about the movement's legitimacy and made efforts to oppose militant versions of Islamism, such as Takfiri ideas. Demographic changes were doing their work, too: the New Muslims were growing older; many of them were getting families and accepting all the responsibilities that came with this.

So what leads some from these largely peaceful reformist movements to become violent? We understand this better if we reflect on how actions by both new groups and their opponents, including state agencies, can contribute to either escalating or aborting a spiral of violence. In the case of the New Muslims, there was an increasing polarisation between the movement and its opponents, with each framing the other as "false Muslims".

However, even this framing did not have to spill over into violence had not the labels applied to the New Muslims by the mass media and government agencies (such as Salafi, Wahhabi and Islamist) not become associated with violence in the region where terrorist groups had already been operating and attempting to co-opt New Muslims into their ranks. As a result, indiscriminate violence started to be used by law-enforcement agencies to curb the movement (including torture, imprisonment without trial, desecration and closure of mosques). The local terrorist groups and Chechen rebels used this further to pressure the New Muslims to join a pan-Caucasian jihad against the Russian "infidel" as the common enemy. In turn, the law-enforcement agencies responded with ever escalating violence, driving the movement's leadership ever closer to joining the terrorist jihadists. And so it happened that some New Muslims became both the victims and perpetrators of violence.

While based on research in the more politically restrictive environment of the northern Caucasus, it shows how treating people with radical and even antisocial views who are nevertheless operating within the law as potential terrorists can easily lead to real terrorists increasing their pool of recruits. Liberal democracy should use its rich repertoire of solutions capable of accommodating those who peacefully disagree with its foundations to cope with such young people, and by doing this make itself more appealing to them as they grow and mature.

• Marat Shterin will be presenting new research into anti-extremism policy at the third of the Westminster Faith Debates, organised by the AHRC Religion and Society Programme, Theos and Charles Clarke on Wednesday 7 March 2012