Jihadi groups killed more than 5,000 people last month, with Iraq topping the league table of deaths, followed by Nigeria, Afghanistan and Syria.

In 664 incidents recorded in November by the BBC World Service and researched jointly with King’s College London, the overall death toll was 5,042, or an average of 168 deaths per day and nearly twice the number of people who were killed in the 11 September 2001 attacks on America.

After Iraq, Nigeria, Afghanistan and Syria, Yemen was fifth in the deadly league table, tying with Somalia, with 37 incidents each.

The data, shared with the Guardian, provides a unique insight into the human cost, intensity, scale and geographical distribution of a phenomenon that has captured headlines and driven political and security agendas across the world.

Nearly half of the deaths from jihadi attacks were attributable to Islamic State

Suicide bombings – 650 – accounted for more deaths than any other method.

It is the first time a global figure of this kind has been collected and analysed so rigorously, said Peter Neumann of King’s College, author of The New Jihadism: A Global Snapshot. “It is as if there had been a 7/7 attack [in London in July 2005] each day, three times a day, for a whole month.”

Sixteen different jihadi groups claimed responsibility for the attacks or were identified as the perpetrators. Not surprisingly, most attacks (44% of the total) were attributed to Islamic State, which operates primarily in Iraq and Syria and shot to international notoriety when it captured the Iraqi city of Mosul last June. It was responsible for 2,206 deaths in 308 incidents.

Jabhat al-Nusra, an al-Qaida affiliate fighting the Syrian government – and now being targeted along with Isis by the US-led international coalition – killed 257 people in 35 attacks.

The Boko Haram movement in Nigeria was the second most deadly group, with 801 deaths in 30 incidents. The Taliban in Afghanistan and Pakistan were responsible for 720 deaths in 151 attacks. Al-Qaida in the Arabian Peninsula (Aqap) killed 410 people in 35 incidents in Yemen. Al-Shabaab killed 266 people in 41 incidents in Somalia and Kenya.

The statistical breakdown shows that the majority of the victims – 2,079 – were civilians and 1,952 of them military, police or officials. Jihadis accounted for 935 of total deaths. Seventy-six of the dead remain unknown. Incidents were gathered by separate teams at BBC Monitoring and another at the International Centre for the Study of Radicalisation and Political Violence at King’s College.

“It is astonishing, given that three years ago everyone in the west was saying: ‘This is over. Al-Qaida is declining. It’s dead. It has been strategically defeated,’” Neumann said. “Everyone thought that the Arab spring was going to usher in freedom and democracy. Now we have a jihadist movement across the world in places where we never suspected it would turn up. And it’s reasonable to say that it is stronger than ever before. It is also a failure of western policy to anticipate it in any shape or form.”

Jihadism was defined by the researchers as “a modern revolutionary political ideology mandating the use of violence to defend or promote a particular, very narrow vision of Sunni Islam.”

The study elaborates: “While jihad is an Islamic concept that means ‘struggle’ and has both military and spiritual connotations, the term jihadism describes a political ideology; and while many Shia groups and individuals refer to themselves as ‘jihadists’, this count focuses on a particular movement categorised by al-Qaida, its affiliates and those who subscribe to a similar philosophy.”

Researchers devised a “jihadist scoring system” reviewed by three independent academics to determine whether jihadism was indeed the predominating motivation of the groups studied. But they also emphasised the fluid nature of the subject. “In places like Libya and Syria the situation is changing fast. Hardly a month goes by without new groups announcing their existence or existing groups becoming involved in coalitions and mergers.”

As well as Isis and Jabhat al-Nusra the study defined Ahrar al-Sham, which is fighting the Syrian government, as jihadist. Neumann predicted more conflict between rival jihadi groups in future. “The overall picture is that of an increasingly ambitious, complex, sophisticated and far-reaching movement –one that seems to be in the middle of a transformation.”

The authors said they recognised the limitations of such a study, especially in the context of civil wars, where attribution remains unclear at times; reporting can be patchy, biased and delayed; data-gathering can be dangerous; and some incidents may go unreported. But they insisted that the definition and categorisation of groups as jihadist, as well as the creation and execution of rules for counting and coding incidents, were guided by the “principles of transparency, clarity, and rigour”.

• Reports on this data will be running across BBC World Service radio and BBC World News TV on 11-12 December