There’s nothing like a riot to bring out the amateur psychologist in all of us. Consider what happened in August 2011, after police killed Mark Duggan, a 29-year-old man from the London suburb of Tottenham. Thousands took to the streets of London and other English towns in the UK’s worst outbreak of civil unrest in a generation. When police finally restored order after some six days of violence and vandalism, everyone from the Prime Minister David Cameron to newspaper columnists of every political persuasion denounced the mindless madness, incredulous that a single killing, horrific as it was, could spark the conflagration at hand. The most popular theory was that rioters had surrendered their self-awareness and rationality to the mentality of the crowd.

This has been the overriding view of crowd behaviour since the French Revolution and the storming of the Bastille. The 19th-century French criminologist Gabriel Tarde likened even the most civilised of crowds to ‘a monstrous worm whose sensibility is diffuse and who still acts with disordered movements according to the dictates of its head’. Tarde’s contemporary, the social psychologist Gustave Le Bon, tried to explain crowd behaviour as a paralysis of the brain; hypnotised by the group, the individual becomes the slave of unconscious impulses. ‘He is no longer himself, but has become an automaton who has ceased to be guided by his will,’ he wrote in 1895. ‘Isolated, he may be a cultivated individual; in a crowd, he is a barbarian… a grain of sand amid other grains of sand, which the wind stirs up at will.’

This is still the prevailing view of mob behaviour, but it turns out to be wrong. In recent years, social psychologists have found that, rather than surrendering rationality and self-awareness, people in crowds define themselves according to who they are with at the time; their social identity determines how they behave.

Stephen Reicher, a social psychologist at the University of St Andrews, says this model of crowd behaviour fits every case of public disorder in the past three decades where data have been collected. It also squares with the most thorough investigation of urban unrest ever conducted: the Kerner Commission report into the urban race riots in Los Angeles, Chicago, Newark, and other US cities between 1965 and 1967. Perhaps the most serious of these took place in Detroit in July 1967 after police raided an unlicensed bar and hundreds of patrons and local residents gathered in the street to protest. After more than four days of violence between the crowd and law enforcement, there were 43 people dead, 1,189 people injured, and more than 2,000 buildings destroyed.

The Kerner report concluded that the key factor behind the unrest – the worst in US history since the Civil War – was widespread deprivation within black communities. It noted that black people were twice as likely as whites to be unemployed, and almost four times as likely to be living in poverty. The report also pointed out that the typical rioter was better educated than the average person in their communities, more socially integrated and less likely to have a previous criminal record. ‘What the rioters appeared to be seeking was fuller participation in the social order and the material benefits enjoyed by the majority of American citizens,’ it concluded. ‘Rather than rejecting the American system, they were anxious to obtain a place for themselves in it.’

This sits neatly with Reicher’s theory, which holds that people in crowds behave not with mindlessness or madness, but by co-operating with those around them. They do not lose their heads but instead act with full rational intent. In London, many (though not all) of the rioters were motivated by a shared sense of grievance at how their communities had been treated by police in the preceding months and years, in particular the frequency with which young black men were being stopped and searched by police in the street for no good reason. When the riot started, many individuals felt that, as members of a dissatisfied group, they had a legitimate cause to join in.

Surprisingly, the model also explains why crowds in emergency situations are disinclined to panic, putting them at higher risk. When the hijacked planes hit the World Trade Center towers in New York on 11 September 2001, most of those inside procrastinated rather than heading for the nearest exit. Even those who managed to escape waited an average of six minutes before moving to the stairs. Some hung around for half an hour, awaiting more information, collecting things to take with them, going to the bathroom, finishing emails, or making phone calls.

Likewise, say researchers, passengers have died in accidents because they just didn’t try to get out. Take the aircraft fire at Manchester airport in the UK on 22 August 1985, when 55 people died because they stayed in their seats amid the flames. John Leach, who studies disaster psychology at the University of Oslo, says a shared state of bewilderment might be to blame. Contrary to popular belief that crowds always panic in emergencies, large groups mill around longer than small groups since it takes them more time to come up with a plan.

The new understanding of crowd dynamics has helped authorities in Europe rewrite the rules on crowd management. In emergencies, individuals are encouraged to avoid the allure of complacency and think of an escape plan. Likewise, many police forces are trained to adopt a more sensitive, communicative approach to large crowds. If crowd behaviour stems from the social norms of the majority rather than the actions of the criminal few, then sending in the riot squad is likely to make things worse, because it treats everyone as aggressors. Presumed aggressors will probably respond in kind, and violence can spiral out.

These changes have been motivated largely by Clifford Stott, a researcher at the University of Leeds. Stott has done most of his work by embedding with crowds – especially with England football fans at away games – and recording what they do. He has been present, Dictaphone in pocket, at some of the most significant episodes of football violence involving England fans in Europe, including the World Cup finals in Italy in 1990 and France in 1998. He has little doubt that most football violence is best understood in terms of broad group identities rather than mindless minions following a hooligan minority. At the tournaments in Italy and France, for example, the vast majority of England fans were peaceful at the start, but together changed their tune when they began to feel that local police were targeting England supporters in particular. By the same token, Scottish football fans rarely got into trouble despite drinking heavily, since non-violence has become part of their identity. By avoiding conflict, Scotland fans help differentiate themselves from England fans, and even punish those within their ranks who get aggressive.

Stott and his collaborators presented their research to the Portuguese Public Security Police (PSP) before the European football championships, scheduled for Portugal for the first time in 2004. They advised the PSP to drop the riot-squad tactics used at most previous tournaments in favour of a lower-profile, firm-but-friendly approach. The Portuguese were receptive. They developed a training programme to ensure that all PSP officers understood the theory and how to translate it into non-confrontational policing. The result was an almost complete absence of disorder at England games during Euro 2004.

In the dark tunnels, where hundreds were trapped with no way of knowing if they would be rescued, there was little panic

Today, the social identity model of crowd behaviour is the framework by which all Union of European Football Associations (UEFA) matches in Europe are policed – though in Russia and in eastern Europe it is still only sporadically applied. Stott is now working to take the model beyond football. In 2009, Her Majesty’s Inspectorate of Constabulary (HMIC), the UK’s independent police inspectorate, asked him to write a report on applying crowd psychology to public policing. HMIC adopted many of his recommendations, and one outcome has been the creation of liaison units in London and elsewhere that send officers in distinctive uniforms into protest crowds to establish contact – not, as some activists suspect, to spy on them. Despite the credibility problems suffered by London police in the wake of the 2011 riots, liaison officers were used successfully around 50 times during the London 2012 Olympic Games.

Years of field research have taught Stott, Reicher and other social psychologists not only that mindless irrationality is rare within crowds, but also that co-operation and altruism are the norm when lives are at stake. When Islamic extremists detonated four bombs on London’s transport system during the morning rush hour on 7 July 2005, killing 52 people and injuring more than 770, there emerged many remarkable stories of helpful behaviour amid the carnage and chaos. In the dark, soot-filled underground tunnels, where hundreds were trapped with no way of knowing if they would be rescued, there was little panic and a general sense that ‘we’re all in the same boat together’, as one survivor put it.

At the University of Sussex, researchers led by the social psychologist John Drury have coined the term ‘collective resilience’, an attitude of mutual helping and unity in the midst of danger, to describe how crowds under duress often behave. There are many documented examples of this. In 2008, Drury’s team interviewed survivors of 11 tragedies from the previous 40 years, including the 1989 Hillsborough football stadium disaster when 96 Liverpool supporters died after being trapped in overcrowded pens, and the IRA bombing that killed six outside Harrods in London in 1983. In each case, most of Drury’s interviewees recalled feeling a strong sense of togetherness during the crisis, and an inclination to help strangers. Without such co-operation, the casualty rates could have been far higher, says Drury, who refers to crowds as ‘the fourth emergency service’ – an attitude not often shared by police. In Drury’s view, it is wrong-headed to blame crowd disasters on the behaviour of the crowd. More often the real problem is poor organisation – too many people in one place – or inadequate venue design.

Drury explains that a crisis, even a minor one such as a train breaking down in a tunnel, creates a ‘psychological crowd’ out of what was previously an aggregate of strangers. You suddenly share a common fate and your sphere of interest ramps up from the personal to the group.

‘I think everyone would accept that one had really gone beyond the definition of identifying the person as a supporter of football,’ said a survivor of the Hillsborough tragedy. ‘I don’t think anyone saw Liverpool fans and Notts Forest fans… People stopped being supporters of a football team and were just people.’

This sense of communion and barriers being broken down is one of the most widely reported crowd experiences. Psychological crowds are often fulfilling and uplifting places to be. Interviewing people after the Egyptian revolution in 2011, I often heard participants say that the days they spent protesting in Cairo’s Tahrir Square were the best of times. ‘The revolution was the greatest event I will live through in my life,’ the science journalist Mohammed Yahia told me. ‘It was a piece of paradise,’ said Basem Fathy, one of the youth organisers who helped coordinate the early protests. ‘No one thought whether you were Christian or Muslim, poor or rich.’

The Cairo-based journalist Ursula Lindsey remembers it like this: ‘Once you saw that crowd, you just felt the tide was on your side. You felt so right. It made people feel fantastic. People were high on it, seeing they were part of such a big group. It was an endless, mind-opening, affirming experience. You saw the evaporation of fear, this elation, because so many people were with you. Those 18 days brought out the best in people. People gave the best versions of themselves.’

The idea that in crowds we might give the best versions of ourselves runs counter to the common view that has prevailed since the French Revolution. Yet it has science on its side: from the cohesion of football crowds to the altruism of disaster victims and the solidarity of revolutionaries, the evidence for the sanity and intimacy of crowds has become overwhelming.