July 1953: late one evening in the woods outside Middle Grove, New York state, three men are having a furious argument. One of them, drunk, draws back his fist, ready to smash it into his opponent’s face. Seeing what is about to happen, the third grabs a block of wood from a nearby pile. “Dr Sherif! If you do it, I’m gonna hit you,” he shouts.

The man with the raised fist isn’t just anybody. He is one of the world’s foremost social psychologists, Muzafer Sherif. The two others are his research assistants. Sherif is angry because the experiment he has spent months preparing for has just fallen apart.

Born in the summer of 1905 and raised in İzmir province, Turkey, during the dying days of the Ottoman empire, Sherif won a place at Harvard to study psychology. But he found himself frustrated by the narrowness of the discipline, which mainly involved tedious observation of lab rats. He was drawn instead to the emerging field of social psychology, which looks at the way human behaviour is influenced by others. In particular, he became obsessed by group dynamics: how individuals band together to form cohesive units and how these units can find themselves at each other’s throats.

In the aftermath of the second world war, he wasn’t the only one interested in this idea. Early in 1953, the Rockefeller Foundation gave Sherif $38,000 – $350,000 (£245,000) in today’s money – to carry out what he hoped would be a career-defining piece of research. This time there would be no rats: the subjects were 11-year-olds, and neither they nor their parents had any idea what they were signing up for.

This is the scene Gina Perry sets at the beginning of The Lost Boys, her new book about Sherif. It’s her second foray into social psychology: Behind the Shock Machine (2013) looked at the notorious Milgram experiments of the early 60s, which studied the extent to which people are prepared to follow orders. Stanley Milgram made subjects think they were delivering potentially fatal electric shocks to participants in another room, and provided a scream soundtrack to make it sound convincing. Two-thirds went along with it, despite the terrifying noises.

He believed he could make the two groups sworn enemies via a series of well-timed 'frustration exercises'

Sherif’s cover story was that he was running a summer camp in Middle Grove. His plan was to bring a group of boys together, allow them to make friends, then separate them into two factions to compete for a prize. At this point, he believed, they would forget their friendships and start demonising one another. The pièce de résistance was to come at the end: Sherif planned to set a forest fire in the vicinity of the camp. Facing a shared threat, they would be forced to work as one team again.

This was a year before the publication of Lord of the Flies. But whereas William Golding sought to show that boys were, by their nature, little devils, Sherif believed that context was everything. Competition over scarce resources could drive people to enmity; place a common obstacle in their way, and they cooperate. During his youth, Perry writes, Sherif witnessed interethnic violence between Turks, Greeks and Armenians that claimed tens of thousands of lives. She believes his desire to understand the causes of that catastrophe – and to show the world a way to prevent another like it – was what drove him.

His work seems particularly relevant in an age of resurgent tribalism. During the 2016 US election, psychologists Yarrow Dunham and David Rand invoked Sherif in a discussion of the apparently unbridgeable divide between Hillary Clinton and Bernie Sanders supporters. They cited his recognition of the “human tendency to forge alliances as the context demands”, writing that “for Democrats, the context is now the threat of a President Trump”. This, they believed, was grounds for optimism. Unfortunately, even a united Democratic party wasn’t enough to see off a tribe that recruited from Michigan, Ohio and Pennsylvania, and stormed the White House.

Facebook Twitter Pinterest Sherif at the Robbers Cave camp. Photograph: The University of Akron

In 50s Middle Grove, things didn’t go according to plan either, though the surprise was of a different nature. Despite his pretence of leaving the 11-year-olds to their own devices, Sherif and his research staff, posing as camp counsellors and caretakers, interfered to engineer the result they wanted. He believed he could make the two groups, called the Pythons and the Panthers, sworn enemies via a series of well-timed “frustration exercises”. These included his assistants stealing items of clothing from the boys’ tents and cutting the rope that held up the Panthers’ homemade flag, in the hope they would blame the Pythons. One of the researchers crushed the Panthers’ tent, flung their suitcases into the bushes and broke a boy’s beloved ukulele. To Sherif’s dismay, however, the children just couldn’t be persuaded to hate each other.

After losing a tug-of-war, the Pythons declared that the Panthers were in fact the better team and deserved to win. The boys concluded that the missing clothes were the result of a mix-up at the laundry. And, after each of the Pythons swore on a Bible that they didn’t cut down the Panthers’ flag, any conflict “fizzled”. By the time of the incident with the suitcases and the ukulele, the boys had worked out that they were being manipulated. Instead of turning on each other, they helped put the tent back up and eyed their “camp counsellors” with suspicion. “Maybe you just wanted to see what our reactions would be,” one of them said.

The robustness of the boy’s “civilised” values came as a blow to Sherif, making him angry enough to want to punch one of his young academic helpers. It turned out that the strong bonds forged at the beginning of the camp weren’t easily broken. Thankfully, he never did start the forest fire – he aborted the experiment when he realised it wasn’t going to support his hypothesis.

But the Rockefeller Foundation had given Sherif $38,000. In his mind, perhaps, if he came back empty-handed, he would face not just their anger but the ruin of his reputation. So, within a year, he had recruited boys for a second camp, this time in Robbers Cave state park in Oklahoma. He was determined not to repeat the mistakes of Middle Grove. There was no mixing at the beginning – neither of the two groups, the Rattlers and the Eagles, were aware of the other’s existence until the second day. But, perhaps more importantly, Sherif relinquished his role as puppet master; a condition laid down by his research associate, OJ Harvey, who knew how volatile Sherif could be and insisted on taking control himself.

At Robbers Cave, things went more to plan. After a tug-of-war in which they were defeated, the Eagles burned the Rattler’s flag. Then all hell broke loose, with raids on cabins, vandalism and food fights. Each moment of confrontation, however, was subtly manipulated by the research team. They egged the boys on, providing them with the means to provoke one another – who else, asks Perry in her book, could have supplied the matches for the flag-burning?

Having got them fighting, the next stage was the all-important reconciliation – and the vindication of Sherif’s theory. One morning, the boys found that their water supply had been cut off. They would have to locate the water tank high on the mountain and work together to remove the rocks Harvey and Sherif had placed over the valve, so they could open it again. “Slowly,” Perry writes, “with the sun beating down and their water canteens emptying, the boundaries between the groups began to blur.” First, she says, they “took turns lifting and carrying the rocks away. But, realising there was a better and faster way of getting the job done, they soon formed a chain, passing the rocks down the line and working as a single team.”

Sherif was elated. And, with the publication of his findings that same year, his status as world-class scholar was confirmed. The “Robbers Cave experiment” is considered seminal by social psychologists, still one of the best-known examples of “realistic conflict theory”. It is often cited in modern research. But was it scientifically rigorous? And why were the results of the Middle Grove experiment – where the researchers couldn’t get the boys to fight – suppressed? “Sherif was clearly driven by a kind of a passion,” Perry says. “That shaped his view and it also shaped the methods he used. He really did come from that tradition in the 30s of using experiments as demonstrations – as a confirmation, not to try to find something new.” In other words, think of the theory first and then find a way to get the results that match it. If the results say something else? Bury them.

All the boys I spoke to had an uneasy feeling about this experience. It has troubled people Author Gina Perry

Perry has called her book The Lost Boys because she traced several of Sherif’s subjects, now men in their 70s, to ask them how they felt about having been guinea pigs. Astonishingly, none of them were aware that the rather unusual camps they had attended were fake.

“I’m not traumatised by the experiment, but I don’t like lakes, camps, cabins or tents,” Doug Griset tells Perry, wryly undermining his own point. He was sent home from Middle Grove after succumbing to a bout of homesickness that Sherif worried would become contagious. Griset marvels at a letter sent to his parents more than 50 years ago to recruit him for the experiment. Those letters, writes Perry, “are a lesson in the art of skilful deception and subtle persuasion”. In them, Sherif explains that “camp directors are interested in finding out what things can be done to give the boys … a wholesome cooperative living experience which will prepare the youngsters for better citizenship and to be leaders in their communities”. How could the parents have known his real aim was to make their sons hate each other – then leave them to deal with a forest fire?

Subterfuge of this kind was to become unacceptable to academic psychologists after Milgram’s experiments, which provoked an outcry. Sherif’s ethical blunders seem milder, but they still had lasting effects. Having expected to act as inquisitor, Perry found herself treated more like an interviewee. “Really, I felt I had to answer their questions: ‘Who was this guy? Why was I involved? How did my parents agree to me going away? What did I do during the experiment?’ All the boys I spoke to had an uneasy feeling about this experience. It has troubled people,” she says.

Does Sherif really deserve his place in social psychology’s hall of fame, given what Perry has uncovered about his methods? Robbers Cave is, after all, a staple of textbooks.

‘Come on out and fight!’: an extract from The Lost Boys by Gina Perry Read more

“I think people are aware now that there are real ethical problems with Sherif’s research,” she tells me, “but probably much less aware of the backstage [manipulation] that I’ve found. And that’s understandable because the way a scientist writes about their research is accepted at face value.” The published report of Robbers Cave uses studiedly neutral language. “It’s not until you are able to compare the published version with the archival material that you can see how that story is shaped and edited and made more respectable in the process.” That polishing up still happens today, she explains. “I wouldn’t describe him as a charlatan … every journal article, every textbook is written to convince, persuade and to provide evidence for a point of view. So I don’t think Sherif is unusual in that way.”

If Middle Grove and Robbers Cave aren’t scientifically rigorous, does that mean they’re of no value? Perry doesn’t think so. “There was a kind of breadth of vision about Robbers Cave that is very much missing in that tightly controlled laboratory deception of something like Milgram. He was trying to tackle big issues.”

And, from today’s perspective, perhaps there is some reassurance to be gleaned from boys’ behaviour at Middle Grove. Despite attempts to influence them that a Russian troll farm would be proud of, they remained independent-minded and did what they thought was best.

“I do think it is a kind of optimistic view,” says Perry. “It makes you smile, doesn’t it? The fact that they mutinied against these guys, really, and refused to be drawn into it.”

The Lost Boys: Inside Muzafer Sherif’s Robbers Cave Experiment by Gina Perry is published by Scribe in Australia on 16 April and in the UK on 26 April.