In 2013, a British man was arrested for planning to kidnap and brutally murder a soldier. The suspect, who had a criminal history, had posted messages on social media in support of violent jihad. In a search of his residence, the police had found a bag containing a hammer, a kitchen knife and a map with the location of a nearby army barracks.

Shortly after his arrest, the suspect was interviewed by a counter-terrorist police officer. The interviewer wanted him to provide an account of his plan, and to reveal with whom, if anyone, he has been conspiring. But the detainee – we will call him Diola – refused to divulge any information. Instead, he expounded grandiloquently on the evils of the British state for 42 minutes, with little interruption. When the interviewer attempted questions, Diola responded with scornful, finger-jabbing accusations of ignorance, naivety and moral weakness: “You don’t know how corrupt your own government is – and if you don’t care, then a curse upon you.”

Watching a video of this encounter, it is just possible to discern Diola’s desire, beneath his ranting, to tell what he knows. In front of him, a copy of the Qur’an lies open. He says he was acting for the good of the British people, and that he is willing to talk to the police because, as a man of God, he wants to prevent future atrocities. But he will not answer questions until he is sure that his questioner cares about Britain as much as he does: “The purpose of the interview is not to go through your little checklist so you can get a pat on the head. If I find you are a jobsworth, we are done talking, so be sincere.”

Even distanced by years from the events in question, it is impossible to watch the encounter without feeling tense. Periodically, Diola turns away from the interviewer and goes silent, or gets up and leaves the room, having taken offence at something said or not said. Each time he returns, Diola’s solicitor advises him not to speak. Diola ignores him, though in a sense he takes the advice: despite the verbiage, he tells his interviewer nothing.

Diola: “Tell me why I should tell you. What is the reason behind you asking me this question?”

Interviewer: “I am asking you these questions because I need to investigate what has happened and know what your role was in these events.”

Diola: “No, that’s your job – not your reason. I’m asking you why it matters to you.”

The interviewer, who has remained heroically calm in the face of Diola’s verbal barrage, is not able to move the encounter out of stalemate, and eventually his bosses replace him. When the new interviewer takes a seat, Diola repeats his promise to talk “openly and honestly” to the right person, and resumes his inquisitorial stance. “Why are you asking me these questions?” he says. “Think carefully about your reasons.”

The new interviewer does not answer directly, but something about his opening speech triggers a change in Diola’s demeanour. “On the day we arrested you,” he began, “I believe that you had the intention of killing a British soldier or police officer. I don’t know the details of what happened, why you may have felt it needed to happen, or what you wanted to achieve by doing this. Only you know these things Diola. If you are willing, you’ll tell me, and if you’re not, you won’t. I can’t force you to tell me – I don’t want to force you. I’d like you to help me understand. Would you tell me about what happened?” The interviewer opens up his notebook, and shows Diola the empty pages. “You see? I don’t even have a list of questions.”

“That is beautiful,” Diola says. “Because you have treated me with consideration and respect, yes I will tell you now. But only to help you understand what is really happening in this country.”

For years, any debate over what constitutes effective interrogation has been dominated by a pervasive folk belief in coercion. From NYPD Blue to 24 and Zero Dark Thirty, we are trained in the idea that interrogators get the job done by intimidating, demoralising and, when necessary, brutalising their subjects. Steven Kleinman, a former army colonel and one of the US military’s most prolific and experienced interrogators, told me it is not just the public that is influenced by popular narratives: “Politicians, policy-makers, senior military officers – people who have never conducted interrogations are somehow just convinced they know what works.”

In 2003, Kleinman tried to stop his fellow soldiers from conducting abusive interrogations of Iraqi insurgents; he later became the first military officer to speak out against such practices. He did so not just because he thought they were wrong, but because he thought they were stupid. Kleinman believes that coercion is counterproductive, because it destroys the trust that underpins a successful interview. Most specialist practitioners agree, as do the scientists who study interrogation. But conventional wisdom in military and law enforcement circles has been very hard to shift.

This is because it is difficult to prove what works. High-stakes interrogations take place in secret, and have rarely been available to objective researchers. In place of cool analysis, colourful but unreliable stories of vital secrets wrenched from fearful suspects have prevailed. In reality, well-run interrogations are rarely dramatic: drama thrives on conflict – something professional interrogators strive to avoid.

A body of scientific literature supports Kleinman’s view, but most of it is based on laboratory experiments, in which students are asked to pretend they have just robbed a bank and interrogators are asked to believe them. The virtue of these experiments is that they allow for controlled trials of specific interrogation techniques; the drawback is that they are easily dismissed by practitioners as academic game-playing.

Psychologists Emily and Laurence Alison from the University of Liverpool. Photograph: Christopher Thomond/The Guardian

Now, two British researchers are quietly revolutionising the study and practice of interrogation. Earlier this year, in a meeting room at the University of Liverpool, I watched a video of the Diola interview alongside Laurence Alison, the university’s chair of forensic psychology, and Emily Alison, a professional counsellor. My permission to view the tape was negotiated with the counter-terrorist police, who are understandably wary of allowing outsiders access to such material. Details of the interview have been changed to protect the identity of the officers involved, though the quotes are verbatim.

The Alisons, husband and wife, have done something no scholars of interrogation have been able to do before. Working in close cooperation with the police, who allowed them access to more than 1,000 hours of tapes, they have observed and analysed hundreds of real-world interviews with terrorists suspected of serious crimes. No researcher in the world has ever laid hands on such a haul of data before. Based on this research, they have constructed the world’s first empirically grounded and comprehensive model of interrogation tactics.

The Alisons’ findings are changing the way law enforcement and security agencies approach the delicate and vital task of gathering human intelligence. “I get very little, if any, pushback from practitioners when I present the Alisons’ work,” said Kleinman, who now teaches interrogation tactics to military and police officers. “Even those who don’t have a clue about the scientific method, it just resonates with them.” The Alisons have done more than strengthen the hand of advocates of non-coercive interviewing: they have provided an unprecedentedly authoritative account of what works and what does not, rooted in a profound understanding of human relations. That they have been able to do so is testament to a joint preoccupation with police interviews that stretches back more than 20 years.

Pausing the Diola video, Emily Alison grimaced. “I call this one ‘the Hannibal Lecter interview’,” she said. “He wants a piece of the interviewer. When I watched this tape the first time I had to switch it off and walk away. I was so outraged, my heart was pounding in my chest. Of course, if you’re in the room, it’s 1,000 times worse.” Laurence Alison nodded. “As the interviewer, you’re bound to have an emotional response,” he said. “What you want to say is, ‘You’re the one in the fucking seat, not me. He’s trying to control you, so you try and control him. But then it escalates.”

The moment that an interrogation turns into an argument, it fails. “You need to remember what your purpose in that room is,” said Emily. “You’re seeking information. You’re not there to speak on behalf of the victims or the police. You might feel better for getting angry, but down that road is retribution. You become the inquisitioner. That’s not why you’re there. If you find yourself having a go at someone, ask yourself: ‘What am I achieving by this?’ Because they will stop talking to you.”

With us on the day we watched the video was an officer in Britain’s counter-terrorist police force with whom the Alisons have been working closely to train an elite cadre of interviewers. “A big thing we talk about is leaving your ego at the door,” he said. “But that’s tough, because cops are used to being in control.”

Emily met Laurence at the University of Liverpool in 1996, shortly after arriving in the UK from her home in Wisconsin. She had applied to join the Madison police force, which she saw as a stepping stone to the FBI, but opted at the last minute to take a masters in “investigative psychology” – the application of psychology to police work (Liverpool was then one of the few institutions in the world to offer it). “This wasn’t long after Silence of the Lambs,” said Emily. “I wanted to be the new Clarice Starling.”

Laurence was a PhD student in the department of forensic psychology, and already a rising star after his contribution to a high-profile public inquiry. In 1993, Colin Stagg was wrongfully accused of the rape and murder of a young mother called Rachel Nickell, on Wimbledon Common. Despite an absence of forensic evidence linking Stagg to the murder, the police made Stagg their prime suspect after deciding that he matched an “offender profile” created by a psychologist.

A covert operation was designed to entrap Stagg, involving an undercover female police officer feigning romantic interest. After the case was thrown out and the police had acknowledged their mistake, Laurence assisted the subsequent inquiry by exposing the pseudoscience on which the profile was based. (In 2008, a convicted killer called Robert Napper was found guilty of the crime.) As part of his research he watched a video of Stagg’s interview, conducted by the police at the time of Stagg’s first arrest, and he was appalled by what he saw. The interviewers, who had clearly made up their mind that Stagg was guilty, were aggressive, insulting, and asked questions that were not designed to find out what the suspect knew but to trip him up or intimidate him into confessing. After failing to do either, they released Stagg and decided to entrap him.

In 1998, Emily and Laurence got married. Laurence continued his academic career. Emily, who had worked as a counsellor in Wisconsin prisons, joined the Cheshire probation service, and later started a consultancy, helping social workers counsel families afflicted by domestic abuse. Alongside their day jobs, the Alisons started collaborating on criminal cases.

Historically, the British police have called on outside experts to help with investigations, but have sometimes ended up listening to quacks. After the Stagg inquiry, however, a list of accredited consultants was drawn up, and Laurence Alison was on it. Every couple of months or so he would get a call, and a question. “It might be, can you help us with this rape in Bath, or a murder in West Mercia,” Laurence said. The police often wanted to know the best way to interview a particular suspect or witness, usually after an initial attempt had gone badly.

Laurence would ask for assistance from Emily, who knew a lot about interviewing difficult people, thanks to her background in counselling. The Alisons would read transcripts, watch video footage, and sometimes monitor interviews in real time, assessing the dynamics of the encounter, searching for a way to get the interviewee to open up. They gained a reputation for offering useful advice. Their first involvement with the counter-terrorist force came in 2004, when they helped police in Northern Ireland investigate the Northern Bank robbery, in which armed raiders, thought to be connected to the IRA, stole £26.5m from bank vaults in Belfast.

In 2010, Laurence was contacted by a US government agency that was commissioning research into interrogation. The High-Value Detainee Interrogation Group, or HIG, was set up in 2009 by President Obama, keen to signal a clean break from the Bush administration, which had sanctioned abusive interrogations in Iraq and Afghanistan. Housed within the FBI, the HIG’s purpose is to ground interrogation in science.

“The last time the US government invested in studying interrogation was 1956,” Steven Kleinman, who works with the HIG, told me. “Great leaps forward in behaviour science have just passed us by,” he said. “Imagine using 1956 technology for signals intelligence!” One consequence has been the triumph of bad science. In Afghanistan, after the 9/11 attacks, military psychologists Dr James Mitchell and Dr John Jessen devised and ran a programme of brutal interrogation techniques for the CIA, based on their own poorly evidenced theories of what works.

The HIG’s chief researchers were particularly interested in Britain, whose counter-terrorist police have earned a reputation for being sophisticated interviewers, partly because the opposite was once true. In 1992, after public enquiries into two miscarriages of justice involving IRA attacks had revealed abusive interrogation practices, parliament passed laws stipulating all interviews be recorded, and making it an automatic right to have a solicitor present. With the option of coercion effectively removed, the British police were forced to think harder about the best way to obtain information from prisoners resistant to giving it. In a minor but significant change they stopped using the word “interrogation”, with its confrontational overtone, and replaced it with “interview”.

Omar Khadr, a Canadian teenager interrogated at Guantanamo. Photograph: PA

The HIG invited Laurence to apply to them for research funding. “I said, I don’t want to do research on students. I want to look at the real thing and extract, from observation, what works.” He set his sights on an audacious goal: persuading the national counter-terrorism unit to give him access to video of its interviews with terrorist suspects. Two years and over 100 phone calls later, in 2012, he secured an appointment with the officer in charge of interviewing. Laurence told him that by studying his team’s methods, he could establish definitively which tactics were effective and which were not. He also offered to use the resulting evidence base to design training that would hone the skills of even specialist interviewers. The officer listened, and following a negotiation over terms, granted access to 181 interviews, a total of 878 hours of tape (further interviews were shared with the Alisons later, including the one I was allowed to view). They included Irish paramilitaries, al-Qaida operatives, far-right extremists, incompetent bunglers caught up in something they didn’t understand, and highly dangerous operatives.

The tapes were housed in a secure police facility in Yorkshire, and the Alisons were not permitted to move them. They took turns to visit, often accompanied by a research assistant, Stamatis Elntib. The police officer assigned to monitor them that day would meet them at the door, then sit with them in the small viewing room as they watched hours of video. After a full day of this, whichever Alison was on duty would check into the nearby Premier Inn, sleep, and return to the facility in the morning for another day’s work.

Each interview had to be minutely analysed according to an intricate taxonomy of interrogation behaviours, developed by the Alisons. Every aspect of the interaction between interviewee and interviewer (or interviewers – sometimes there are two) was classified and scored. They included the counter-interrogation tactics employed by the suspects (complete silence? humming?), the manner in which the interviewer asked questions (confrontational? authoritative? passive?), the demeanour of the interviewee (dominating? disengaged?), and the amount and quality of information yielded. Data was gathered on 150 different variables in all.

Watching and coding all the interviews took eight months. When the process was complete, Laurence passed on the data to Paul Christiansen, a colleague at Liverpool University, who performed a statistical analysis of the results. The most important relationship he measured was between “yield” – information elicited from the suspect – and “rapport” – the quality of the relationship between interviewer and interviewee. For the first time, a secure, empirical basis was established for what had, until then, been something between a hypothesis and an insider secret: rapport is the closest thing interrogators have to a truth serum.

In 1943, Major Sherwood Moran, of the US Marines, published a memorandum on the interrogation of enemy prisoners of war, and distributed it to troops throughout the Pacific theatre. Moran was a missionary who had raised a family in Tokyo before the war. When the Japanese attacked Pearl Harbor in 1941, he was 56, and living in Boston. Realising that his fluency in Japanese language and culture might be helpful to the American war effort, he joined the Marines. Moran soon became known as an unusually effective interrogator of Japanese soldiers, who were famously resistant interviewees. Like Islamist terrorists today, the Japanese were fanatically, suicidally committed to their cause, and deeply hostile to Americans.

In his memo, Moran explained why he eschewed the bullying methods used by other interrogators. He believed that if the prisoner was forcibly reminded he was facing his conqueror, he would be placed “in a psychological position of being on the defensive”. Moran’s premise was that even the most implacable prisoner had a story that he wanted to tell; the interrogator’s job was to create an atmosphere in which he felt willing and able to tell it. The surest way to do that is to treat him as a fellow human being, rather than an enemy: “Make him and his troubles the centre of the stage, not you and your questions of war problems. If he is not wounded or tired out, you can ask him if he has been getting enough to eat. … If he is wounded you have a rare chance. Begin to talk about his wounds. Ask if the doctor has attended to him. Have him show you his wounds or burns.”

Moran did not believe in making the prisoner feel scared or powerless. Stripping a prisoner of his dignity merely reinforced his determination not to speak. The aim should instead be to get into his mind and heart – to achieve “intellectual and spiritual rapport”.

Today, most experienced interrogators agree. In 2012 the psychologist Melissa Russano asked 42 US interrogators what they believed made for successful interviews. The participants, who responded anonymously, had performed interrogations in Afghanistan and Iraq, with some of the most hardened and resistant interviewees in the world. These are some of their responses:

“Being nice is far and away going to be more fruitful than being an asshole. I could scare the crap out of you in the next 10 seconds, if I really wanted to. But you know, what is that going to do?”

“You have to present a very empathetic, supportive environment.”

“Rapport is what we do. I was trying to think of why I’ve had success, or we as a team have had success with these guys, and I think, in some respect, that they are interested to tell their story – they’re interested to tell the ‘why’. So I think talking to them in a way that is non-judgmental [gets results].”

For most of its history, interrogation has been entwined with accusation and retribution. In 16th-century Europe it was used to justify the trial and execution of women accused of witchcraft. Suspects would not avoid torture until they stopped lying and confessed the truth – that is, confessed to whatever wildly implausible crime with which their interrogators arraigned them. The accusations were contrived to resolve communal grievances or satisfy public hunger for justice, however rough. Interrogation was revenge, disguised as truth-seeking.

Today, interrogations still get framed as accusatory moral dramas, and not just on TV. American police officers are trained in the Reid Technique, developed in the 1950s by John Reid, a former Chicago cop. In a pre-interrogation interview, the detective assesses a suspect’s credibility by observing his body language, such as fidgeting or eye movements. (There is no evidence these are reliable cues to deceit.) Once he decides that a suspect is lying, the interrogator moves into confrontational mode, in an effort to break the suspect’s resistance. The technique has been consistently associated with false confessions. It survives, at least in part, because it makes the interrogator feel in control, positioning him as a heroic protagonist.

Accusatory models of interrogation lay great emphasis on body language, partly because the hostile interviewer has already cut off his richest source of information: words. Aggression kills dialogue, as other participants in the Russano study attest:

“Screaming and yelling … puts up a wall.”

“Playing the bad cop … doesn’t work. Being nasty, being confrontational, pounding the desk, throwing things. Calling the person names. Just anything that you would expect that would just turn somebody off from wanting to talk to you.”

Despite its reputation among elite practitioners, “rapport” has been vaguely defined and poorly understood. It is often conflated with simply being nice – Laurence Alison refers to this, derisively, as the “cappuccinos and hugs” theory. In fact, he observes, interviewers can fail because they are too nice, acquiescing too quickly to the demands of a suspect, or neglecting to pursue a line of purposeful questioning at a vital moment.

The best interviewers are versatile: they know when to be sympathetic, when to be direct and forthright. What they rarely do is impose their will on the interviewee, either overtly, through aggression, or covertly, through the use of “tricks” – techniques of unconscious manipulation, which make the interviewer feel smart but are often seen through by interviewees. Above all, rapport, in the sense used by the Alisons, describes an authentic human connection. “You’ve got to mean it,” is one of Laurence’s refrains.

The Alisons named their research project Orbit (Observing Rapport-based Interpersonal Techniques). Part of its purpose is to provide an anatomy of rapport, the better to understand what creates and destroys it. At the heart of the Alisons’ model is an insight from a neighbouring field. During the years when she worked on police cases with Laurence, Emily Alison had come to see interrogation as a close relation of addiction counselling. Both involve getting someone who does not want to be in the same room as you to talk about something they do not want to talk about.

Around two decades ago, the practice of addiction counselling was transformed by the application of a simple principle: patients should feel responsible for their choices. Emily wondered if it wasn’t time for interrogation to catch up.

In 1980, a 23-year-old South African called Stephen Rollnick started work as a nurse’s aid in a rehabilitation centre for alcoholics. The clinicians shared a confrontational approach to the job. They believed their clients were lying to themselves, and others, about the severity of their problem. Before setting the patient on the road to recovery, the clinician needed to challenge the patient on their dishonesty and strip away their illusions – to break their resistance.

This clinic was hardly atypical. The post-war medical consensus on addiction treatment regarded patients as wayward children who needed to be taught how to behave. The counsellor’s job was to tell the addict the truth about their condition, and, if they denied it, to do so again more forcefully until they accepted it. To Rollnick, this seemed bound to poison the relationship. In the coffee room, he observed that the off-duty conversations of the counsellors were imbued with disdain for their patients.

One of the clients under Rollnick’s care was an alcoholic called Anthony, who would leave group sessions having barely said a word. One day, he walked out for the last time. Rollnick discovered the next morning that Anthony had shot his wife and then himself in front of their young children. Shattered, Rollnick resigned from the centre, left South Africa, and settled in the UK, where he embarked on a course in clinical psychology at Cardiff University.

‘You need to remember what your purpose in that room is – you’re seeking information, not retribution.’ Photograph: Joshua Bright/The Guardian

A couple of years in, Rollnick came across a new paper written by a young American psychologist called William Miller, and was startled by how much he agreed with it. Miller argued that counsellors were having precisely the wrong kind of conversation with their clients. Addicts were caught between a desire to change and a desire to maintain their habit. As soon as they felt themselves being judged or instructed, they produced all the reasons they did not want to change. That isn’t a pathology, Miller argued, it’s human nature: the more we feel someone trying to persuade us to do something, the more we dwell on the reasons we should not. By insisting on change, the counsellor was making himself feel better, while reinforcing the addict’s determination to carry on.

Miller argued that rather than instigating confrontation, counsellors should focus on building a relationship of trust and mutual understanding, enabling the patient to talk through his experiences without feeling the need to defend himself. Eventually, and with the counsellor gently shaping the dialogue, the part of the patient that wanted to get better would overcome the part that did not, and he would make the arguments for change himself. Having done so, he would be motivated to follow through on them. Miller called this approach “motivational interviewing” (MI).

Rollnick used Miller’s method in his clinical practice, with encouraging results. On meeting Miller at a conference, he told him about his enthusiasm for MI. The two men wrote a book together, developing the ideas in Miller’s paper. Motivational Interviewing revolutionised the field of counselling and therapy, and its techniques became widely practised. Empirical studies found MI to be a far more effective treatment than traditional methods.

Implicit in Miller and Rollnick’s critique of traditional counselling was the uncomfortable suggestion that counsellors should turn their professional gaze upon themselves and question their own instinct to dominate. Instead of thinking of himself as an expert sitting in judgment, the counsellor needed to adopt the more humble position of co-investigator. As Miller put it to me, “The premise is not ‘I have you what you need, let me give it to you.’ It’s ‘You have what you need and we’ll find it.’ The patient must feel “autonomous” – the author of their own actions.

Emily Alison, who had trained in MI while working as a counsellor for the probation service in Wisconsin, noticed that interrogations failed or succeeded for similar reasons as therapeutic sessions. Interrogators who made an adversary out of their subject left the room empty-handed; those who made them a partner yielded information. The best ones suspended moral judgment and conveyed genuine curiosity. She concluded that the detainee, like the addict, wants to feel free, despite or rather because of their confinement, and that the interviewer should help them do so.

The Alisons’ analysis of the terrorist tapes confirmed this. One of their most striking findings is that suspects are likelier to talk when the interviewer emphasises their right not to. “The more pressure you put on a person, the less likely they are to speak to you. You need to make them feel responsible for their choices,” said Laurence. “You can’t bullshit, you’ve got to mean it.” He slips into character. “Ian, you don’t have to speak to me today. Whether you do or not isn’t up to me. It isn’t up to your solicitor. It’s up to you.

“These are powerful tools to get inside someone’s head,” said Laurence. “But they’re not tricks. You have to be genuinely curious. There’s a reason this person has ended up opposite you, and it’s not just because they’re evil. If you’re not interested in what that is, you’re not going to be a good interrogator.”

Within a two-week stretch early in the summer of this year, Manchester and London were struck by terrorist attacks. In the days that followed there was an urgent push to uncover any networks in which the perpetrators – now dead – were enmeshed. Anyone who had been in contact with them needed to be wrung for information. The burden of that task fell to a select group of specialist interviewers, drawn from counter-terrorist units across the country. Something they had in common is that they were all alumni of the Alcyone course, an intensive six-day workshop designed by the Alisons in partnership with the police. More than 150 officers have now taken the course, which a counter-terrorist officer told me was by far the best interrogation training the police have ever had.

At the heart of the Alcyone course is a series of role-play exercises. Actors from a company called Characters, which specialises in professional role-playing, take the parts of interviewees. The roles might include a gang boss with ties to terrorists, or a young woman who has been sharing extremist propaganda on social media. The attendees are sent into the room with instructions to elicit specific information. They are told something about the “suspect”, but they don’t know what kind of character they are about to interview. It might be someone sullen and monosyllabic, fiercely belligerent, or charming but evasive.

The simulations are intended to be as testing as possible. “We wanted to create something harder than the real world,” says Laurence. The actors, who have done the course many times, are now highly experienced interviewees, and expert at needling, provoking or eluding their inquisitors. As soon as an encounter begins, its fictional nature is almost completely forgotten; even veteran interviewers can be pushed to the edge of self-control. Course facilitators, observing from outside the room, intervene when necessary. “Sometimes we design a character that will press the buttons of that particular person,” says Laurence. “We want to see if they can emotionally self-regulate under pressure.” Simply put, can they stop themselves losing it?

Studies of interrogation are often preoccupied with the question of how to detect deception, but even a lie is information; the hard problem for an interrogator is a suspect who says nothing. As one counter-terrorist officer who worked with the Alisons told Laurence, “I don’t care if they’re lying. I just want them to talk.” Terrorists can say nothing in a variety of ways. Irish paramilitaries were trained to focus their gaze on a spot on the wall and remain utterly silent. Some suspects give only monosyllabic answers, or stick to scripted responses, or simply turn their chair around, presenting the interviewer with the back of their head (Laurence demonstrated this to me – even in the context of make-believe, it was deeply unsettling). Islamic extremists, like Diola, are prone to long ideological rants. All such tactics have the potential to be doubly effective, because they get under the skin of the interviewer, throwing him off his plan by goading him into anger.

On the opening day of the Alcyone course, Laurence introduces the research and gives a crash course in “interpersonal psychology”, which is concerned with communication and how it breaks down. An interviewee – who might be hostile, cooperative, terrified, or some combination of the above – exerts an emotional force on the interviewer that is hard to resist. Skilled interrogators are adept at managing their own automatic responses, like sailors able to ride the sudden swells of a choppy sea.

The premise of interpersonal psychology is that in any conversation, the participants are asking for status – to feel respected and listened to – and communion – to feel liked and understood. “Power, love,” says Laurence. “The fundamental elements of all human behaviour.” Conversations only go well when both parties feel they are getting their fair share of each.

A father who opens the door to his daughter when she comes home late might adopt a confrontational style, implicitly inviting a contrite response. But his daughter, feeling her agency being denied, pushes back, which provokes her father’s anger. A power struggle ensues, until the conversation terminates with one or both stomping off to their bedroom. If the father had emphasised his love for his daughter, a conversation about acceptable norms might have developed. But doing so isn’t easy, partly because children know exactly which buttons to press. “I tell (the police), if you can deal with teenagers you can deal with terrorists,” says Laurence.

As we were talking, Laurence stepped into role, pointing at my notepad. “Can I have that?” Instinctively, I shook my head. “Why not? Give me a couple of sheets of paper.” I declined. Laurence raised his voice: “Oh come on, are you fucking joking?” I felt trapped, unable to think of a reason not to give him paper but unwilling to back down. Giving it to him felt like a small humiliation.

“It’s a classic test of autonomy,” explained Lawrence. “People don’t like being controlled, and sometimes they put something in the room, like, ‘I want a notebook’, to disrupt the balance of power. If you don’t deal with it, it becomes a roadblock. So either give him the paper or explain why you won’t.”

An interview fails when it becomes a struggle for dominance, in which the interviewee’s way of asserting himself is to tell his interviewer nothing. “In a tug of war, the harder you pull, the harder they pull,” says Laurence. “My suggestion is, let go of the rope.” I thought back to how Diola’s second interviewer had opened him up:

Only you know these things Diola. If you are willing, you’ll tell me, and if you’re not, you won’t. I can’t force you to tell me – I don’t want to force you. I’d like you to help me understand. Would you tell me about what happened?

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.