When I heard that Johann Hari had written a book about the war on drugs, two immediate concerns sprang to mind. The first was whether anyone would trust a word he wrote.

The author used to be the Independent’s star columnist, a prolific polemicist and darling of the left, until his career imploded in disgrace when it emerged in 2011 that many of his articles contained quotes apparently said to him but in fact lifted from his interviewees’ books, or from previous interviews by other journalists. Worse, he was exposed as a “sockpuppet”, or someone who anonymously furthers his own interests online. Using a false identity, Hari had maliciously amended the Wikipedia pages of journalists he disliked – among them the Telegraph columnist Cristina Odone and the Observer’s Nick Cohen – accusing them of antisemitism, homophobia and other toxic falsehoods. Under the same pseudonym, he had also edited his own Wikipedia page, lavishly flattering his profile to, as he puts it, “big myself up”. The Independent suspended him, four months later he resigned, and no British newspaper has published his journalism since. He has never spoken publicly about the scandal, until now.

My other worry was whether anyone would want to read yet another polemic about drugs. I wouldn’t, and I’m quite interested in the subject. The prohibition-versus-legalisation debate tends to be interminably dreary, chiefly because neither side ever seems to change anybody’s mind.

“I think that’s totally right,” Hari agrees. “I did not want to write a 400-page polemic about the drug war. I didn’t want to have an argument about it, I wanted to understand it.” For that matter, he admits, “It’s struck me that, actually, polemic very rarely changes people’s minds about anything.” He says so as a former columnist? “A recovering former columnist, yes.” He laughs. “It’s not just that polemic doesn’t change people’s minds. It says nothing about the texture of lived experience. People are complex and nuanced, they don’t live polemically.”

Hari’s book turns out to be a page-turner, full of astonishing revelations. I had no idea that the war on drugs was single-handedly invented by a racist ex-prohibition agent, who needed to find a new problem big enough to protect his departmental budget. One of the first victims of his ambition was Billie Holiday, whose heroin addiction enraged him to the point where he hounded her to death. After he’d had the singer jailed for drugs, she was stripped of her performing licence, and as she unravelled into destitution and despair, his agents continued to harass her, even summoning a grand jury to indict her as she lay dying under police guard in a hospital bed.

The opposite of addiction is not sobriety. The opposite of addiction is human connection Johann Hari

Hari travelled all over the world meeting other casualties of the drug war: a transsexual former crack dealer in Brooklyn; a homeless junkie in Vancouver who mobilised the local heroin addicts into activists and rewrote the city’s drug laws; a housewife from Ciudad Juárez who marched across Mexico to shame the politicians and cartels protecting her daughter’s murderer.

Woven between the human dramas are Malcolm Gladwell-ish examinations of the surprising science and statistics of drug use, and of the varying success of drug policies. Hari goes to Portugal, where all drug possession was decriminalised 13 years ago, and where even the police chief of the Lisbon drug squad now admits, “The things we were afraid of didn’t happen.” He also visits Tent City, a prison in the Arizona desert where the inmates live in tents in temperatures of 44C, wear T-shirts proclaiming I AM BREAKING THE NEED FOR WEED or I WAS A DRUG ADDICT, and are shackled into a chain gang every day and marched in public while reciting chants of repentance.

I’ve got to know Hari a bit over the past two years, and it’s quite hard to picture him in the badlands of the drug war. He doesn’t look much like an underworld adventurer. He could pass for almost a decade younger than his 35 years, is slightly knock-kneed and prone to giggly yelps, and readily admits he can’t be trusted to make it to the corner shop and back without getting lost. His north London flat, where we meet, is full of books and almost nothing else, its sole concession to domesticity a massive flatscreen TV. Hari puts his general air of unworldly distraction down to his dyspraxia, but it comes across as donnish.

There was nothing academic about his background, growing up in suburban north London in the 80s. His Glaswegian mother worked in a refuge, his Swiss father was a bus driver, and they were pleased but rather puzzled by their son’s obsession with books. “My mum and my dad and my brother all left school when they were 16. I was the first person in my family to go to a fancy university.” After graduating with a double first in social and political sciences from Cambridge, he joined the New Statesman and quickly established his name as a columnist. He was hired by the Independent, where he won Young Journalist of the Year in 2003 and became the youngest ever recipient of the George Orwell prize in recognition of his political reporting.

Johann Hari: “I want to make it clear that I’m not in any way attributing anything I did to that drug use. They are totally separate things.” Photograph: Richard Saker

As a journalist, Hari wrote a lot about the war on drugs, and was always a passionate opponent. His book is presented as an objective investigation, but did it really change his mind about anything? “Oh yes. I think the thing that shocked me the most was the stuff about addiction. I thought I knew about addiction. But addiction is not what we’ve been told it is at all.”

When Hari began the book three years ago, he was familiar with the two prevailing theories: people become dependent on drugs either because they lack self-control, or because the chemicals are so inherently addictive that they hijack the brain. Addiction is a moral weakness, or it is a disease, but implicit in either analysis is the theoretical possibility that if we could get rid of the drugs, we would solve the problem.

One of Hari’s earliest memories is of trying to rouse a relative from a drug-induced stupor, and his ex-boyfriend is a crack and heroin addict. “So I’d seen addiction in people I loved, and I could see it wasn’t that they were just selfish, morally flawed people. I never believed that. So I erred towards thinking, well, obviously it must be a disease.” Seminal experiments conducted on rats in the 70s appeared to have proved this. Offered a choice between pure water and water laced with heroin, the rats quickly became addicted to the opiate and kept taking the drug until it killed them.

But something didn’t add up. “Every day, all over the world, hospital patients are given medical heroin, diamorphine, very often for long periods. And virtually none of them afterwards goes out and tries to score on the street. Which made me think, the issue here can’t just be the drug.”

Hari went to Vancouver to meet a psychology professor, Bruce Alexander, who had been similarly puzzled, so had replicated the original experiments. This time, instead of experimenting on solitary rats locked in empty cages, he offered the choice of clean or drugged water to rats kept in what he called Rat Park, a kind of rat heaven full of wheels and coloured balls and delicious food, and other rats to play and mate with. When these rats tried heroin, they weren’t very interested.

“They just didn’t like it. None of them overdosed. Even more strikingly, he then took rats that had become addicted in the isolated cages, and put them into Rat Park. And they almost immediately stopped using. What Alexander had found is that we’ve fundamentally misunderstood what addiction is. It isn’t a moral failing. It isn’t a disease. Addiction is an adaptation to your environment. It’s not you; it’s the cage you live in.”

The book is populated by a compelling cast of meth users, junkies and crack addicts. Other than addiction, what they have in common is heartbreaking early trauma and abuse. Childhood violence and prostitution, abandonment and homelessness, all led their victims to the same remedy: a narcotic anaesthetic for pain and loneliness. “Human beings have an innate need to bond. Healthy, happy people bond with other humans. But if you can’t do that because you’re so traumatised by your childhood that you can’t trust people, you may well bond with a drug instead.” The scientific evidence of the correlation is so overwhelming, Hari writes, that “child abuse is as likely to cause drug addiction as obesity is to cause heart disease”.

Child abuse is as likely to cause drug addiction as obesity is to cause heart disease Johann Hari

“What I learned is that the opposite of addiction is not sobriety,” Hari says. “The opposite of addiction is human connection. And I think that has massive implications for the war on drugs. The treatment of drug addicts almost everywhere in the world is much closer to Tent City than it is to anything in Portugal. Our laws are built around the belief that drug addicts need to be punished to stop them. But if pain and trauma and isolation cause addiction, then inflicting more pain and trauma and isolation is not going to solve that addiction. It’s actually going to deepen it.”

He breaks off, looking anxious. “But I didn’t tell their stories, because I thought they were a better way of persuading people of an argument. It’s a book of stories about people, because I think stories are a fundamentally better way of thinking about the world.”

Nevertheless, these stories build a compelling case for the legalisation and regulation of drugs. If, as the book suggests, 90% of us can take drugs safely without harming ourselves or others, and criminalising the 10% who can’t only turns them into desperate thieves and prostitutes, then the war on drugs is not merely unwinnable, but inherently counterproductive. But one of the stories Hari tells is mentioned so fleetingly that a reader could blink and miss it. In just a few sentences, Hari writes that he himself had been addicted to a drug for several years.

Provigil is an anti-narcolepsy prescription drug, popular among Ivy League students for its reputed power to turbo-charge the brain. When Hari discovered it in 2009, “I thought: great, I’ve maxed out the amount of antidepressants you can use – here’s something that will speed things up even more.” He’d been prescribed the antidepressant Seroxat at 17 and, barring one or two brief breaks, had been taking it ever since. Now, with Provigil, Hari was thrilled to discover, “you can do even more work, and be constantly processing information, and sleep only four hours a night”. He began buying the drug on the internet – and for a while it worked. But when he tried to stop taking it, he failed. “When you’re prone to depression, there can be a strong temptation, or there was for me anyway, to try to accelerate through it – to speed up, to kind of outrun the feelings of depression and I did that for years.

“But this is totally unrelated to the things I did wrong journalistically,” he says quickly. “This is really important. I did those things before and during the use of this drug. So I want to make it clear that I’m not in any way attributing anything I did to that drug use. They are totally separate things.”

This is the last time he says anything quickly. The moment we come to his scandal, all the animation drains from him; he turns still and pale, and speaks in halting sentences prefaced by painful silences. He stopped taking both Provigil and Seroxat one week after leaving the Independent, but can’t be sure what withdrawal was like because, “It’s hard to separate the challenge of stopping those drugs from the wider challenge of what was happening at that time.” I ask if he would place himself in the 10% vulnerable to addiction, and he says, “Probably at that point, yes. Not now, because I’ve changed the way I live so much that I wouldn’t put myself in that category any more.” But when I ask how his susceptibility relates to his childhood, he falls silent.

I’m ashamed of what I did. I did some things that were really nasty and cruel Johann Hari

“Look,” he says eventually, “I can talk to you about why what happened in my life happened. But I just think that’s a way of trying to invite sympathy, and that would be weaselly. If you tell a detailed personal story about yourself, you’re inherently asking people to sympathise with you, and actually I don’t think people should be sympathetic to me. I’m ashamed of what I did. I did some things that were really nasty and cruel.”

Suspicions began circulating online in 2011, when bloggers noticed uncanny similarities between quotes in Hari’s work and previously published interviews and books. The New Statesman began to dig further, and soon the internet was awash with incriminating examples. At the same time, several journalists who’d clashed with Hari in the past, including Nick Cohen and Cristina Odone, began to wonder in public about the identity of a mysteriously vengeful Wikipedia contributor who’d been editing their pages. He called himself “David Rose” and began issuing inventively elaborate online denials of the accusation that he was really Johann Hari. After his IP address was traced back to the Independent’s offices, the deception crumbled. The humiliation must have been toe-curling, but when I ask how he dealt with it, he says, “I just think that would be asking people to see it from my point of view.” Sympathy should be for the people he smeared online, for the Independent and its readers: “Not for me.”

Hari had sounded considerably less remorseful when the plagiarism allegations first surfaced. He found them “bemusing”, he wrote on his website, and justified using quotes interviewees had not said to him because his interviews were “intellectual portraits”. The defence sounded rather grandiosely self-serving – so why should anyone trust his new-found contrition? “I think, when you’re in the middle of being attacked, obviously your defence mechanisms go up and you can’t think clearly. It’s the moment when you most need your good judgment and are the least able to bring it.”

The mystery is why someone so clever could have behaved so stupidly. I ask Hari to explain what he’d been thinking, and he literally winces.

“I’m very reluctant to go into a personal narrative and give the why. Most people restrain their self-aggrandising and cruel impulses, and I failed to. I failed badly. I think when you do that, when you harm people, you should shut up, go away and reflect on what happened. Going on about myself would just be arrogant and actually repeating being nasty, and that’s what I’m trying not to be. When you fuck up, you should privately reckon with the harm you have caused and you should pay a big price.”

When the scandal broke, the George Orwell prize board ordered Hari to return his award. The Independent published a personal apology and sent him off to journalism school in New York on unpaid leave, but in January 2012 he resigned from the paper. Was the price he paid disproportionately high? Hari shakes his head. “It was incredibly humiliating, yes. It was absolutely devastating, and I fell apart. But I would not want to live in a culture where people could be horrible about other people under a pseudonym online, or act as if something someone had written had been said directly to you, and not pay a big price for it.”

For his new book, Hari has posted audio files online of every interview, so he is obviously worried about his credibility. Does he think this elaborate transparency will restore it? “Well, I fucked up and it’s perfectly right for people to be sceptical. I know I’ve got work to do in regaining trust.”

I ask how he thinks his disgrace changed him. “In two really big ways. One was just slowing down; writing much more slowly, living much more slowly, being less work-obsessed.” He is single, and shares his flat with a primary schoolteacher, one of his oldest friends. Most of his friends stood by him through the scandal, and a new one has been Russell Brand, for whom Hari has worked off and on since helping him prepare his 2013 standup tour, Messiah Complex. “We have long political conversations, and sometimes that sparks something useful for him, and then I send him links or books about what we’ve discussed.” He helps produce Brand’s podcast, the Trews, but when I ask about the rumours that he ghost-wrote Revolution, he laughs. “No. I didn’t write a word of it. As I suspect anyone who reads it can tell, those are all Russell’s words.”

It’s a relief, he says, not to be “in a state of mania any more, ripped up into a frenzy of constant opinionating and polemicising in my room”. I’m amazed more columnists don’t lose it, unhinged by the insatiable outrage of the Twittersphere, but when I ask if he thinks the register of public debate is becoming dangerous, he smiles. “Funnily enough, one of the good things about not being a columnist is that I don’t have to have an opinion on things like this. You’re probably right, and it sounds very persuasive to me, but I don’t want to join an angry argument against angry arguments.”

The other big change, he says, is that “I lost my taste for rendering judgment on people. I think now I’m more interested in understanding why people are the way they are.” What does he miss about his former life as a columnist? This time the silence lasts so long that I wonder if he heard the question. “It was the only job I ever wanted to do,” he says finally, and looks ashen. But he is already working on his next book, and says his new life makes him happier and healthier than his old one ever could.

If disgrace turned out to be a release, it still doesn’t feel that way to Hari. “No.” He shakes his head. “I’m not going to present it like a redemptive fable. It was awful, and I wish I hadn’t done it, and I wish I could go back and undo it.”