Around halfway through Johann Hari’s new book, he recalls visiting a conference in Sweden organised by the World Federation Against Drugs – as its own blurb puts it, “a multilateral community of non-governmental organisations and individuals” which rather optimistically claims to “work for a drug-free world”. The WFAD’s position on the so-called “war on drugs” is the opposite of Hari’s: he favours liberalisation and decriminalisation, the Federation support prohibition. In that sense, his visit to Stockholm is a behind-enemy-lines exercise that you would have thought would produce no end of anecdotes.

But no: oddly, the whole trip is over and done with in five brisk paragraphs. Moreover, though Hari credits the conference’s keynote speaker, Robert DuPont, with being “the man who created many of the metaphors that help us to understand drugs today”, an interview with him results in quotes that run to a mere 52 words. Hari describes DuPont, the first director of the US government’s National Institute on Drug Abuse, delivering “the knockout speech” of the Swedish event, “summing up a conference that warned that chemicals can hijack your brain and cause chemical slavery”. One hundred and seventy pages later, this oration is referenced in the book’s extensive notes. “Du Pont himself did not use the imagery of hijacking or chemical slavery in his speech, and does not like these metaphors,” Hari admits, “but they recurred at the conference many times.”

With any other writer, awkward moments such as this might not seem terribly important. But, as is well known to a crowd of media-watchers who tend to look at the world through Twitter, Chasing the Scream arrives three years after Hari was discovered to have plagiarised other people’s work, misrepresented the material he got from interviews and, under an alias, to have spread malicious falsehoods about other journalists via Wikipedia. What that entails for any reviewer is obvious enough: though it might be nice to set aside the events of 2011 and allow him a fresh start, his misdemeanours inevitably colour your experience of the book.

Hari well knows this, hence explanatory notes that stretch to nearly 60 pages, and the fact that all the recorded interviews for the book will be available online (from the publication date onwards, apparently). But what Chasing the Scream betrays is a little more complicated than the zero-sum stuff of truth and fiction. He took the very modern career path of becoming a high-profile polemicist before he had done much reporting, and perhaps as a result his writing is too melodramatic, a little naive, and reluctant to give a fair shout to the other side of the argument – things reflected in a tone that too often falls into being either shrill, or over-emotional. Some of this, I dare say, may shine some light on how he landed himself in all that trouble.

That is not to deny that Chasing the Scream is an important and largely convincing book, nor that Hari has done an incredible amount of work on it. His aim is to recount the history of the “war on drugs”, report on its serial absurdities and perverse consequences, and then explore the alternatives to prohibition. With real skill, he establishes the essential triad that defines any scenario involving the use, sale and policing of any illicit substance, via portraits of three celebrated figures: Harry Anslinger, the first commissioner of the Federal Bureau of Narcotics, who placed a completely neurotic view of drug-taking at the heart of US public life, where it remains; the New York master criminal Arnold Rothstein, whom Hari credits with the invention of the modern drug gang; and Billie Holiday, whose awful personal history was of a piece with her alcoholism and addiction to heroin. These interrelated stories serve as archetypes: though the text moves swiftly to the present day and flips through an array of other case studies, it is these three lives that show just how monotonously unchanging – and therefore futile – the essential rules of the “war on drugs” have always been.

Hari’s run of compelling latter-day characters takes us from a transsexual reformed dealer in Brooklyn to a bereaved mother in Mexico, and on in turn to an array of would-be reformers who have tried to offer an alternative to the war’s self-defeating logic. Most breathtaking is the tale of a punitive anti-drug regime in Arizona that took the life of Prisoner 109416 – aka Marcia Powell – a fortysomething crystal meth addict who in May 2009 was left outdoors in a metal cage for four hours in searing heat by prison officers encouraged to believe that addicts had rendered themselves subhuman. Her terrible treatment underlines what across-the-board prohibition and the cruelties that often come with it do to users: as Hari points out, under these conditions, people drawn to drugs because of emotional stability or a profound lack of self-esteem have their core problems made immeasurably worse, whereupon the spiral of addiction only deepens.

As the book nears its polemical climax, he deals very well with places – Switzerland, Uruguay, the US states of Colorado and Washington, and particularly Portugal – where liberalising measures have pointed to a way out of the drug wars’ current stupidities. The last country in particular is a revelatory case study: since 2001, to be busted is to be subject to a process aimed at separating genuine addicts from recreational users – with the former offered help, up to “a job with a decent wage, away from the world where you used drugs”. It is some token of the draconian approaches used elsewhere that in the context of a lot of what Hari has seen, such modest measures seem almost visionary.

Some of the text is let down by a taxonomic expedient Hari shares with the authorities he rails against: talking in general terms about “drugs”, when the difference between, say, a weekend marijuana habit and a life ruined by crystal meth suggests that as catch-all term, the “d” word can be all but useless. “Drug use is deeply widespread,” he writes, “and mostly positive.” What this last word is intended to mean is not made clear. I am as much of an advocate of liberalisation as he is, but there is not much that is “positive” about meth, heroin or crack – nor, come to think of it, skunk weed, or the pick-me-up that is powdered cocaine, which makes casualties of plenty of users and hyperventilating bores of those who snort it recreationally. Hari half-acknowledges some of this, but too often glosses over what the concerted use of particular drugs sometimes entails. That issue need not diminish the merits of pro-liberalisation arguments, but it has to be tackled.

His biggest problems, though, are a tendency to insert himself into the cracks between his stories, and his often histrionic turn of phrase. No one, it seems, has explained to him the strengths of the show-don’t-tell school of non-fiction writing. He tells the grim story of a cop’s rape of a heroin-addicted woman and the resultant birth of a child who went on to be a dealer, but then ends it with a real clunker: “a child of the drug war in the purest sense – he was conceived on one of its battlefields”. Bemusingly, the same character is described as “an electrical storm with skin”: in the notes Hari makes it clear to the reader he has “used this phrase before, in a column for the Independent, describing Bette Davis”. Later, when he compares the compulsion to gamble with drug addiction, he superfluously points out that “you don’t inject a deck of cards into veins; you don’t snort a roulette wheel”.

Chasing the Scream is a powerful contribution to an urgent debate, but this is its central problem: in contrast to the often brutal realities it describes, it uses the gauche journalistic equivalent of the narrative voice found in Mills & Boon novels. Amid Mexican sand dunes, he tells us, Hari thought about the drug wars’ endless downsides as he “ran my fingers through the prickly hot white sand” and crassly imagined the joyous lives of local teenagers in a world free of gangsters (“Juan, stripped of his angel wings, is chatting with Rosalio about World of Warcraft”). By the end, as he discusses the details of taxing marijuana with a civil servant from Colorado, he says that he is “bored at last, and I realise a tear of relief is running down my cheek”. Thanks to such melodrama, and the book’s slightly excitable tone, one conclusion is all but inescapable. The title of Chasing the Scream is a reference to the young Harry Anslinger’s experience of hearing a drug-addicted woman howl for a fix, but it might easily apply to the sensibility of the author himself.

• To order Chasing the Scream for £14.99 go to bookshop.theguardian.com or call 0330 333 6846.