A psychiatrist who once "treated" me used to recite this rueful little mantra: "They say failed doctors become psychiatrists, and that failed psychiatrists specialise in drugs." By drugs this psychiatrist meant drugs of addiction – and his "treatment" of me consisted of prescribing Temgesic, a synthetic opiate, as a substitute for the heroin I was more strongly inclined to take. So, he undertook this role: acting, in effect, as a state-licensed drug dealer; and he also attempted a kind of psychotherapy, talking to me about my problems and engaging with my own restless critique of – among many other things – psychiatry itself. Together we conceived of doing some sort of project on drugs and addiction, and began undertaking research. On one memorable fact-finding trip to Amsterdam, we ended up smoking a great deal of marijuana as well as drinking to excess – I also scored heroin and used it under the very eyes of the medical practitioner who was, at least nominally, "treating" me.

All of this happened more than 20 years ago, and I drag it up here not in order to retrospectively censure the psychiatrist concerned, but rather to present him and his behaviour as a perversely honest version of the role played by his profession. For what, in essence, do psychiatrists specialise in, if not mood-altering drugs? Or, to put it another way, what do psychiatrists have to offer – over and above the other so-called "psy professions" – beyond their capacity to legally administer psychoactive drugs, and in some cases forcibly confine those they deem to be mentally ill?

Psychiatry is undergoing one of its periodic convulsions at the moment – one that coincides with the publication by the American Psychiatric Association of the fifth edition of their hugely influential "Diagnostic and Statistical Manual of Mental Disorders" (DSM–5) – and I think we should all take the opportunity to join in the profession's own collective navel-gazing and existential angst. After all, while the influence of the talking cures is pervasive in our society – running all the way up the scale from anodyne advice dispensed on daytime TV shows, to the wealthy shelling out hundreds of pounds a week to pet their neuroses in the company of highly qualified black dog walkers – psychotherapy and psychoanalysis remain essentially voluntaristic undertakings; only psychiatry deals in mandatory social care and legal sanction. Besides, only psychiatry partakes of the peculiar mystique that attaches to medical care. We may dismiss the opinions of all sorts of counsellors and therapists, secure in the knowledge that their very multifariousness is indicative of their lack of overall traction, but psychiatry, dealing, as it claims, with well-defined maladies – and treating them with drugs and hospitalisation – exerts an enormous pull on our collective self-image. Just what the nature of this pull is, and how it has come to condition our understanding of ourselves and our psychic functioning, is what I wish to unpick.

Full-blown mental illness is an extremely frightening phenomenon to observe – let alone experience. And much of the debate that surrounds the efficacy of contemporary psychiatry is warped by the knowledge – lurking in the wings of our minds – that we wish to have as little as possible to do with it. We may understand rationally that psychosis isn't a contagion, yet still we turn aside from the street soliloquisers and avoid the tormented gazes of those being "cared for in the community". Arguably, the response of those who treated a trip to Bedlam to view the madmen and women as an entertainment had the virtue of at least being a form of contact. At their peak, mental hospitals such as Bedlam (and formerly known as "lunatic asylums") housed over 100,000 inmates, many of whom had been confined for behaviours that today would be regarded as lifestyle choices, such as socialism or sexual promiscuity. The hospitals were also dumping grounds for patients who we now know to have had organic brain diseases. It's sobering for those on the left to realise that the first politician to commit to their abolition was Enoch Powell. By the early 1990s many long-stay inmates had been returned to the outside world, but their lives were for the most part still grossly circumscribed: living in sheltered accommodation and visited by mental health teams, confined not by physical walls but by the chemical straitjackets of neuroleptic drugs.

An engraving of a bedridden patient at the New York City Lunatic Asylum Hospital in the late 1860s. Photograph: Stock Montage/Getty Images

Still, if you wish to visit Bedlam you can do so. The locked mental wards of our hospitals present a terrifying spectacle of seriously disturbed patients shouting, yelping, gurning and shaking – I know, I've seen them. And it's the much-repressed knowledge that this is going on that helps, I would argue, to prevent too much criticism of the psychiatric profession. Just as we are quietly grateful to prison officers for banging up criminals, so too we are grateful for psychiatrists and psychiatric nurses for providing a cordon sanitaire between us and flamboyant insanity. Yet while the regime under which those diagnosed with mental pathologies has changed immensely in the last half-century, the prognosis remains no better. Some say that it is manifestly worse, and that psychiatry itself is to blame. But the truth is that hardly anyone – apart from the professionals, whose livelihoods depend on it – can either be bothered to wade through the reams of scientific papers concerned with the alternative treatment regimens, or understand the different methodologies arrived at to assess competing claims.

Early in Our Necessary Shadow, his lucid, humane and in many ways well-balanced account of the nature and meaning of psychiatry, Tom Burns, professor of social psychiatry at Oxford University, makes a supremely telling remark: "I am convinced psychiatry is a major force for good or I would not have spent my whole adult life in it." This is a form of the logical fallacy post hoc ergo propter hoc ("After this, therefore because of this"), and it seems strange that an academic of such standing should so blithely retail it because, of course, if we reverse the statement it makes just as much sense: "Having spent my whole adult life as a psychiatrist I must maintain the conviction that it is a major force for good." After all, the alternative – for Burns and for thousands of other psychiatrists – is to accept that in fact their working lives have constituted something of a travesty: either locking up or drugging patients whose diseases are defined not by organic dysfunction but by socially unacceptable behaviours. Burns has the honesty and integrity to admit that the major mental pathologies – schizophrenia, bipolar disorder, depression inter alia – cannot be defined in the same way as physical diseases, and he cleaves to the currently fashionable view of psychiatry as seeking to understand mental maladies through the tripartite lens of the social, the psychological and the biological. He also states that he sees the role of psychotherapy as central to the practise of psychiatry – and in this he dissents from the more mainstream "biological" model of treatment that has been in the ascendancy since the 1970s.

But what Burns cannot quite bring himself to do is give up the drugs. In a 333 page book (complete with a glossary, a bibliography and an index), there are just three references to the most commonly prescribed psychiatric drugs: the SSRIs, or selective serotonin reuptake inhibitors (such as Prozac and Seroxat). When he does consider the SSRIs, he notes that they may indeed be overprescribed (as of 2011 46.7m prescriptions had been written in the UK for antidepressants), and in particular that they may be used to "treat" commonplace unhappiness rather than severe depression. What he doesn't venture near are the systematic critiques of antidepressants – and neuropharmacology in general – that have emerged in recent years. The work of Irving Kirsch, whose meta-analysis of SSRI double-blind trials revealed that in clinical terms – for a broad spectrum of depressed patients – SSRIs acted no better than a placebo, is something Burns doesn't want to look at. He also doesn't wish to examine too closely the underlying "chemical imbalance" theory of depression on which the alleged efficacy of the SSRIs is based, presumably because he knows that it's essentially bunk: no fixed correlation has been established, despite intensive study, between levels of serotonin in the brain and depression.

Antidepressant tablets. Photograph: Jonathan Nourok/Getty Images

I've swerved into consideration of antidepressants because I believe the exponential increase in their use is a function of the problem of legitimacy that psychiatry currently faces. Psychiatrists, of course, tell the public that the vast majority of these drugs are prescribed by general practitioners – not by them. But what has made it possible for someone recently bereaved or unemployed to have a prescription written by their doctor to alleviate their "depression", is, I would argue, very much to do with psychiatry's search for new worlds to conquer, an expedition that has been financed at every step by big pharma. Put bluntly: unable to effect anything like a cure in the severe mental pathologies, at an entirely unconscious and weirdly collective level psychiatry turned its attention to less marked psychic distress as a means of continuing to secure what sociologists term "professional closure". After all, if chlorpromazine (commonly known as Largactil) and other neuroleptics don't cure schizophrenia – any more than lithium "cures" bipolar illness – then why exactly do you need a qualified medical doctor to dole them out?

The proliferation of new psycho-pharmacological compounds has advanced in lock-step with the proliferation of new mental illnesses for them to "treat". As Ian Hacking observes in a review of DSM–5 in the current London Review of Books, the first DSM – published in 1952 – and its successor in 1968, were heavily influenced by the psychoanalytic theories then dominating psychiatry in the US. In 1980, with DSM–III there came a step-change. Hacking traces this to the efficacy of lithium in managing mania: "Now there was something that worked … clear behavioural criteria were necessary to identify who would benefit from lithium." James Davies begins his book, Cracked: Why Psychiatry Is Doing More Harm Than Good, with an examination of how these behavioural criteria were arrived at by the compilers of DSM–III and its subsequent incarnations. You may be thinking that all this is so much arcane knowledge – and wondering why we in Britain should be preoccupied by a diagnostic manual published in the US. But in fact the ICD (International Classification of Diseases) used by British doctors is compiled in the same way as the DSM – indeed most NHS psychiatrists favour the latter over the former. In the US it's simple: your insurance won't pay out unless you are diagnosed with a malady detailed in the DSM, but in Britain we have a less tangible – but for all that pervasive – form of socio-medical discrimination: no sick note – and no social benefits – unless what ails you conforms to the paradigms set out in DSM.

The focus of Davies's critique is that the criteria for what constitutes ADHD (attention deficit hyperactivity disorder), or autism, or indeed depression, are not arrived at by any commonly understood scientific procedure, but rather by committee: psychiatrists getting together and pooling their understanding of how patients with these maladies "present" (in the jargon). Under these circumstances it becomes somewhat easier to understand how the tail can begin to wag the dog: rather than arriving at a commonly agreed set of symptoms that constitute a gestalt – and hence a malady – psychiatrists become influenced by what psycho-pharmacological compounds alleviate given symptoms, and so, as it were, "create" diseases to fit the drugs available. This in itself, Davies might argue, explains why there are more and more new "diseases" with each edition of the DSM: it isn't a function of scientific acumen identifying hitherto hidden maladies, but of iatrogenesis: doctor-created disease. So, while it may well be general practitioners who do the doling out, psychiatrists are required to legitimate what they are doing and provide it with the sugar-coating of scientific authenticity. It's a dirty, well-paid and high-status job – but someone has to do it, no?

The vast number of "hyperactive" children in the US prescribed Ritalin is so well attested to that it's become a trope in popular culture – just like the SSRI-munching depressive. But these are our version of low-level "care in the community", the sad are becoming oddly co-morbid (afflicted with the same sorts of diseases) with the mad. Davies treads a familiar path in his critique of the influence of the multinational pharmaceutical companies on the structure and practice of psychiatry. If you aren't familiar with the fact that almost all drug trials are funded by those who stand to profit from their success then … well, you jolly well should be. You should also be familiar with the extent to which university research departments and learned journals are funded by those who stand to profit – literally – from their presumed objectivity. The money generated by the SSRIs in particular is vast, easily enough to warp the dynamics and the ethics of an entire profession, and indeed I would agree with Davies that it has in fact done just this. The sections of his book that deal in particular with the way big pharma has moved into markets outside the English-speaking world and effected a wholesale cultural change in their perception of sadness (rebranding it, if you will, as chemically treatable "depression"), simply in order to flog their dubious little blue pills, make for chilling reading.

Actually, Burns would agree with some of this critique as well; and recall, he's a psychiatrist who fervently believes that his profession has been, and continues to be, a force for good. Davies is a psychologist, and to the outsider the fierceness of his attack might be dismissed as part of a turf war among the psy professions (Irving Kirsch is a clinical psychologist as well). However, I don't think it helps anyone to see the current imbroglio as simply a function of late capitalism in its most aggressive aspect. I'm afraid I have to mouth the old lily-livered liberal shibboleth at this point and observe that, yes, we are all to blame; and our responsibility is just as difficult for us to acknowledge because we are largely unaware of it. We don't consciously collude in the chemical repression of the psychotic (and Davies produces quite convincing statistics to support the view that those with psychosis actually recover better if they aren't medicated at all), any more than we consciously collude in the fiction of depression as a chemical imbalance that can be successfully treated with SSRIs.

Instead, what both clinicians and patients experience is quite the reverse: we feel absolutely bloody miserable, we can't get up in the morning, we are dirty and unkempt, and we go along to our GP and are prescribed an antidepressant, and lo and behold we recover. My GP, who has just retired, and who is a wise and compassionate man who I absolutely trusted, told me that he prescribed SSRIs because they worked, and I believed him. That they worked because of the overpoweringly efficacious curative power we believe doctors and their nostrums to possess rather than because of any real change in our brain chemistry was beside the point for him – and I suspect it's beside the point for the vast majority of patients as well. By the same token, Burns is at pains to stress, contra-DSM, that the great strength and skill of the practising psychiatrist lies in being able to intuit diagnoses by empathising with patients. Diagnosis, for Burns, is an art form – not a science. By his own account I've little doubt that he's a good and effective psychiatrist who can make a real difference to the lives of those plagued by demons that undermine their sense of self. One of my oldest friends is a consultant psychiatrist who I've actually seen practising in just this way, with preternatural flair and compassion.

In both their cases, however, I feel about them rather the way I do about the last archbishop of Canterbury: I consider Rowan Williams to be a wise and spiritual man mostly despite rather than because of his Christianity; and I think many psychiatrists are good healers mostly despite rather than because of the medical ideology of mental illness to which they subscribe.

Interestingly there is one large sector of the "mentally ill" that Burns believes are manifestly unsuitable for treatment – drug addicts and alcoholics. He points to the ineffectiveness of almost all treatment regimens, possibly because the cosmic solecism of treating those addicted to psychoactive drugs with more psychoactive drugs hits home despite his well-padded professional armour. Elsewhere in Our Necessary Shadow he seems to embrace the idea that self-help groups of one kind or another could help to alleviate a great deal of mental illness, and it struck me as strange that he couldn't join the dots: after all, the one treatment that does have long-term efficacy for addictive illness is precisely this one.

Psychiatrists are notoriously unwilling to endorse the 12-step programmes, and argue that statistically the results are not convincing. There may be some truth in this – but there's also the inconvenient fact that there's no place for psychiatrists, or indeed any of the psy professionals, in autonomously organised self-help groups. Burns agrees with Davies that our reliance on psychiatry, and by extension, psycho-pharmacology, may well be related to our increasingly alienated state of mind in mass societies with weakened family ties, and often non-existent community ones. Surely self-help groups can play a large role in facilitating the rebirth of these nurturing and supportive networks? But Burns seems to feel that just as we will always need a professional to come and mend the septic tank, so we will always need a pro to sweep out the Augean psychic stables. I'm not so sure; psychiatry has been bedevilled over the last two centuries by "treatments" and "cures" that have subsequently been revealed to be significantly harmful. From mesmerism, to lobotomy, to electroconvulsive therapy, to Valium and other benzodiazepines – the list of these nostrums is long and ignoble, and I've no doubt that the SSRIs will soon be added to their number.

Sooner or later we will all have to wake up, smell the snake oil, and realise that while medical science may bring incalculable benefit to us, medical pseudo-science remains just as capable of advance. After all, one of the drugs that Irving Kirsch's meta‑analysis of antidepressant trials revealed as being just as efficacious as the SSRIs was … heroin.