The new edition of the DSM replaces DSM-IV, which appeared in 1994. The DSM is the standard – and standardising – work of reference issued by the American Psychiatric Association, but its influence reaches into every nook and cranny of psychiatry, everywhere. Hence its publication has been greeted by a flurry of discussion, hype and hostility across all media, both traditional and social. Most of it has concerned individual diagnoses and the ways they have changed, or haven’t. To invoke the cliché for the first time in my life, most critics attended to the trees (the kinds of disorder recognised in the manual), but few thought about the wood. I want to talk about the object as a whole – about the wood – and will seldom mention particular diagnoses, except when I need an example.

Many worries have already been aired. In mid-May an onslaught was delivered by the Division of Clinical Psychology of the British Psychology Society, which is sceptical about the very project of standardised diagnosis, especially of schizophrenia and bipolar disorders. More generally, it opposes the biomedical model of mental illness, to the exclusion of social conditions and life-course events. On a quite different score, Allen Frances, the chief editor of DSM-IV, has for years been blogging his criticisms of the modifications leading to DSM-5. More and more kinds of behaviour are now being filed as disorders, opening up vast fields of profit for drug companies. I shall discuss none of these important issues, and will try to be informative and even supportive until the very end of this piece, where I address a fundamental flaw in the enterprise.

Who needs the 947 pages of the DSM-5? All that most consumers need is the DSM-5 Diagnostic Criteria Mobile App. The more interesting question is who needs the DSM anyway? First of all, bureaucracies. Everyone in North America who hopes their health insurance will cover or at least defray the cost of treatment for their mental illness must first receive a diagnosis that fits the scheme and bears a numerical code. For example, opening the book at random, I find 308.3 for Acute Stress Disorder. The coding is required both by American private insurers and by Medicare. It is also required for the universal health insurance plans provided in Canadian provinces.

There is another quite different bureaucratic use. Why is this a ‘statistical’ manual? Because its classifications can be used for studying the prevalence of various types of illness. For that one requires a standardised classification. In a sense, the manual has its origins in 1844, when the American Psychiatric Association, in the year of its founding, produced a statistical classification of patients in asylums. It was soon incorporated into the decennial US census. During the First World War it was used for assessing army recruits, perhaps the first time it was put to diagnostic use.

Although the manual is American, it is much used elsewhere, despite the fact that the International Classification of Diseases, drawn up under the auspices of the World Health Organisation in Geneva, is usually seen as the official manual, if there is one. DSM-5 gives ICD codes when they match, and there is a project aimed at harmonising the two rulebooks. For an American, however, being assigned a DSM code determines whether your health insurance will pay for treatment, and what kind of treatment you get. (The DSM itself carries no recommendations for treatment.) A diagnosis may also have other more subtle effects on how patients think of themselves, how they feel and how they behave. Especially since nowadays, when told their diagnosis, patients tend to look it up online. There they obtain a sort of stereotype of how they ought to be feeling and behaving. Typing Acute Stress Disorder into Google will give you about 400,000 results.

The DSM presents itself as a manual for clinicians. The word is intended to be neutral, applicable in the competing schools of psychiatry, psychology, psychoanalysis and so on. Webster’s defines a clinician as ‘one qualified or engaged in the clinical practice of medicine, psychiatry, or psychology, as distinguished from one specialising in laboratory or research techniques in the same fields’. Most leading English-language journals of psychiatry require that research papers discussing a mental illness characterise it using the DSM. This has passed relatively unnoticed, perhaps being thought of even as a good thing because it helps clarify concepts. Hence it came as a bombshell when, a week before DSM-5 was published, Thomas Insel, the head of the US National Institute for Mental Health – the primary funder of research in the field – announced that the NIMH was going to abandon the DSM because it dealt only with symptoms. He wanted science; he wanted genetic and neurological research, and believed that, as in any other field of medicine, this ought to be used to define disease entities.

A furore ensued, the cat among the pigeons. But the cat couldn’t care less about the pigeons (diagnoses preparatory to treatment); it was after mice – the biochemical or neurological basis of mental illness. If you take Webster’s literally, the DSM is (as it insists) for clinicians, while some more aetiological system of classification may be wanted for research. For those of us who doubt the NIMH medical model of all forms of madness, there is indeed cause for concern, but there is no principled contradiction between having a manual for clinicians and different guidelines for research. I do not deny there is a tension, but the two can coexist well enough.

Moreover, the DSM is a work in progress. Within weeks of the appearance of DSM-III in 1980, people were discussing what DSM-IV should look like. After DSM-III came DSM-IIIR (R for ‘revised’) in 1987, DSM-IV in 1994, DSM-IV TR (TR for ‘text revision’) in 2000, and now DSM-5. Some suggest that there will never be a ‘DSM-6’, on the grounds that the whole endeavour is self-destructing. Don’t count on it. It is on the contrary likely that the manual will become more attuned to neurological causes as these gradually conquer more and more of psychiatry. The DSM is a living, organic creature, kept alive by myriad worker bees. At the end of the book there is a list of about a thousand individuals, almost all medically qualified, who served as ‘Work Group Advisors’, carrying out ‘DSM-5 Field Trials in Academic Clinical Centres’ etc. Many thousands of students, technicians, secretaries and so forth must also have been involved. This is a deeply entrenched enterprise, fully supported by the immense American Psychiatric Association, with its 36,000 members. The DSM and its related publications are also said to be very profitable – to the tune of $5 million a year, according to the New York Times.

The first DSM (1952) and its successor, DSM-II (1968), were heavily influenced by the psychoanalysis then dominant in the United States. But with DSM-III in 1980 there was a new beginning. There were two notable causes, aside from the waning of psychodynamic therapy. First was the discovery of a genuinely effective drug for controlling mania. The Australian John Cade found that lithium really helped, and after a lot of scepticism (and many unwitting overdoses) the Federal Drug Administration approved its use in 1970; in 1974 it was approved for the treatment of manic depression. Before that, there was really no effective chemical treatment for any mental illness, but now there was something that worked. So clear behavioural criteria were necessary to identify who would benefit from lithium. Second was a comparative study in 1972 of diagnoses of schizophrenia in London and New York. It was a rude comeuppance. Schizophrenia was diagnosed about twice as frequently in New York as in London. Symptoms were agreed on, but not the final diagnosis. ‘Operational’ criteria had to be fixed. Since we did not understand the causes of most mental illness – or rather there were too many incompatible theories of causation – we should rely on syndromes, on observable patterns of symptoms, behaviour in short, on which there could be some agreement. This approach is often called Kraepelinian, after the great German psychiatrist Emil Kraepelin (1856-1926). Kraepelin divided serious psychosis into what he called ‘dementia praecox’ and ‘manic depression’. The former was redescribed by Eugen Bleuler in about 1910, and renamed schizophrenia. The latter, once called folie circulaire, is now called bipolar disorder, in order to exclude unipolar depression and unipolar mania. The distinction seems first to have been insisted on by the East German psychiatrist Karl Leonhard, in his systematic nosology of 1957.

And here they are in DSM-5, Schizophrenia 295.90 – but now with the addition of numerous subtypes – and Bipolar I and Bipolar II, 296.89, the latter described somewhere as ‘Bipolar lite’ (‘lite’ as in low-alcohol beer or diet Coke). But there are a lot of other codes in the chapter on ‘Schizophrenia Spectrum and Other Psychotic Disorders’ and the subsequent chapter on ‘Bipolar and Related Disorders’. These codes are our current means of describing and organising most of what was once just called madness or insanity. (Most of the diagnoses in the present DSM bear on some kind of dysfunction, but I would never speak of insanity in connection with them.) If I started trying to explain the new categories under schizophrenia, I would get lost in the forest. Indeed, in reading these sections I felt unable to see the tree – schizophrenia – for all the branches that were on display.

In order to suggest the global effects of this American manual, I’ll examine one particular disorder. In Pharmaceutical Reason: Knowledge and Value in Global Psychiatry (2005) Andrew Lakoff writes about gene-hunting drug companies which want lots of spit and blood samples so they can try to match up a disease with DNA, devise a way to detect the malady through DNA markers and then find a new drug that will ameliorate the symptoms. Mental disorders have to be identifiable by means of the DSM, because the US is the biggest market for medications. Partly to avoid ethics committees, and partly to keep a global net in place, the gene-hunters often go to impoverished places. In one case, a French drug company wanted DNA from bipolar patients. There was an underfunded mental hospital in Argentina, but it was psychodynamic in practice. Bipolar disorder is Kraepelinian, not Freudian, and so the hospital had no patients diagnosed as bipolar. The drug company made an offer the hospital could not refuse. So it reclassified its patients to DSM standards; doctors rethought and the patients experienced the symptoms in new ways. Such are the mechanisms of cultural imperialism.

We know a lot more than we did forty years ago, but we still don’t understand these classic forms of madness. We have lithium for bipolar disorder, where the primary problem is often that the patient ‘stops taking his meds’. There are numerous cocktails of drugs that relieve different forms of schizophrenia. The criteria for schizophrenia itself have been shifting around ever since Bleuler, although they have been stabilising in successive editions of DSM. Bleuler paid little attention to delusions and hallucinations, but later, hearing voices (auditory hallucinations) was sometimes critical to the diagnosis. Now this is played down. Lots of people hear voices, and many of them want to look after themselves. In the UK there is a Hearing Voices Network; the World Hearing Voices Congress meets later this year in Melbourne. This is an instance of patients trying to take control of their difficulties. The example I am most familiar with is autism, where neurodiversity and autism pride movements hold that autism is a difference from neurotypicals, not a disorder.

One of the reasons the manuals are so difficult to read is that the criteria take the form of menus. To take my example drawn at random, Acute Stress Disorder has two primary criteria, A and B. Under A the patient must have suffered something horrible ‘in one (or more) of the following ways’ – choose one or more from four. Under B we read ‘Presence of nine (or more) of the following symptoms in any of the five categories of …’ and there follows a list of 14 symptoms divided into five groups. And that is one of the simplest menus in the book.

This menu-like organisation has always been used in the DSM. DSM-5 owns up to two difficulties that anyone trying to use previous editions quickly experienced: NOS and comorbidity. NOS stands for ‘Not Otherwise Specified’. This is sensibly invoked when one does not have a good case history, as in an emergency room. But in the context of the DSM there was a problem. An entry would begin with a generic disorder, pass to various species and subspecies, and finally to NOS. Thus in DSM-IV, genus: ‘Schizophrenia and Other Psychotic Disorders’. Eight species: e.g. Schizophrenia. Five subspecies: e.g. Catatonic Type (295.20). After the first seven species with their subspecies, we come to the eighth: Psychotic Disorder, NOS (298.9). Some 32 generic disorders end with a species NOS, where patients are judged to fall under the generic heading but not under any of the specific headings.

What is happening here? The truth perhaps is that most psychiatrists and other clinicians do not bother with a DSM coding until they have to fill in the paperwork. They do their thinking in terms of prototypes, not definitions. They have a general picture of what a schizophrenic person is like, with various versions of varying degrees of specificity. An experienced clinician can often recognise a schizophrenic without needing much discussion or contact. Sometimes the species of schizophrenia is evident – catatonics are basically out of it, immobile, withdrawn, incapable of being aroused. But often the schizophrenic does not fit any of the subspecies criteria very well, providing another NOS for the bureaucrats.

DSM-5 does its best to drop NOS, but often ends up with a mess. Thus we now have ‘Schizophrenia Spectrum and Other Psychotic Disorders’ with a structure pretty different from that of DSM-IV. There is now a species ‘Catatonia’, with two subspecies, ‘Catatonia Associated with Another Mental Disorder (Catatonia Specifier)’ (289.89), and ‘Catatonic Disorder due to Another Medical Condition’ (293.89). The generic entry ends with a noncoded ‘Unspecified Catatonia’. This applies when we cannot make out the underlying condition, or the ‘full criteria are not met’, or if we simply lack information. Then we read ‘coding note: Code first 781.89 … followed by 293.89 unspecified catatonia’, which sounds very much like NOS. And 781.89 does not occur in the numerical list of codes at the back of the book.

Then there is comorbidity, which means that a patient may satisfy several diagnoses. Certainly someone can have multiple sclerosis and catch pneumonia. Hypertension often accompanies cancer. But here we are concerned with systematically overlapping diagnoses to the point that it is unclear that it makes sense to talk of the primary ailment. Throughout the book, many of the diagnoses include a paragraph headed ‘comorbidity’. Here is the entry for Bipolar I disorder:

Co-occurring mental disorders are common, with the most frequent disorders being any anxiety disorder (e.g. panic disorder, social anxiety disorder, specific phobia), occurring in approximately three-fourths of individuals, ADHD, any disruptive impulse-control or conduct disorder (e.g. intermittent explosive disorder, oppositional defiance disorder), and any substance abuse disorder (e.g. alcohol abuse disorder) occur in over half of individuals with Bipolar I disorder.

This shows us that the classification of mental illnesses is not at all like the classification of animals, vegetables or minerals. I spoke of genera, species and subspecies. This sort of hierarchy has been fixed ever since a young Swede arrived in Amsterdam in 1735, carrying the first draft of a ‘system of nature’ in which the three kingdoms of plants, animals and minerals were arranged by orders, classes, genera and species. It turned out to work poorly for minerals, but we still use the Linnaean system of taxonomy for the classification of living things. The system was an instant hit, and for the next century people tried to classify everything found in nature according to this scheme – including the chemical elements. Only when Darwin said ‘All true classification is genealogical’ did the penny drop: the Linnaean system works only when what is being classified arises in nature through something like descent. (Of course we organise things, especially people, into hierarchies all the time, witness the army, but I am talking of what we encounter in nature.)

The first stab at a medical diagnostic manual was made by a friend and exact contemporary of Linnaeus, with the rather daunting name of François Boissier de Sauvages de Lacroix, a physician and botanist in Montpelier. In 1763 Sauvages published his Nosology Methodica, explicitly stating in its title that it was modelled on the classification of plants. He had ten classes of illness, of which the eighth was madness. Each class was divided first into genera and then into species, producing 2400 kinds of malady.

There have been many systems for classifying mental illness since then, but all seem to me to be on the botanical model, and that has been their fatal flaw. Many other kinds of illness are very like plants, and can be uniquely characterised, as Kraepelin tried to do, by a distinctive pattern of symptoms when a cause is not yet known. We don’t use NOS in the rest of medicine, and we do not have much systematic comorbidity. Perhaps in the end the DSM will be regarded as a reductio ad absurdum of the botanical project in the field of insanity. I do not say this because I believe that most psychiatry will, some day, be reduced to neuroscience, biochemistry and genetics. I take no stance on that here. The NIMH said it would stop using DSM because it lacked ‘validity’. In fact the DSM-5 has made a great effort to make sure it meets the criteria for what it sees as validity. That is not my problem. I am making a claim grounded more on logic than on medicine. Sauvages’s dream of classifying mental illness on the model of botany was just as misguided as the plan to classify the chemical elements on the model of botany. There is an amazingly deep organisation of the elements – the periodic table – but it is quite unlike the organisation of plants, which arises ultimately from descent. Linnaean tables of elements (there were plenty) did not represent nature.

The DSM is not a representation of the nature or reality of the varieties of mental illness, and this is a far more radical criticism of it than Insel’s claim that the book lacks ‘validity’. I am saying it is founded on a wrong appreciation of the nature of things. It remains a very useful book for other purposes. It is essential to have something like this for the bureaucratic needs of paying for treatment and assessing prevalence. But for those purposes the changes effected from DSM-IV to DSM-5 were not worth the prodigious labour, committee meetings, fierce and sometimes acrimonious debate involved. I have no idea how much the revision cost, but it is not that much help to clinicians, and the changes do not matter much to the bureaucracies. And trying to get it right, in revision after revision, perpetuates the long-standing idea that, in our present state of knowledge, the recognised varieties of mental illness should neatly sort themselves into tidy blocks, in the way that plants and animals do.