In 1886, Pliny Earle, then the superintendent of the state hospital for the insane in Northampton, Massachusetts, complained to his fellow psychiatrists that “in the present state of our knowledge, no classification of insanity can be erected upon a pathological basis.” Doctors in other specialties were using microscopes and chemical assays to discern the material causes of illness and to classify diseases accordingly. But psychiatrists, confronted with the impenetrable complexities of the brain, were “forced to fall back upon the symptomatology of the disease—the apparent mental condition, as judged from the outward manifestations.” The rest of medicine may have been galloping into modernity on the back of science, but Earle and his colleagues were being left in the dust.

Thirty years later, they had not caught up. In 1917, Thomas Salmon, another leading psychiatrist, echoed Earle’s worry in an address to his colleagues, drawing their attention to the way that their reliance on appearances had resulted in a “chaotic” diagnostic system, which, he said, “discredits the science of psychiatry and reflects unfavorably upon our association.” Psychiatry, Salmon continued, needed a nosology that would “meet the scientific demands of the day” if it was to command public trust.

In the century that has passed since Salmon’s lament, doctors in most medical specialties have only gotten better at sorting our suffering according to its biochemical causes. They have learned how to turn symptom into clues, and, like Sherlock Holmes stalking a criminal, to follow the evidence to the culprit. With a blood test or tissue culture, they can determine whether a skin rash is poison ivy or syphilis, or whether a cough is a symptom of a cold or of lung cancer. Sure-footed diagnosis is what we have come to expect from our physicians. It gives us some comfort, and the confidence to submit to their treatments.

But psychiatrists still cannot meet this demand. A detailed understanding of the brain, with its hundred billion neurons and trillions of synapses, remains elusive, leaving psychiatry dependent on outward manifestations for its taxonomy of mental illnesses. Indeed, it has been doubling down on appearances since 1980, which is when the American Psychiatric Association created a Diagnostic and Statistical Manual of Mental Disorders (D.S.M.) that intentionally did not strive to go beyond the symptom. In place of biochemistry, the D.S.M. offers expert consensus about which clusters of symptoms constitute particular mental illnesses, and about which mental illnesses are real, or at least real enough to warrant a name and a place in the medical lexicon. But this approach hasn’t really worked to establish the profession’s credibility. In the four revisions of the D.S.M. since 1980, diagnoses have appeared and disappeared, and symptom lists have been tweaked and rejiggered with troubling regularity, generally after debate that seems more suited to the floors of Congress than the halls of science. The inevitable and public chaos—diagnostic epidemics, prescription-drug fads, patients labelled and relabelled—has only deepened psychiatry’s inferiority complex.

But it’s not entirely clear that psychiatrists want a solution to the problem, at least not to judge from what happened when the experts conducting the most recent revision of the manual, the D.S.M.-5, were offered one. A group of seventeen prominent doctors—biological psychiatrists, experts in diagnostics, subspecialists in the field of depression, and even a historian—petitioned the D.S.M.-5’s mood-disorders committee to add a diagnosis they named melancholia.

The proposal was not so much an innovation as a retrieval of an old idea. Melancholia is one of the most venerable of psychiatric disorders, noted by doctors at least as far back as Hippocrates, who attributed its characteristic dejection and unresponsiveness to external events to an excess of black bile. But melancholia lost its place in psychiatric nosology in 1980, when all forms of depression were consolidated under a single diagnostic label—“major depressive disorder”—of which melancholia was only a variant. It was the D.S.M. equivalent of calling Pluto just another ice dwarf in the Kuiper Belt.

The group argued that this was a grievous scientific error and cited evidence that melancholia was qualitatively different from other forms of depression. Some of the evidence was derived from the same kind of clinical observation that is the backbone of the D.S.M. For instance, people who showed the characteristic clinical symptoms—an unshakeable despondency and sense of guilt that arises from nowhere, responds to nothing, and dissipates for no apparent reason—also displayed some distinctive physical signs: hand-wringing, for instance, and psychomotor retardation, an easily perceived slowing down of movement, thought, and speech. But some of the group’s proof was of precisely the kind that psychiatrists had been looking for since the nineteenth century. Thirty years of replicated studies had shown that patients with those signs and symptoms had a sleep architecture and cortisone metabolism that was distinct from that of other people, both normal and depressed. A night in a sleep lab could detect the reduced deep sleep and increased REM time characteristic of melancholics, and a dexamethasone suppression test (D.S.T.) could determine whether or not a patient’s stress hormones were in overdrive, as is generally the case among melancholic patients. And melancholia responded better than other kinds of depression to two treatments: tricyclic antidepressants (the first generation of the drugs) and electroconvulsive therapy (E.C.T., better known as shock therapy). Treatment success rates with this population reached as high as seventy per cent, much more robust than the anemic results found in trials that mixed melancholic and non-melancholic depression, and melancholics were less likely to respond to placebos.

Distinctive signs, symptoms, lab studies, course, and outcome—if melancholia wasn’t the Holy Grail, it was at least a sip from the chalice of science, one disorder that could go beyond appearances. You would think that the committee would at least have been eager to consider it as a partial remedy for ongoing concerns about the profession’s lack of scientific rigor. But the panel barely gave melancholia the time of day, let alone a full-on floor debate, relegating it to the same slush pile as the proposed Parental Alienation Syndrome and Male-to-Eunuch Gender Identity Disorder. And the main obstacle was exactly what you would think was melancholia’s main strength: the biological tests, especially the D.S.T. “I believe you and your colleagues are fundamentally correct,” committee member William Coryell wrote to the melancholia advocates, by way of explaining his panel’s inaction. But “the inclusion of a biological measure would be very hard to sell to the mood group.” Coryell explained that the problem wasn’t the test’s reliability, which he thought was better than anything else in psychiatry. Rather, it was that the D.S.T. would be “the only biological test for any diagnosis being considered.” A single disorder that met the scientific demands of the day, in other words, would only make the failure to meet them in the rest of the D.S.M. that much more glaring.

Coryell also noted that the melancholia proposal represented a departure from the notion, central to the D.S.M.’s descriptive method, that that the criteria for depression constitute a single disorder that every patient who matches the description has. “Depression is depression is depression,” another mood-disorders specialist has said, and so, Coryell counselled, evidence for such a “sweeping change” in paradigm would have to be “extensive and compelling.” (Coryell declined to comment for this article.)

This notion—that the apparent mental condition is all that can matter—underlies not only the depression diagnosis but all of the D.S.M.’s categories. It may have been conceived as a stopgap, a way to bide time until the brain’s role in psychological suffering has been elucidated, but in the meantime, expert consensus about appearances has become the cornerstone of the profession, one that psychiatrists are reluctant to yank out, lest the entire edifice collapse.

Gary Greenberg’s new book, “The Book of Woe: The DSM and the Unmaking of Psychiatry,”from which some of this essay is adapted, will be published in May.