In 1996, some researchers discovered that depressed people often had an unusual version of the serotonin transporter gene 5-HTTLPR. The study became a psychiatric sensation, getting thousands of citations and sparking dozens of replication attempts (page 3 here lists 46).

Soon scientists moved beyond replicating the finding to trying to elucidate the mechanism. Seven studies (see here for list) found that 5-HTTLPR affected activation of the amygdala, a part of the brain involved in processing negative stimuli. In one especially interesting study, it was found to bias how the amygdala processed ambiguous facial expression; in another, it modulated how the emotional systems of the amygdala connected to the attentional systems of the anterior cingulate cortex. In addition, 5-HTTLPR was found to directly affect the reactivity of the HPA axis, the stress processing circuit leading from the adrenal glands to the brain.

As interest increased, studies began pointing to 5-HTTLPR in other psychiatric conditions as well. One study found a role in seasonal affective disorder, another in insomnia. A meta-analysis of twelve studies found a role (p = 0.001) in PTSD. A meta-analysis of twenty-three studies found a role (p = 0.000016) in anxiety-related personality traits. Even psychosis and Alzheimer’s disease, not traditionally considered serotonergic conditions, were affected. But my favorite study along these lines has to be 5-HTTLPR Polymorphism Is Associated With Nostalgia-Proneness.

Some people in bad life situations become depressed, and others seem unaffected; researchers began to suspect that genes like 5-HTTLPR might be involved not just in causing depression directly, but in modulating how we respond to life events. A meta-analysis looked at 54 studies of the interaction and found “strong evidence that 5-HTTLPR moderates the relationship between stress and depression, with the s allele associated with an increased risk of developing depression under stress (P = .00002)”. This relationship was then independently re-confirmed for every conceivable population and form of stress. Depressed children undergoing childhood adversity. Depressed children with depressed mothers. Depressed youth. Depressed adolescent girls undergoing peer victimization. They all developed different amounts of depression based on their 5-HTTLPR genotype. The mainstream media caught on and dubbed 5-HTTLPR and a few similar variants “orchid genes”, because orchids are sensitive to stress but will bloom beautifully under the right conditions. Stories about “orchid genes” made it into The Atlantic, Wired, and The New York Times.

If finding an interaction between two things is exciting, finding an interaction between even more things must be even better! Enter studies on how the interaction between 5-HTTLPR and stress in depressed youth itself interacted with MAO-A levels and gender. What about parenting style? Evidence That 5-HTTLPR x Positive Parenting Is Associated With Positive Affect “For Better And Worse” What about decision-making? Gender Moderates The Association Between 5-HTTLPR And Decision-Making Under Uncertainty, But Not Under Risk. What about single motherhood? The influence of family structure, the TPH2 G-703T and the 5-HTTLPR serotonergic genes upon affective problems in children aged 10–14 years. What if we just throw all the interesting genes together and see what happens? Three-Way Interaction Effect Of 5-HTTLPR, BDNF Val66Met, And Childhood Adversity On Depression.

If 5-HTTLPR plays such an important role in depression, might it also have relevance for antidepressant treatment? A few studies of specific antidepressants started suggesting the answer was yes – see eg 5-HTTLPR Predicts Non-Remission In Major Depression Patients Treated With Citalopram and Influence Of 5-HTTLPR On The Antidepressant Response To Fluvoxamine In Japanese Depressed Patients. A meta-analysis of 15 studies found that 5-HTTLPR genotype really did affect SSRI efficacy (p = 0.0001). Does this mean psychiatrists should be testing for 5-HTTLPR before treating patients? A cost-effectiveness analysis says it does. There’s only one problem.

ALL.

OF.

THIS.

IS.

LIES.

Or at least this is the conclusion I draw from Border et al’s No Support For Historical Candidate Gene Or Candidate Gene-by-Interaction Hypotheses For Major Depression Across Multiple Large Samples, in last month’s American Journal Of Psychiatry.

They don’t ignore the evidence I mention above. In fact, they count just how much evidence there is, and find 450 studies on 5-HTTLPR before theirs, most of which were positive. But they point out that this doesn’t make sense given our modern understanding of genetics. Outside a few cases like cystic fibrosis, most important traits are massively polygenic or even omnigenic; no one gene should be able to have measurable effects. So maybe this deserves a second look.

While psychiatrists have been playing around with samples of a few hundred people (the initial study “discovering” 5-HTTLPR used n = 1024), geneticists have been building up the infrastructure to analyze samples of hundreds of thousands of people using standardized techniques. Border et al focus this infrastructure on 5-HTTLPR and its fellow depression genes, scanning a sample of 600,000+ people and using techniques twenty years more advanced than most of the studies above had access to. They claim to be able to simultaneously test almost every hypothesis ever made about 5-HTTLPR, including “main effects of polymorphisms and genes, interaction effects on both the additive and multiplicative scales and, in G3E analyses, considering multiple indices of environmental exposure (e.g., traumatic events in childhood or adulthood)”. What they find is…nothing. Neither 5-HTTLPR nor any of seventeen other comparable “depression genes” had any effect on depression.

I love this paper because it is ruthless. The authors know exactly what they are doing, and they are clearly enjoying every second of it. They explain that given what we now know about polygenicity, the highest-effect-size depression genes require samples of about 34,000 people to detect, and so any study with fewer than 34,000 people that says anything about specific genes is almost definitely a false positive; they go on to show that the median sample size for previous studies in this area was 345. They show off the power of their methodology by demonstrating that negative life events cause depression at p = 0.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001, because it’s pretty easy to get a low p-value in a sample of 600,000 people if an effect is real. In contrast, the gene-interaction effect of 5-HTTLPR has a p-value of .919, and the main effect from the gene itself doesn’t even consistently point in the right direction. Using what they call “exceedingly liberal significance thresholds” which are 10,000 times easier to meet than the usual standards in genetics, they are unable to find any effect. This isn’t a research paper. This is a massacre.

Let me back off a second and try to be as fair as possible to the psychiatric research community.

First, over the past fifteen years, many people within the psychiatric community have been sounding warnings about 5-HTTLPR. The first study showing failure to replicate came out in 2005. A meta-analysis by Risch et al from 2009 found no effect and prompted commentary saying that 5-HTTLPR was an embarrassment to the field. After 2010, even the positive meta-analyses (of which there were many) became guarded, saying only that they seemed to detect an effect but weren’t sure it was real. This meta-analysis on depression says there is “a small but statistically significant effect” but that “we caution it is possible the effect has an artifactual basis”. This meta-analysis of 5-HTTLPR amygdala studies says there is a link, but that “most studies to date are nevertheless lacking in statistical power”.

Counter: there were also a lot of meta-analyses that found the opposite. The Slate article on the “orchid gene” came out after Risch’s work, mentioned it, but then quoted a scientist calling it “bullshit”. I don’t think the warnings did anything more than convince people that this was a controversial field with lots of evidence on both sides. For that matter, I don’t know if this new paper will do anything more than convince people of that. Maybe I trust geneticists saying “no, listen to me, it’s definitely like this” more than the average psychiatrist does. Maybe we’re still far from hearing the last of 5-HTTLPR and its friends.

Second, this paper doesn’t directly prove that every single study on 5-HTTLPR was wrong. It doesn’t prove that it doesn’t cause depression in children with depressed mothers in particular. It doesn’t prove that it doesn’t cause insomnia, or PTSD, or nostalgia-proneness. It doesn’t prove that it doesn’t affect amygdala function.

Counter: it doesn’t directly prove this, but it casts doubt upon them. The authors of this paper are geneticists who are politely trying to explain how genetics works to psychiatrists. They are arguing that single genes usually matter less than people think. They do an analysis of depression to demonstrate that they know what they’re talking about, but the points they are making apply to insomnia, nostalgia, and everything else. So all the studies above are at least questionable.

Third, most of these studies were done between 2000 – 2010, when we understood less about genetics. Surely you can’t blame people for trying?

Counter: The problem isn’t that people studied this. The problem is that the studies came out positive when they shouldn’t have. This was a perfectly fine thing to study before we understood genetics well, but the whole point of studying is that, once you have done 450 studies on something, you should end up with more knowledge than you started with. In this case we ended up with less.

(if you’re wondering how you can do 450 studies on something and get it wrong, you may be new here – read eg here and here)

Also, the studies keep coming. Association Between 5-HTTLPR Polymorphism, Suicide Attempts, And Comorbidity In Mexican Adolescents With Major Depressive Disorder is from this January. Examining The Effect Of 5-HTTLPR ON Depressive Symptoms In Postmenopausal Women 1 Year After Initial Breast Cancer Treatment is from this February. Association Of DRD2, 5-HTTLPR, And 5-HTTVNTR With PTSD In Tibetan Adolescents was published after the Border et al paper! Come on!

Having presented the case for taking it easy, I also want to present the opposite case: the one for being as concerned as possible.

First, what bothers me isn’t just that people said 5-HTTLPR mattered and it didn’t. It’s that we built whole imaginary edifices, whole castles in the air on top of this idea of 5-HTTLPR mattering. We “figured out” how 5-HTTLPR exerted its effects, what parts of the brain it was active in, what sorts of things it interacted with, how its effects were enhanced or suppressed by the effects of other imaginary depression genes. This isn’t just an explorer coming back from the Orient and claiming there are unicorns there. It’s the explorer describing the life cycle of unicorns, what unicorns eat, all the different subspecies of unicorn, which cuts of unicorn meat are tastiest, and a blow-by-blow account of a wrestling match between unicorns and Bigfoot.

This is why I start worrying when people talk about how maybe the replication crisis is overblown because sometimes experiments will go differently in different contexts. The problem isn’t just that sometimes an effect exists in a cold room but not in a hot room. The problem is more like “you can get an entire field with hundreds of studies analyzing the behavior of something that doesn’t exist”. There is no amount of context-sensitivity that can help this.

Second, most studies about 5-HTTLPR served to reinforce all of our earlier preconceptions. Start with the elephant in the room: 5-HTTLPR is a serotonin transporter gene. SSRIs act on the serotonin transporter. If 5-HTTLPR played an important role in depression, we were right to focus on serotonin and extra-right to prescribe SSRIs; in fact, you could think of SSRIs as directly countering a genetic deficiency in depressed people. I don’t have any evidence that the pharmaceutical industry funded 5-HTTLPR studies or pushed 5-HTTLPR. As far as I can tell, they just created a general buzz of excitement around the serotonin transporter, scientists looked there, and then – since crappy science will find whatever it’s looking for – it was appropriately discovered that yes, changes in the serotonin transporter gene caused depression.

But this was just the worst example of a general tendency. Lots of people were already investigating the role of the HPA axis in depression – so lo and behold, it was discovered that 5-HTTLPR affected the HPA axis. Other groups were already investigating the role of BDNF in depression – so lo and behold, it was discovered that 5-HTTLPR affected BDNF. Lots of people already thought bad parenting caused depression – so lo and behold, it was discovered that 5-HTTLPR modulated the effects of bad parenting. Once 5-HTTLPR became a buzzword, everyone who thought anything at all went off and did a study showing that 5-HTTLPR played a role in whatever they had been studying before.

From the outside, this looked like people confirming they had been on the right track. If you previously doubted that bad parenting played a role in depression, now you could open up a journal and discover that the gene for depression interacts with bad parenting! If you’d previously doubted there was a role for the amygdala, you could open up a journal and find that the gene for depression affects amygdala function. Everything people wanted to believe anyway got a new veneer of scientific credibility, and it was all fake.

Third, antidepressant pharmacogenomic testing.

This is the thing where your psychiatrist orders a genetic test that tells her which antidepressant is right for you. Everyone keeps talking these up as the future of psychiatry, saying how it’s so cool how now we have true personalized medicine, how it’s an outrage that insurance companies won’t cover them, etc, etc, etc. The tests have made their way into hospitals, into psychiatry residency programs, and various high-priced concierge medical systems. A company that makes them recently sold for $410 million, and the industry as a whole may be valued in the billions of dollars; the tests themselves cost as much as $2000 per person, most of which depressed patients have to pay out of pocket. I keep trying to tell people these tests don’t work, but this hasn’t affected their popularity.

A lot of these tests rely on 5-HTTLPR. GeneSight, one of the most popular, uses seven genes. One is SLC6A4, the gene containing 5-HTTLPR as a subregion. Another is HTR2A, which Border et al debunked in the same study as 5-HTTLPR. The studies above do not directly prove that these genes don’t affect antidepressant response. But since the only reason we thought that they might was because of evidence they affected depression, and now it seems they don’t affect depression, it’s less likely that they affect antidepressant response too.

The other five are liver enzymes. I am not an expert on the liver and I can’t say for sure that you can’t use a few genes to test liver enzymes’ metabolism of antidepressants. But people who are experts in the liver tell me you can’t. And given that GeneSight has already used two genes that we know don’t work, why should we trust that they did any better a job with the liver than they did with the brain?

Remember, GeneSight and their competitors refuse to release the proprietary algorithms they use to make predictions. They refuse to let any independent researchers study whether their technique works. They dismiss all the independent scientists saying that their claims are impossible by arguing that they’re light-years ahead of mainstream science and can do things that nobody else can. If you believed them before, you should be more cautious now. They are not light-years ahead of mainstream science. They took some genes that mainstream science had made a fuss over and claimed they could use them to predict depression. Now we think they were wrong about those. What are the chances they’re right about the others?

Yes, GeneSight has ten or twenty studies proving that their methods work. Those were all done by scientists working for GeneSight. Remember, if you have bad science you can prove whatever you want. What does GeneSight want? Is it possible they want their product to work and make them $410 million? This sounds like the kind of thing that companies sometimes want, I dunno.

I’m really worried I don’t see anyone updating on this. From where I’m sitting, the Border et al study passed unremarked upon. Maybe I’m not plugged in to the right discussion networks, I don’t know.

But I think we should take a second to remember that yes, this is really bad. That this is a rare case where methodological improvements allowed a conclusive test of a popular hypothesis, and it failed badly. How many other cases like this are there, where there’s no geneticist with a 600,000 person sample size to check if it’s true or not? How many of our scientific edifices are built on air? How many useless products are out there under the guise of good science? We still don’t know.