“I’ve done my research.” If you’ve ever debated someone who disagrees with a scientific consensus, then you’ve probably encountered that sentence, especially if they were an anti-vaccer. It is the mantra of the anti-science movement, but it’s nearly always misused. You see, in science, doing research generally means conducting a scientific study and adding new information to the general body of scientific knowledge. Nevertheless, I don’t want to dwell on semantics, and I think that people should educate themselves; however, if you are going to educate yourself, then you have to read good sources (which in science means the peer-reviewed literature), and you can’t cherry-pick which papers to read and which papers to ignore. This is where our story turns to Sherri Tenpenny’s “Vaccine Research Library” (VRL).

The title sounds great, doesn’t it? A single library that houses all of the literature on vaccines would be a wonderful tool; however, the VRL does not contain all of the literature on vaccines. Instead, it only contains the papers that oppose vaccines. So, rather than being a legitimate research tool, it is actually the most glorious confirmation bias generator that I have ever encountered. I could not have asked for a more beautiful example of cherry-picking sources. Therefore, in this post I will not only explain why the VRL is a load of crap, but I will also use it as an illustration of how not to do research.

Note: Although they don’t house all of the literature on vaccines, you can find most studies on PubMed and Google Scholar. So use them if you want to actually be well-informed.

What is the purpose of the VRL?

I’m going to let Tenpenny answer this question for me, because her statements are better than anything I could write (I suggest that you don’t drink anything while reading this section because her justification for this website is honestly pretty funny).

Pro-vaccine information is as abundant and as easy to find as ice in Antarctica. But there is a large body of overlooked medical and scientific research that shows the other side – and chronicles the heartbreaking disasters and long-term health consequences caused by vaccines. The problem is that locating this information can be challenging, difficult to interpret and very time consuming to dig out.

On a different part of the site, she says,

In 2011, we realized how difficult – and time consuming – it is to find mainstream medical references documenting the harm being caused by vaccines. Finding these “needles in the haystack” is a tedious and time-consuming task.

Now, a rational person would think that maybe there is a scientific reason that pro-vaccine papers are so predominant, but that doesn’t stop Tenpenny from plowing forward. Further, she clearly contradicts herself. First, she says that there is a large body of anti-vaccine literature, then she goes on about how hard it is to find these papers, and she refers to them as “needles in the haystack.” So which is it? Are they abundant or aren’t they, and if this body of “overlooked” research is so large, then why is it hard to locate? Why do you have to dig it out? The vast majority of journals archive their abstracts in Google Scholar so if there is actually a large body of literature showing that vaccines are dangerous, then it should be easy to find those papers. The fact that it is difficult to find anti-vaccine publications actually demonstrates just how weak the anti-vaccine position truly is. So Tenpenny is really defeating her own argument.

Another section of the page says (the bold text and bizarre capitalization are in the original):

Convinced that Vaccines are Unsafe but Need Scientific Proof? You need information that gives you “The Other Side of the Story.”

Here we have the real problem. As I have frequently argued, anti-vaccers (and anti-scientists in general) have no interest in being well-informed. They don’t actually care about facts. Rather, they care about protecting their preconceptions. This “library” is not designed for people who actually want to learn about vaccines. Rather, it is intended for those who have already decided that vaccines are dangerous. Stephen Colbert brilliantly described this way of thinking when he coined the word “truthiness,“and it aptly describes the purpose of this website. It isn’t for people who want to carefully analyze the facts and evidence. Rather, it’s for people who know in their gut that vaccines are bad, and it is intended to bolster an existing belief rather than help people to evaluate evidence. Tenpenny makes this explicitly clear with statements like,

They want evidence to support what they intuitively know: The Party Line about vaccines is a charade, perpetuated to bolster profits and expand Big Pharma’s cartel.

Once again, it’s about cherry-picking evidence to support a belief rather than actually informing yourself about the topic. According to Tenpenny, however, her site will help to balance your knowledge.

Now, all in one place, is the irrefutable science you need to defend your position against vaccines. You will be able to prove your point, protect your health and that of your children, write balanced news stories, or support legal cases.

Think about how absurd this is for a minute. First, she claims that reading a tiny subset of the literature will give you irrefutable evidence. Then, she claims that totally ignoring the majority of the literature will help you to write balanced news stories! It’s like me saying, “here is a paper proving that the earth is flat. It disproves all of the papers saying that the earth is round, and it will let you write a balanced news story on why the earth is actually flat.”

To conclude this section, I want to give and discuss one final quote from her site which I find particularly amusing (again the emphasis is in the original).

Concerned that reviewing all this information will be time consuming? “Pre-search” takes the “grunt work” out of your research. How much time do you spend on the Internet searching and researching…, searching and researching…, and searching and researching…..for reliable scientific facts about the problems associated with vaccines?Because browsers and web crawlers deliver a large number of results, it can take hours to troll through page after page…after page…after page of search results. Then clicking on link after link. Then skimming through reams of material to find a particular fact. What’s worse is the exasperation you feel when you come up empty-handed – after investing so much time, you didn’t find what you were looking for.

…

Now think about how much you are paid per hour in your Day Job. Take that dollar amount times the hundreds, even thousands, of hours you spend on the Internet, searching for information that can be frustratingly difficult to find.

…

The annual membership rate has been drastically reduced: A one year membership to the Library is worth thousands of dollars and hundreds of hours of your time, you can have full access to thousands of references for only $9.98 per month (for quick research of a specific topic) or only $99 – for a full year!

Let me paraphrase this, “Are you tired of spending hours trying to find that one anecdote that supports your preconceptions? Is cherry-picking data taking up too much of your time? Are you annoyed with having to scroll past website after website that says you’re wrong? Well then I have a deal for you, because I’ve cherry-picked the internet for you! Now, for the low price of $100 per year, you can have all information that conforms to your distorted view of reality without having to be bothered with the thousands of studies that say you’re full of crap! Order now, and we’ll even include a free jar of cherries.”

Addendum (26-1-2016): Originally, there were two paragraphs here that questioned Tenpenny’s financial motives for making this site, but as someone pointed out in the comments, they were admittedly speculative, and I don’t really think that they are relevant to the point that I am trying to make, so I removed them.

What is in the VRL?

At this point, I think it is clear that the VRL is not motivated by an honest desire to be well-informed. Nevertheless, let’s look closer because regardless of the motivations for constructing this site, if Tenpenny actually found a large body of properly conducted studies showing that vaccines are dangerous, then we should take those studies seriously. I’m clearly not about to give Tenpenny one penny of my money, however, so I activated a free trial version of the VRL. This admittedly only gave me access to part of the library, but I see no reason to think that the rest of it would be substantially different.

Before I describe the contents of the library, I want to remind everyone that not all scientific studies are equal. Some designs produce very robust, reliable results, whereas others produce very weak, unreliable results. So you should always be careful to avoid the trap of latching onto a study just because it agrees with you. You have to carefully evaluate the study and look at the design that was used to determine whether or not the results are reliable (I explained the hierarchy of evidence in more detail here).

With that in mind, it probably won’t surprise you to learn that the vast majority of studies in the library rank very low on the hierarchy of evidence. For example, there are a large number of case reports. These are the lowest category on the hierarchy of evidence because they are basically just glorified anecdotes. If a doctor observes someone having a heart attack after receiving a vaccine, for example, they would write a case report on it, but that does not in any way shape or form prove that the vaccine caused the heart attack. It could be a total coincidence that the person had a heart attack after the vaccine. In fact, using anecdotes and case reports to draw causal conclusions is a logical fallacy known as post hoc ergo propter hoc. So, rather than proving that vaccines are dangerous, these case reports should (and are) used as the basis for starting large, robustly designed studies to actually test whether or not vaccines cause the reported symptoms, but you don’t see many of those large studies in the VRL because they tend not to fit anti-vaccers’ preconceptions.

Of the studies that did use robust designs, the sample sizes tended to be small, and many of them suffered serious methodological flaws, were published in questionable journals, etc. So rather than being a collection of studies that prove that vaccines are dangerous, the VRL is really a collection of the lowest quality, weakest studies on vaccines. To be clear, there are a few decent studies in the list, but many of those are misrepresented, and you always have to consider scientific papers within the broader context of the literature (more on that later).

What really amazed me about the contents of the VRL, however, was Tenpenny’s ability to cherry-pick within a study. For example, I was very surprised to see a review paper (Shepard et al. 2006) on Hepatitis B infections and vaccinations (remember, reviews are one of the highest levels of evidence). The presence of this paper confused me because it is overwhelmingly supportive of vaccines. Here is an excerpt from the abstract:

Vaccination against HBV infection can be started at birth and provides long-term protection against infection in more than 90% of healthy people. In the 1990s, many industrialized countries and a few less-developed countries implemented universal hepatitis B immunization and experienced measurable reductions in HBV-related disease…Further progress towards the elimination of HBV transmission will require sustainable vaccination programs with improved vaccination coverage, practical methods of measuring the impact of vaccination programs, and targeted vaccination efforts for communities at high risk of infection.

So why on earth is a paper that encourages increased vaccination efforts in the library that supposedly proves that vaccines are dangerous? It’s there because of three sentences.

The earliest recognition of the public health importance of hepatitis B virus (HBV) infection is thought to have occurred when it appeared as an adverse event associated with a vaccination campaign. In 1883 in Bremen, Germany, 15 percent of 1,289 shipyard workers inoculated with a smallpox vaccine made from human lymph fell ill with jaundice during the weeks following vaccination. The etiology of “serum hepatitis,” as it was known for many years, was not identified until the 1960s , and only following the subsequent development of laboratory markers for infection was its significance as a major cause of morbidity and mortality worldwide fully appreciated.

Before I talk about those sentences, I want to make something else clear about the VRL. Access to this library does not give you access to the papers themselves (despite the fact that her page about the VRL clearly implies that you get the full papers). Rather, you get abstracts and a brief blurb from Tenpenny where she has highlighted the “important” parts of the paper for you. In other words, she is cherry-picking within studies! She is actually encouraging people to not only pick and choose which studies to accept, but to actually pick and choose which sentences to accept. Her excerpt from the Shepard et al. study illustrates that perfectly (the emphasis was hers, btw). Out of an entire review that talks about the massive body of literature showing that the Hepatitis B vaccine is useful, she wants you to read just three sentences. In other words, this entire paper describes why she is full of crap, but she wants you to ignore that and focus on three sentences from the introduction instead. It’s the most absurd and outlandish level of cherry-picking that I have ever seen.

Further, why she thinks that these three sentences show that vaccines are dangerous is beyond me. My guess is that she is arguing that the vaccine was contaminated with Hep B, to which I respond, so what? It makes absolutely no sense to say, “the vaccine was contaminated in 1883, therefore it is dangerous now.” Medical technologies have come a long way since 1883. It’s like saying, “the earliest computers were massive and slow, therefore modern computers are no good.” It seems that Tenpenny is suggesting that we should ignore the massive body of evidence supporting the vaccine and focus instead on a mistake that was made (and corrected) decades ago.

Note: Someone is probably getting ready to accuse me of hypocrisy since I also highlighted just a few sentences from the paper, but before you do that, realize that I was simply using those sentences to show that the paper was pro-vaccine. I am not in any way shape or form suggesting that you use those sentences as evidence that the vaccine is safe. For that, you need to read the entire paper (not just the abstract) as well as the rest of the literature on the topic. Finally, unlike Tenpenny’s quote, mine was actually representative of the paper.

Why Tenpenny’s method doesn’t work

Science is a messy process, and reaching a firm conclusion generally involves lots of studies from numerous research groups. As a result, the body of literature on any given topic will contain lots of statistical noise. In other words, there will generally be lots of preliminary studies with small sample sizes or weak designs, and there will be multiple studies that reached the wrong conclusion just by chance. This is why whenever you are trying to learn about a scientific topic, you have to look at the entire body of literature, not just a few cherry-picked studies. There is so much researching being done, that there are lots of bad papers out there (sometimes at no fault of the authors), and you can find a paper to support almost any position that you can think of. There are, for example, still people who think that the earth is flat, and if you start with that assumption, you can find “evidence” and even a few scientific papers to support it (for example, Benard et al. 1904, which you can find an excerpt from here). This is why it is so important that you avoid the single study syndrome. Individual studies have a high probability of being wrong, but it is far less likely that a large body of studies is wrong.

You should never latch onto a single study as irrefutable proof of your position, but that is exactly what Tenpenny is encouraging you to do. In her mind (and in the minds of anti-scientists more generally) all that you have to do to prove your position is find one study that agrees with you (or even one sentence). It doesn’t matter if the study was done correctly, it doesn’t matter what the sample size was, it doesn’t matter if the study used a robust design, it doesn’t matter if there are a thousand other studies that disagree with you. According to her way of thinking, finding that one study is all that you need, but that’s clearly not how science or logic actually works. Replication is one of the central tenets of science, and scientists only reach a consensus after a result has been replicated multiple times and supported by numerous studies. So Tenpenny is ignoring a fundamental principle of science. Further, what she is doing is actually a logical fallacy known as the Texas sharpshooter fallacy. This fallacy occurs whenever you focus on the subset of data that appears to support your position, while ignoring a much larger body of data that refutes your position.

Additionally, she is ignoring a fundamental principle of rational thought: you always have to start with an unbiased question. It’s fine to ask a question like, “are vaccines safe?” then look for answers to that question, but Tennpenny and her followers are starting with the assumption that they are dangerous, then looking for evidence to support that assumption. The problem is that if you do that, if you start with a conclusion, then you will always find something which supports that conclusion (at least in your mind).

Now, invariably some anti-vaccer reading this is going to say, “you’re committing a hasty generalization fallacy. Not all anti-vaccers are like that. I actually have looked at both sides and become well-informed.” In which case, my response is, why do you reject the thousands of papers that clearly demonstrate that vaccines are safe and effective? I’m guessing that it’s either because you have read a few faulty, low quality studies and are choosing to rigidly cling to them (in which case you are doing exactly what Tenpenny is) or you are blindly rejecting them for one of the flawed reasons that I described here. To put this another way, where’s your evidence? If your position is actually based on an unbiased review of the data, then surely you can provide me with a large body of high quality, properly controlled, robustly designed studies that have been replicated by other research groups which show that vaccines are dangerous and which provide a valid explanation for why thousands of other studies disagree with them. Unless you can do that, then you are succumbing to the same confirmation bias as Tenpenny, and you are picking and choosing what evidence to accept (no, the vaccine inserts, VAERS, and NVICP do not count as evidence that vaccines are dangerous, see the links for details).

Conclusion

In this post, I have been focusing specifically on Tenpenny and the anti-vaccers who follow her, but everything that I have been talking about is widely applicable to everyone. We are all prone to confirmation biases (myself included). It’s ingrained in our psychology to latch onto evidence that supports our views and disregard evidence that doesn’t. The key, therefore, is to acknowledged that tendency and strive to overcome it. If we are going to actually be well informed on any topic, then we must ensure that we are not simply succumbing to confirmation biases. We have to look at the entire body of evidence, not just the subset that conforms to our preconceptions. That’s why I find the VRL so infuriating. Rather than helping people to become truly open-minded, it insists that people should close their minds to any evidence that supports vaccines, and it openly encourages people to adhere to confirmation biases. It equates gut feelings with actual evidence, and it encourages people to seek out “proof” for their views rather than testing whether or not those views are actually justified. This, in my opinion, is the worst form of pseudoscience and pseudoskepticism, because it doesn’t just mislead people about the evidence. Rather, it misleads them about the way to evaluate the evidence. If you want to truly understand our marvelous universe, then you must train yourself to recognize and avoid this false skepticism, and you must always accept the possibility that you might be wrong. So to any anti-vaccers reading this, I’m not trying to attack you, and I don’t think that you’re stupid, but you have been seriously mislead and misinformed about the evidence and how to evaluate that evidence. You need to learn to recognize confirmation biases and you have to consider the entire body of evidence, not just the pieces of evidence that support your view.