As I’m putting the (hopefully) final touches on a short textbook that I’m writing entitled “Handbook on Science Literacy”, I’ve been thinking a lot about how to recommend a person go about systematically investigating a scientific issue without having any background in it. Sure, you can learn how to read and understand a scientific article, but let’s be honest—far too many people choose instead to do a quick web search and let that settle the question. This practice works okay in some instances, but in others it produces misleading or wrong answers.

I want to share with you my strategies for flunking out of the University of Google.

This is one instance where flunking is a good thing. A graduate of the University of Google chooses to accept only information that supports his or her position, and ignores or dismisses information in conflict with it. A graduate of the University of Google will not be able to answer the question “What kind of evidence would change your mind on this subject?” It’s insidious, because once their opinions are formed in this way, they tend to identify with other people who share those opinions, and any new information that comes their way will either be accepted or rejected on the basis of which position they’ve already taken (the cultural cognition effect)

None of us want to be that kind of person.

Flunking out requires a decent amount of work, and the willingness to accept that you might be wrong about a subject from time to time. You’ll need to become more aware of your own cognitive biases, and have some strategies for overcoming them.

So as a preliminary step down the road to science literacy, I’ve put my thoughts on this together into a guide to learning about a subject in which you have no background. It’s an exercise; please don’t shortcut the process and go to Wikipedia, or you’ll miss the whole point.

1. Start by identifying a question that you want to find an accurate answer to, such as “Do vaccines cause autism?” or “Is genetically modified food harmful to human health?”, or “Do cell phones cause cancer?”. Try to keep this as specific as possible. Remember: one question can build upon another, but answer them one at a time.

2. Now turn that question into a testable null hypothesis. In the examples above, your hypotheses could be:

“Vaccines do not cause autism”

“Genetically modified food is not harmful to human health.”

“Cell phones do not cause cancer.”

Remember that in order to accurately answer your question, your hypothesis must be falsifiable! That is, you must be able to articulate what kind of evidence would disprove it. If you can’t, it might not be a scientific question, or you might have already made up your mind on the subject. Be honest with yourself: are you trying to find the truth on a subject, or just the kind of information you’re comfortable with?

3. Write down a few notes on what kinds of evidence would disprove your hypothesis. How would you design an experiment to collect this evidence?

4. Open up three separate browser windows:

https://scholar.google.com/

http://www.ncbi.nlm.nih.gov/pubmed

https://www.google.com/

5. Type key words from your hypothesis into the search bar of each page. Examine the first few pages of results from each search, taking notes summarizing what you found in each type of search. Note the type of publication and whether that publication was likely peer reviewed, as best as you can tell. Why peer review matters so much. Journals indexed on PubMed will be peer reviewed*. Books aren’t often peer reviewed, so I tend to assume they aren’t unless I know otherwise.

Here are notes on the first two results from my Google Scholar search just to give you an idea of how I do it:

Vaccines, autism, chronic inflammation: the new epidemic—book published by the National Vaccine Information Center (probably not peer reviewed)

Vaccines and autism: evidence does not support a causal association—journal article published in Clinical Pharmacology and Therapeutics (peer reviewed)

Once you have notes summarizing your results from all three types of search engines, answer the following questions:

As best as you can tell from their titles, how many of your results support your hypothesis? How many reject your hypothesis?

How do the results of your search differ between Google Scholar, PubMed, and regular Google? Why do you think that is?

6. Now choose a few of the references you found. How many you look at is up to you, and dependent on constraints such as time, which articles are open access, which books you can actually see online or in a library, etc. Just keep in mind that if you really want accurate information on a subject, the more references you look at, the better.

Be sure to look at both references that seem to support and references that seem to refute your hypothesis. You may need to look at the abstract of journal articles to determine this. Yes, here I said you shouldn’t start with an abstact when you’re reading a paper. But in order to find relevant papers, the abstract can be very helpful!

I highly recommend that you actually take the time to carefully read through the papers. But if you’re struggling, try to focus on extracting these pieces of information from the source:

What are the authors trying to test, specifically? What is their takeaway message? What is the evidence they put forward to support their takeaway message? What is the quality of the research design? Does it test a hypothesis? Does it have a large number of subjects? For medical studies, what type of study is it? Use this excellent guide to help you assess the level of evidence in a medical study

Now look at the source itself. If it’s in a journal, Google the journal and see whether it’s credible. For example, contrast the journals Lancet and Medical Hypotheses. In which do you think you’ll find more accurate information?

Now look at the authors of the papers, books, or web articles you’ve found. Do they have the appropriate expertise in the subject you’re researching? What do you find when you Google them? Are they respected by other experts who are working on the same issue? What are some of the criticisms of them that you find? Do you find those criticisms credible or troubling? Do they have any financial conflicts of interest in the subject that they’re researching such as industry positions, or products for sale on their website that purport to “cure” whatever condition they’re publishing about?

Hopefully you can now see that when you’re doing a search for information, the quality of the results matters. You’re going to get more accurate information from PubMed (or equivalent search engines in relevant disciplines) than you will from Google. You’re going to get more trustworthy, evidence-based information from peer-reviewed papers than from random websites. You’re going to get better, more reliable information from good journals than from bad journals.

7. At this stage, revisit your hypothesis. Do you think the evidence supports or rejects it? Did the evidence you find fit with the answers you gave in question 3? What specifically do you find most convincing on this issue? Why is that?

8. How would you summarize the case against the conclusions you’ve drawn? What are the flaws in the arguments advanced by that position? What evidence do they lack?

9. Finally, but perhaps most importantly: What further evidence would change your mind on this issue?

I welcome your suggestions and feedback on this guide. If you’d like to read more strategies for becoming scientifically literate, my book will be out this fall. I’ll update this post with a link to it, as soon as it’s available.

*Note that PubMed doesn’t index all peer-reviewed journals from all disciplines. You may need to substitute a different search engine for this. (I use Google Scholar a lot).

Many thanks to Colin, Dorit, and Grant for their helpful feedback on this post.