A 2019 paper in the Advances in Methods and Practices in Psychological Science found that most psychology textbooks, instructors, and students misinterpret ‘statistical significance’ and p values. Talk about a headline! More important than the headline, however, are the right interpretations and what we can do to correct widespread misinterpretations. In this post, I explain what the authors’ findings and the three solutions they propose.

I decided to explain all of this on Twitter. I’ll include the entire thread below. (Don’t worry. It’s not a Tweetstorm. It’s just 5 tweets. And there’s no moral outrage involved.

89% of intro. psych. textbooks described statistical significance incorrectly (https://t.co/0rGuWQ50u3)



So did 100% of psych. ugrads, 80% of methodology instructors and 90% of scientific psychologists (https://t.co/ivLPQfiuKn)



Yikes!



So, um, what's the right description? (1/5) — Nick Byrd (@byrd_nick) September 14, 2019

K. And what's a p value?



"A p value…indicates the proportion of *hypothetical* test statistics [derived from a ton of simulations of random samples from the same population assuming a certain effect size] that are equal to or [greater than the observed] test statistic".????(3/5) pic.twitter.com/78lue5snFa — Nick Byrd (@byrd_nick) September 14, 2019

How can we correct widespread misinterpretation of 'statistical significance' (besides Twitter threads)?



Cassidy et al. mention

– Fix the textbooks

– Remove discussions of the term from books

– Use their free teaching materials on @OSFramework: https://t.co/j5NU2oKkLQ



(5/5) ✌️ pic.twitter.com/YyYiLukrNf — Nick Byrd (@byrd_nick) September 14, 2019

So that’s it. It seems like a pretty big and complicated problem. But Cassidy and colleagues do a pretty good job of sorting out the problem, quantifying it, and offering some reasonable solutions. Now for the hard part: implementing the solutions. It’s going to take more than an academic paper and a Twitter thread to course correct our discussions of ‘statistical significance’ and p values. Indeed, it will take lots of action from lots of people—textbook authors, methodological instructors, etc. Solutions that require widespread action are often difficult to achieve—but collective action problems are a topic of another post for another day.

Related Posts