Last month I argued that a news article misrepresented the feelings of economists on school vouchers. I got a bit of pushback from people who thought I had just misread it, and that it wasn’t deceptive at all (1, 2, 3), and wrote an addendum post sticking to my guns.

One of my New Years resolutions is to be more empirical, so I thought this would be a good opportunity for an experiment. I offered to bet at 10:1 odds in the other person’s favor that most people shown the article would believe economists were against school vouchers. Noah Smith originally took me up on the bet, but later stopped cooperating with me about it and wouldn’t give a straight answer to the question of whether he was retracting his offer. Vikram Sasi agreed to continue the bet; I’m not sure if he disagreed with my assessment, thought the odds were too good to refuse, or just wanted the experiment to go ahead.

I proposed that we show some random people the article, then ask them a few easy comprehension questions to make sure they were putting in a decent effort. Then we would ask the following:

According to this article:

a) Most economists support privatizing education

b) Most economists oppose privatizing education

c) Unsure / the article doesn’t say

The true answer was (c) – the article doesn’t give enough information to tell you what economists think (which turns out to be mostly unsure, with more supporting the privatization than opposing). But I thought the article heavily (and falsely) implied b, and expected most people to give that answer.

Before Vikram and I could agree on exactly how we were going to do this, we got scooped by two other people who did the experiment without asking us.

Commenter Sripada got 40 people to take the survey on MTurk, of whom 35 answered the easy questions right and qualified for inclusion. Of those 35, 32 of them (91%) answered (b), misinterpreting the article to claim a consensus against privatizing education. Only one of them (3%) gave the correct answer (c), accurately noting that the article doesn’t tell us anything about an economic consensus for or against.

Hanne Watkins, a real research psychologist, also did a 50 person MTurk survey. I don’t have enough access to her data to limit the sample to people who answered the easy questions right, but among all participants 78% believed that most economists opposed privatization, compared to only 10% who correctly noted that the article didn’t say.

Vikram graciously accepted these studies as settling the bet, and since more than half of people misinterpreted the article in both, he paid me my $10.

I’m really happy about this, not just because I won ten dollars but because I feel like it was a rare example of getting to decisively settle a disagreement. Instead of arguing about whether or not the article was misleading, we figured out a way to test it, did the test, and now we know.

I should add that some people might think the article was non-misleading in a different way than I did that can’t be settled empirically (Noah seems to be in this group), and other people might think the particular set of questions the survey asked was unfair or biased. This is why I wish I’d gotten to bet somebody who I knew really disagreed with me about this, so at least that disagreement could have been settled for sure. As a consolation prize, I will take my ten dollars and my increased certainty that the article didn’t accurately convey economists’ thoughts.

Thanks to Vikram, Sripada, and Hanne for their help with this.