How To Fudge Your Findings

The Wall Street Journal has an article today analyzing the success of school voucher programs in Milwaukee. Whether or not you are personally interested in school voucher programs in Milwaukee, the WSJ’s article is worth reading as an example of journalistic technique. Specifically, it shows you what to do if you need to report facts you find ideologically uncomfortable.

The WSJ’s analysis of the Milwaukee program yielded discouraging findings for voucher advocates. On average, voucher students performed the same or possibly slightly worse than public school students on state exams, even though voucher students appear to be slightly less poor than public school students. (82% of public school students qualify for reduced-priced lunches versus 79% of voucher students.) While the performance difference was so tiny that it might be statistically negligible (I am not sure, because not much detail is offered on how the numbers were obtained), since these students are less poor (and being less poor will tend to mean fewer other obstacles to academic success), this does suggest that taking a voucher and going to a private school may be, on average, a worse decision than simply staying in public school. The analysis also found that the higher a school’s percentage of voucher students, the worse it did, and that “the bulk of those with the highest voucher enrollments fall in the bottom quartiles of private schools for results on state exams.” If a Christian academy—most of the voucher schools are religious—opens up to replace a public school, and we transfer all of the students from the public system there and shut the public school, with the academy now consisting entirely of former public school students on vouchers, we will expect these students to do worse than they would have in the public system. Because public school enrollment in Milwaukee is steadily decreasing, and voucher enrollment is steadily increasing, these findings should be troubling.

The WSJ’s Milwaukee findings should caution against any attempt to simply voucherize the public school system. And yet, have a look at how the paper chose to frame its presentation of those findings. The headline is: “Do School Vouchers Work? Milwaukee’s Experiment Suggests An Answer.” The answer it suggests, of course, is “No,” if “work” is taken to mean “improve student performance.” But the WSJ doesn’t say that. Instead, the subheadline is “A Wall Street Journal analysis of the school system’s 27-year-old program suggests the concept works best when private institutions limit the number of public students.” This is, journalism students take note, what is known in the industry as “burying the lede.” The question is: how do vouchers perform when compared with public schools? The answer is “on average, either the same or a smidge worse.” But what the Journal leads with is the answer to a different question, namely “Well, which kinds of voucher programs do better than average?” (The print version—see above—is even sneakier, by the way: “Do vouchers work? Look to Milwaukee.” If you want to try constructing one of these yourself, remember to use “look to” rather than “look at.”)

Subtle framing choices are made, then, that help massage results that are non-promising for vouchers. Don’t focus on whether they do well or not, focus on what factors are present when they do well, thereby directing your audience’s attention toward successful voucherized schools rather than unsuccessful ones. A journalist could reply, of course, that this is not necessarily biased: they just think it’s productive to see what good voucher-schools are doing well. But here we should note another important fact about the WSJ’s analysis: it talks about above-average voucher schools, yet not about above-average public schools. The writer wants to convince us that voucher programs can do well (“The Milwaukee findings offer a potential road map for the Trump administration”) by showing us what a voucher program needs to do in order to outperform the average (i.e. ensure that voucher students do not make up the majority of the student body). There are public schools that outperform the average, too. Yet we don’t hear them cited as evidence that a public school system can work. Instead of comparing above-average voucher schools to above-average public schools, the writer compares above-average voucher schools to average public schools. In doing so, they imply that a well-designed voucher program can beat the public school system, without examining whether a well-designed public school system could achieve the same results.

There are all kinds of questions raised by the WSJ’s voucher analysis. Consider the main conclusion: for a school that accepts vouchers to outperform the public school average, the key factor is its percentage of voucher students. If most of the students are on vouchers, the school will probably do very badly. If voucher students are a small minority in an ordinary private school, they will probably do better than average. This suggests that vouchers are only succeeding in cases where the voucher students reap the benefit of attending a wealthy school where there are rich students to pay for the resources that will be used by the poor students. Indeed, the article gives an example of a high school where tuition is $12,400 a year, which is $5,000 more than the voucher. The school cuts its tuition for the voucher students, but it doesn’t cut the amount of education it gives the voucher students. That means that the students are being subsidized. They are getting an expensive education at a discount price. Private schools are only willing to extend that subsidy so far, and the school with the $12,500 tuition caps voucher enrollment at 12.5% so that it doesn’t give too much fancy education away for free.

This means the following: a good way for voucher students to beat their public school counterparts is by receiving subsidies beyond the voucher. If you have a voucher and you are among the small number of students who get admitted to a wealthy private school willing to give you more than your voucher’s-worth, you will probably do better than you would have at a public school. But of course, at a public school there wouldn’t have been a massive additional subsidy coming from rich people. This means that what looks like a public-private question may actually just be a funding question: if Milwaukee public schools had the same resources as the wealthy private schools with small numbers of voucher students, how would they do? (We don’t know, from what the Journal presents, how much subsidization is responsible for the performance gains. That means it’s hard to say whether a school could improve performance of voucher students by limiting their voucher enrollment, or whether the ones with limited enrollment just happen to be the wealthiest.)

To better understand what the Wall Street Journal found, let’s imagine we were talking about colleges. The equivalent to the existing system would be a world in which Bernie Sanders’ free college program had been fully implemented, so that college functioned like elementary through high school: you didn’t have to pay tuition, you could just enroll in a public college like you can enroll in elementary, middle, or high school. There would still be private colleges. Some of these would be exclusive and spend a lot on their students, like Stanford, and some would not be as exclusive and would not spend as much on their students, like the University of Phoenix. The free state colleges are a “public option” but if you have money, or can get a scholarship, you can opt out of them, just like you can with private high schools.

Okay, so in our “free public colleges” world, someone comes along and introduces a voucher program. Students can go to the free public college. Or they can take a tuition voucher and go to one of the private colleges, e.g. Stanford/Phoenix. In the parallel with contemporary politics, this program would be pushed by conservatives who despise the free college system, because it reduces choice and competition by giving students only one free option. They want poor students to be able to go anywhere they like. Ultimately, they would like to eliminate public colleges altogether, because they don’t believe government should be in this business.

Here are the Milwaukee findings, adapted to Free Public College World: students who took their vouchers to expensive private colleges, which admit a small number of voucher students, do very well. Students who took their vouchers and go to programs that consist mostly of voucher students do less well. On the whole, voucher students do no better than students at Sanders State. The Wall Street Journal comes along, tells us that students who use vouchers at expensive private colleges do very well, and then recommends we scale up this model and voucherize the whole system.

We can see instantly what the problem here is. Stanford, for example, spends about $70,000 in educational resources per student. The University of Phoenix spends a pittance. The voucher programs will only work for the students who manage to take their vouchers to fancy colleges. But those schools will necessarily want to limit the number of students to whom they give colossal subsidies. The model can’t be scaled up much because private schools are only going to want to allow a small number of voucher students to “piggyback” off the rich kids. (Indeed, the WSJ article quotes a private school official saying parents were wary of subsidizing voucher kids.) Large numbers of voucher students may well end up at our University of Phoenix equivalent. What would happen if we were to voucherize our whole system? We don’t know, but it’s possible that students would tend to end up in the crappy majority-voucher colleges way more than the high-spending elite ones with enrollment caps. Yet when these same results are found at the high school level, they are presented as a “road map” for voucherization under Trump.

This has always been my concern about school privatization schemes: I fear a world in which perfectly decent public schools are sapped of funding, with students given vouchers that leave them in worse private alternatives. My dystopian world is one in which corporations decide to open private schools that offer nothing but job training, public schools are steadily defunded, and students get shunted off to FedEx Academy. (It may be hard to resist, if it comes with the prospect of a job at the end. And libertarians like Bryan Caplan have suggested that school should be job training instead of useless rubbish like “literature” and “the arts.”) Actually, my true dystopia may already exist: the “online charter school” where kids sit in a room with a computer all day and learn literally nothing.

Meanwhile, we may be overlooking ways that public schools could be improved. We know what an above-average voucher-taking school does (i.e. allow poor kids to use the rich kids’ resources). But what does an above-average public school do? The fact that the Wall Street Journal is uninterested in this shows you that their discussion of vouchers is ideologically rather than empirically driven. Vouchers are not being pushed because there is robust evidence that they help students. They are pushed because public schools are socialism, and socialism is seen as necessarily worse than the free market because Choice And Competition make everything better. That’s why you’ll only ever see conservatives answer the question “Do vouchers work?” with “Here’s what happens when vouchers work.”

If you appreciate our work, please consider making a donation or purchasing a subscription. Current Affairs is not for profit and carries no outside advertising. We are an independent media institution funded entirely by subscribers and small donors, and we depend on you in order to continue to produce high-quality work.