U.K. professors Matthew Goodwin and Eric Kaufmann wanted to, in their words, “help organize and join a public debate about immigration and ethnic change,” so they held an event in London titled “Is Rising Ethnic Diversity a Threat to the West?”

Goodwin and Kaufmann acknowledge it “was certainly a provocative title,” but when people got provoked and criticized them, it bothered the professors, and they went to the online magazine Quillette to air their grievances. “Rather than a genuine debate,” they write, the panel discussion “was interpreted as an open attack on immigrants and minorities.” An open letter signed by self-described anti-fascist activists accused them of framing the debate “within the terms of white supremacist discourse.”

The letter didn’t call for the debate to be canceled, but it still offended Goodwin and Kaufmann’s scientific sensibilities. A lot of people in the U.K. and other Western democracies feel threatened by immigration and rising diversity. No matter what one thinks of that feeling, it is politically potent, and therefore worth exploring. It’s wrong, Goodwin and Kaufmann argue, to oppose public debate and scientific inquiry on the grounds that some questions violate political correctness.

To determine whether their debate was actually offensive — and if so, to whom — Goodwin and Kaufmann conducted a survey. Respondents saw a poster advertising “Is Rising Ethnic Diversity a Threat to the West?” and rated how offensive they found it on a scale of 0–100. The average was 28, and there was no statistically significant difference between whites and minorities. The one subgroup that scored their offense over 50 self-identified as extremely liberal (1 on a standard scale of 1–7).

Evidently, this was another case of political correctness run amok, with far left ideologues claiming offense on behalf of racial minorities who did not find it offensive themselves.

That’s also the point Quillette editor in chief Claire Lehmann chose to highlight, sharing a graph of Goodwin and Kaufmann’s findings on social media.

“Offense,” Lehmann wrote, “was predicted by political ideology, not race.”

But that’s not really what the evidence shows.

Bad Social Science

The graph Lehmann shared, which also features in Goodwin and Kaufmann’s article, is a good example of bad social science.

The first red flag that jumped out at me is the gap between the red and blue lines, which differentiate the white and non-white respondents. No matter the political ideology, the gap between whites and non-whites is about the same. That’s not impossible, but it’s weird. The size of the racial gap among people who share a political ideology should vary, due to randomness if nothing else.

And white vs. non-white might not be a good way to measure it. The poster advertised an event on immigration, and there’s no reason to think non-whites’ opinions are interchangeable. A Briton whose ancestors arrived from India 150 years ago might think differently than someone who came from North Africa last year. Maybe blacks, Arabs, and east Asians have similar attitudes towards immigration and diversity, but maybe they don’t. And many immigrants living in the U.K. are whites from Eastern Europe, especially Poland.

The second red flag is the confidence intervals. Those are the vertical lines running through each dot, and they’re huge. According to statistics, there is a 95% chance that the answer falls somewhere along that line. For example, white moderate conservatives — red line, ideology 5 — fall between about negative 10 and positive 40. A range like that has little explanatory value.

The explanation for these red flags can be found in the text below the graph: “N=80 (41 white, 39 non-white).” That means there were 80 people in the survey, which is a small sample. Goodwin and Kaufmann divide those 80 people into two racial categories and six political ideologies. Assuming their respondents are evenly distributed among those 12 racial-ideological groups, there are just six or seven people in each category.

That’s the best case scenario. Since the authors don’t specify how many are in each group, it’s possible that one or two people represent the entire population of, say, non-white liberals (ideology 2). Maybe one racial group is evenly distributed, while the other clusters in one ideology. Smaller confidence intervals likely indicate the sample contains more self-identified moderates than either extreme.

There are other problems. R-squared, which ranges from 0–1, measures how well the statistical model fits the data, with 1 a perfect score. Goodwin and Kaufmann’s graph has a R-squared of 0.18, which isn’t disqualifying on its own, but indicates their model isn’t a good fit.

Additionally, the authors say they controlled for age, education, gender, and whether the respondents were British or American, which is the right thing to do. But it slices the sample even further — 48 subcategories for 80 people — to the point where some likely have one respondent, or none at all. Unsurprisingly, the authors find these controls aren’t statistically significant. As a result, they drop them from the model. That’s the wrong thing to do.

All of this is a complicated way of saying Goodwin and Kaufmann’s sample is way too small to test what they say they’re testing.

They find that extreme liberals are significantly more likely to take offense than any other group. I think this is probably accurate, if a bit circular — part of self-identifying as extremely liberal these days involves finding offense in questions like “is rising ethnic diversity a threat?” — but with such a small sample it could be a fluke. With 80 respondents and six political ideologies, a handful answering differently could render the result insignificant.

But the biggest problem is in the presentation. Goodwin and Kaufmann’s study, and Lehmann’s hook for their article, emphasize that race wasn’t statistically significant. It’s the crux of their argument. They present the lack of statistical significance as evidence that whites and non-whites are equally unoffended, which is necessary to claim that lefties took offense on behalf of minorities who were not upset.

Maybe that’s true and maybe it isn’t. But Goodwin and Kaufmann didn’t ask enough people to know.

Grievance, Bias, and Quillette

Mistakes like this are especially egregious coming from Quillette, which presents itself as a critic of bad social science.

The website regularly publishes critiques of academic disciplines that scholars James Lindsay, Helen Pluckrose, and Peter Boghossian labeled “grievance studies” — namely those that study race, gender, sexuality, and similar topics relating to identity. The running criticism is that these disciplines are driven by ideology rather than science. Because of political correctness and left-wing identity politics, they publish work that confirms their biases, shun evidence that undermines their pre-conceived beliefs, and try to stifle free inquiry. Quillette aims to provide a platform to counter that.

In a recent profile in Politico, Lehmann explained that she founded the website because: “I particularly wanted to criticize feminism, and I couldn’t get published in the Australian media if I was critical of feminism.”

In the last month, two Quillette pieces criticized the claim that transgender identity is innate, arguing that this denies evolution and creates a “new patriarchy.” Another recent article lamented that anyone who would “dare to discuss the science of sex differences” will get accused of sexual harassment. Many transgender people and people who’ve experienced sexual harassment find arguments like these offensive, because they declare their experiences and inner lives invalid. Quillette publishes a lot of them.

Goodwin and Kaufmann’s complaint about the criticism of their panel on immigration and diversity fits the same theme. They denounce it as an attempt to silence free inquiry:

To turn our backs on the scientific method is to yield to our biases, abandon the quest for objectivity and simply engage in a naked struggle for power.

But what is their race and ideology graph if not a yield to biases? Before conducting the survey, they believed their left-wing critics were unreasonably claiming offense on behalf of minorities, and then they declared that their study supports that claim, even though it doesn’t. At the very least, Goodwin, Kaufmann, Lehmann, and Quillette were insufficiently concerned with the possibility the study didn’t reach their preferred conclusion to catch the basic small sample size error.

I do not object to questions about immigration and diversity (though the obvious follow up — “threat to what?” — often leads to racist answers).

I do not object to those who criticized Goodwin and Kaufmann’s event (though critics who escalated to harassment were out of bounds).

I do not object to gender studies, feminism, or queer studies either (though like all social science disciplines, they sometimes produce work I find ridiculous).

But I do object to bad social science, especially from people who spend a lot of time denouncing others for privileging ideology over evidence.