I.

A sample of Thursday’s talk at Yale:

These are four headlines describing the same study, Milkie, Nomaguchi and Denny (2015). The study found that of twenty or so outcomes, only three of them – all measuring delinquent behavior among teenagers – show significant effect from time spent with parents (and this result remains after Bonferroni correction). So Vox has a great argument for their headline. The National Post has an okay argument for their headline even though it’s kind of cherry-picked. The Washington Post just sort of reads between the lines and figures that if it’s not quantity of time that helps kids, it must be quality. And FOX also reads between the lines and figures that if moms spending time with their kids has no effect, the argument from opportunity costs suggests mothers are spending too much time with their kids.

None of them are completely outright lying. And indeed, most of the articles eventually explain what I just said, halfway down the article, in one or two short sentences that most readers will skim over. But the rest of the article uses the study to support whatever the news source involved wants it to support, and so people will come up with four diametrically opposed conclusions from this one study depending on which source they read.

II.

Here’s a study that I wasn’t able to include in the presentation because it just came out recently. As per the Rice University press release: Overweight Men Just As Likely As Overweight Women To Face Discrimination.

The paper included two studies. In the first, men went into stores either with or without fat suits and try to do some things – ask if there were job openings, ask for a job application, ask an employee for help, try to buy some things, et cetera. Then they measured the men’s success across both conditions to see if they had more trouble when they appeared overweight.

In the second, subjects were asked to rate videos of an employee giving a marketing spiel for a new product; once again, the employee was either wearing or not wearing a fat suit. They measured the subjects’ ratings across conditions to see if they ranked the overweight employees lower.

The first study only included men, and so could not possibly have determined whether men were more or less likely than overweight women to face discrimination. The second study actually did have both male and female employees involved, and although it really wasn’t their main interest, the researchers did a post hoc evaluation to find the effect in each sex. In all three of the outcomes where discrimination was found, women faced more discrimination than men. They didn’t significance-test the comparison, but just from eyeballing it, it was probably significant.

So a paper in which one study does not compare men to women, and the other study finds women facing more discrimination than men, the press release somehow gets phrased as “Overweight men just as likely as overweight women to face discrimination in retail settings”. Huh.

You might wonder, “Does it really matter what a press release says? Does anyone read the exact wording?” Yes. Many other news sources copied the phrasing, for example Medical Daily’s Fat Discrimination Is The Same Regardless Of Gender. One such copycat, PsyPost.com copied the press release nearly word for word, including the title. Then it got posted on Reddit and now has 5189 upvotes and 1572 comments. So there’s that.

III.

“But at least it correctly raised awareness of how weight discrimination is a big problem in the retail setting, right?”

The paper measured a ton of different outcomes. Let’s focus on Study 1. The actor in the fat suit was supposed to ask if there were job openings (there were) and see if the company told him. Then he was supposed to ask for an application form and see if they gave it to him. Then he was supposed to walk in as a customer and see if employees greeted him. Then he was supposed to ask the employees to recommend him an item and see if they did. Then he was supposed to ask them to recommend him a second item and see if they did.

No difference was found between overweight and normal-weight actors in any of those five experiments. Two of them had ceiling effects that probably made the attempt futile, but the other three didn’t, and there wasn’t even a trend toward discriminating against the overweight guy.

So what did they find discrimination on? They say that detected “interpersonal discrimination”, ie discrimination based not on any quantifiable outcome but based on how friendly/warm the person interacting with the actor seemed toward him. They determined this by self-rating and other-rating; that is, the actor wrote down how friendly he thought the store clerks were toward him, and a spy surreptitious observer who had placed herself near the interaction also rated this for corroboration. Their rating scales included twelve items including “how many times did the clerk nod”, “how friendly did the clerk seem?”, “how much eye contact was the clerk making?” and “how much comfort level did the clerk seem to have?”. The experiment found a statistically significant difference between the fat-suit-wearing and non-fat-suit-wearing trials and concluded that there was interpersonal discrimination.

But hold on a second! The study says nothing about anyone being blinded. In fact, it’s really hard to blind an actor to the fact that he is going into some stores while wearing a fat suit and other stores while not wearing a fat suit. As far as I can tell, everybody involved was in on the study from the beginning. If your boss tells you “I want you to rate how much comfort level clerks have with you for this study on fat discrimination”, it seems really possible to me that there might be a slight tendency to overrate the clerks who interacted with thin-you, and to underrate the clerks who interacted with fat-you.

How slight a tendency? Clerks dealing with fat people got an average rating of 2.3 (seven point scale, lower is better), and those dealing with thin people of 2.0.

So after finding no discrimination on five objectively measurable outcomes, they find very subtle discrimination on an unblinded subjective outcome practically designed to produce placebo effects.

We move on to the second study, where participants (as usual, psychology students) are rating video presentations given by fat vs. thin people. This is supposedly tying into the “retail industry” theme of the paper, but honestly it seems kind of forced to me.

Anyway, the participants are asked to rate their presenters on seven measures: overall quality of presentation, overall attitude toward product being presented, overall attitude toward the store that would employ a person such as this, intention to support the store, employee’s appearance, employee’s carelessness, and employee’s professionalism. The results:

There was no difference between how participants rated overweight vs. normal-weight presentations overall.

There was no difference between how participants rated products presented by overweight vs. normal-weight people.

There was no difference between how participants rated stores staffed by overweight vs. normal-weight people.

There was no difference between how likely participants were to support stores staffed by overweight vs. normal-weight people.

There was a difference in how participants ranked the appearance, carelessness, and professionalism of overweight vs. normal-weight people.

The first four results are encouraging. What about the last three?

Well, I feel like if you ask people to rank someone based on “their appearance”, and your subjects answer based on how they look, you kind of walked into that one. Oh no, people rank conventionally attractive people as having better appearances than less conventionally attractive people! Someone call John Ioannidis to double-check this astonishing result!

“Carelessness” and “professionalism” are perhaps less excusable, but c’mon, you had them watch a two-minute video. When you give someone zero information on a thing, and you force them to make a judgment on the thing, then yes, stereotypes are their best source of information. If you showed me a picture of an average-looking man and an average-looking woman and say “Quick! Which of these people is more likely to like baking cupcakes?!” I’ll pick the woman, not because I think all women are obsessed with cupcakes or because I go around looking at every woman I see as a cupcake factory, but because you asked me a stupid question and ensured stereotypes were the only thing I had to go on.

Then the authors find that this was mediated by explicitly-expressed stereotypes against fat people, which is kind of interesting, but doesn’t make the nonsignificant things any more significant.

So to sum up: there was no discrimination against the overweight on any objective measure of the actual retail experience, including positions advertised, applications given, greetings offered, or customers served. There was also no discrimination against the overweight on presentation evaluations in terms of overall evaluation, evaluation of employee, evaluation of product, or evaluation of company.

There was a tiny amount of discrimination on a subjective measure rated by unblinded observers aware of the purpose of the study. There was also some evidence on three subtler ratings of the presentation that seemed designed to ask participants impossible questions in order to force them to stereotype. However, these meaningless scales did not effect the raters’ overall impressions as measured any of four different ways.

IV.

Here is the reporting from the news outlets that passed their first test and didn’t frame it as men and women facing equal amounts of weight discrimination.

Business Insider: Researchers Had Men Pretend To Be Obese – And The Results Are Disturbing, which says that “this research highlights the importance of including men in discussions about weight stigmatization,” and “the authors also advocate organizational efforts to combat negativity against heavy customers and potential employees…the first step may be for individuals to become aware of how strong weight biases are.”

AskMen: Young Men Who Appear Overweight Suffer Interpersonal Discrimination. “Researchers disguised six thin young men as obese customers or job applications and found that they were victims of microaggressions. Basically people were a little bit more jerky towards them.”

Oximity: Overweight Men Often Snubbed At The Mall. “Shopping malls can be hostile places for overweight men, regardless of whether they’re customers or simply looking for a job.”

The Health Site: Men, Here’s One More Reason For You To Lose Weight. “Ruggs said that these findings were another reminder that there was still more work to be done in terms of creating equitable workplaces for all employees, potential employees and consumers. She concluded that this was something organisations could take an active role in, and said that companies could do better job training on customer relations as part of the employees’ new-hire process. ”

Now I’m almost missing the kind of random scattershot media bias we found on the time-spent-with-children study. Here every media outlet reports the results the same way that the study’s author and the press release reports the results.

This is not a totally wrong interpretation, any more than “six hours a week will tame your teen” is a totally wrong interpretation of the childhood study. But if I myself were writing an article on this study, it would be SURPRISINGLY LITTLE DISCRIMINATION FOUND AGAINST OVERWEIGHT MEN, and mention somewhere in the middle that some discrimination was found on a few sketchy variables. Instead, we get DAILY DISCRIMINATION AGAINST OVERWEIGHT MEN and WE NEED TO INSTITUTE SENSITIVITY TRAINING PROGRAMS IN RETAIL ENVIRONMENT, and they mention somewhere in the middle that a lot of important variables came out negative.

A lot of studies work like this. You test ten or twenty complicated variables, you get positive results on some, negative results on others, some of those results seem plausible, other results seem like maybe you made a mistake somewhere or didn’t have enough power or whatever, and then you make an interpretation based on your personal bias. Then it goes from the researcher’s personal bias to the abstract to the press release to the headlines to the mind of the average reader, dropping subtlety at each step, until “No discrimination against overweight men, except where the study was practically designed to ensure false positives” becomes “Rampant discrimination against overweight men everywhere” becomes “Overweight men are discriminated against just as much as overweight women.”

Don’t get me wrong. I expect there probably is lots of discrimination against overweight men. And I think this study’s project of trying to find it and convince people of its existence was worthwhile. But I don’t think you should get to convince everyone that science has proven X, unless science has actually proven X. The process that produced these headlines is strong enough to produce any headline you want, with the part where you actually do the study becoming more and more of a ritual or a formality. There are just too many degrees of freedom between the study and the reporting.

Stalin once said that “those who vote decide nothing; those who count the votes decide everything.” It’s starting to look like those who do the studies decide nothing and those who report the studies decide everything. The only solution is to actually read the study and not just the headlines. Sometimes we might even have to – God help us – read beyond the abstract.