Many replication initiatives are about removing the weeds from the scientific record. Uhlmann’s effort—the Pipeline Project—ensures that only flowers bloom in the first place. “The idea was to see if findings are robust before they find their way into the media and into everyone’s lectures,” he says.

Replicating studies before publication could also lead to fewer bruised egos, and less drama. “It’s less sensitive when something fails before publication. No one’s even heard of the effect yet. My reputation isn’t riding on it. There’s less defensiveness.”

“Having researchers replicate each others’ findings before they are published is likely to be a critical step in fostering meta-science’s goal of turning the lens of science onto itself,” says Jonathan Schooler from the University of California, Santa Barbara.

At first, Uhlmann thought he’d do a study swap with just one other group. “I sent out 15 emails, thinking that one lab would say yes,” he says. “I got ten yeses.” He eventually recruited 25 teams across six countries, all of whom gave time and effort in exchange for minimal rewards. This unexpected altruism, he says, shows that “there’s a real enthusiasm for experimenting with better ways of doing science.”

The teams replicated ten of Uhlmann’s experiments many times over. All the studies focused on moral judgments, and our tendency to evaluate actions based on what they reveal about a person’s character, rather than how bad they are in absolute terms. For example, Uhlmann found that a manager who mistreats ethnic minority employees is seen as worse than one who mistreats everyone. A company that airbrushes a model to have perfect skin is seen as more dishonest than one that hires a model whose skin is already perfect. An animal rights activist who is caught hunting is seen as more immoral than a big game hunter.

These findings all checked out, along with three others. But four of Uhlmann’s results stumbled in the replication gauntlet in at least one crucial way. “I couldn’t predict which ones would work and which ones would not,” he says.

For example, he found that a company that doesn’t respond to accusations of misconduct is judged just as harshly as one that’s found guilty—but his replicators found that the silent party is judged five times more harshly. He found that a person who tips in pennies is judged more negatively than one who leaves the same tip in bills—and although his American collaborators found the same thing, those from other countries didn’t.

Uhlmann specifically designed the Pipeline Project to pre-emptively counteract criticisms that were levied at earlier initiatives. Critics have often said that replicators are too incompetent to successfully repeat their experiments—but Uhlmann chose accomplished peers who worked in the same field. “It’s hard for me to argue that someone’s not a competent researcher because I picked them!” he says.