Pundits and activists have long blamed the "mainstream media" for having an outsized effect on public perceptions. Whatever side of the political spectrum you're on, some people say, it seems as if large media outlets like the New York Times or FOX News exert too much power over the national conversation. Ideas from non-mainstream media, according to this logic, get drowned out. But a new long-term study reveals that small media outlets have a far greater effect on public discussions than anyone realized.

To be more precise, it only takes three or more stories from small news outlets covering the same topic to make discussions of that topic go up by 62.7 percent on Twitter.

It took a group of Harvard researchers five years to reach this conclusion. They did it by tracking the effects of stories covered by 48 small media outlets, measuring how they affected conversations on Twitter. Harvard political scientist Gary King and his colleagues explain in the journal Science that they honed in on 11 broad topics in public policy, ranging from refugees and race to food policy and domestic energy production.

"If we'd been conducting this study 100 years ago, we would have gone into town squares and listened to what people said on soap boxes," said King. "Today, it's Twitter."

The public sphere

To conduct their experiment, the researchers asked editors if they would be interested in publishing stories related to one of the policy topics. Journalists could write anything they wanted, from any political perspective, as long as it was relevant to the broad category of, say, food policy. "They would pick the [story] angle and we would approve it for the experiment, so there was total control on both sides," King said.

Next the researchers picked two random consecutive weeks expected to be "slow news" weeks. One week would be the experiment week, when the stories came out, and the other would be a control. The experiment took five years, King explained, because it was so hard to coordinate. If big news broke related to the topic, it would introduce too many confounding factors to measure the effects of their media outlets. They would have to scrap the experiment that week and start again at a different time.

Ethics were a huge issue for the researchers. They wanted to interfere as little as possible in the normal functioning of media organizations. "It was like the Prime Directive on Star Trek," King joked. But he was also deadly serious. He and his colleagues spent months working with news organizations to be sure the only thing that was different was when the stories ran, not what the stories were. (For a fascinating account of the study design and ethical issues, see the team's Supplementary Material.)

The hardest part was analyzing the impact these stories had. Using an algorithm from data analytics company Crimson Hexagon, the group mined Twitter for every public tweet that might be relevant to the topic during the experiment and control weeks. Working with a team of researchers, many of them Harvard undergrads, the group combed through those two weeks' worth of tweets to be sure they were actually relevant. In other words, they wanted tweets about food policy, not just people writing, "food is yummy nom nom nom."

Study co-author Ariel White, now a political scientist at MIT, was one of the first people to analyze the huge volume of tweets for relevance. She said it was a good lesson in how the media can sway public conversation. Though the study revealed little about people's personal beliefs, it uncovered their "participatory opinion, their willingness to talk to each other in the public sphere." Today's policymakers are watching Twitter to see what people are talking about, she added, and therefore people's willingness to tweet about a topic becomes politically significant.

Fake news

White also learned some unexpected lessons. When she first started the project over five years ago, she wasn't a Twitter user yet, and she was overwhelmed by the "offensiveness" of what she found. She needed to check in on the undergrads analyzing tweets to make sure they "weren’t getting overloaded by some of the awful stuff that people write on social media."

After years of this occasionally harrowing data analysis, King and White determined that it only took three stories in small outlets to boost public discussion of a given topic by 62.7 percent on Twitter. The same number of stories also swayed public opinion by about 2.3 percent in the direction of the stories' ideological positions. So media may not be able to change our opinions very much, but it can certainly change what we're talking about.

This study also sheds some light on the fake news phenomenon on social media. "One thing we solidly measure is the ability of the media to set the agenda," White said. Small outlets publishing fake news could be doing that, too. "The things [the media] talk about are what we as Americans talk about," White concluded—for better and for worse.

Science, 2017. DOI: 10.1126/science.aao1100