Facebook’s efforts to reduce misinformation in its news feed since the 2016 election have opened the company to all manner of criticism, including allegations of political bias from both left and right. But a new study from researchers at Stanford University, New York University, and Microsoft Research suggests they might actually be working—at least, to some extent.

The study, released as a working paper Friday afternoon, examines how Facebook and Twitter users interacted with articles from 570 sites that have been identified by at least one credible source as a purveyor of “fake news”—that is, patently false, intentionally misleading, or hyperpartisan content. It finds that engagement on stories from those sites rose steadily on both Facebook and Twitter until shortly after the 2016 U.S. presidential election. Beginning in early 2017, however, those sites’ engagement began to drop off on Facebook—even as it kept rising on Twitter.

While the authors caution that the study is “far from definitive,” it’s noteworthy as perhaps the first large-scale empirical study that directly examines the efficacy of Facebook’s ongoing campaign against misinformation. Its findings could serve as a guidepost as the company continues to reckon with its influence on civil society.

The conclusion that Facebook’s efforts are working requires a number of caveats, co-author and Stanford economist Matthew Gentzkow told me in a phone interview Friday. Those include the fact that misinformation on Facebook remains widespread—far more so than on Twitter—and that the study’s list of “fake news” sites is neither exhaustive nor authoritative. Still, the new analysis points to a strikingly clear trend line, one that should encourage those tempted to view the battle against online misinformation as unwinnable.

In all, Facebook engagement—the sum of likes, shares, and comments—on articles by the “fake news” sites in the study has declined more than 50 percent between the 2016 election and July 2018, the authors conclude, using data tracked by the content marketing firm BuzzSumo. That result stands out when compared to the trends on Twitter, where retweets of the same sites rose significantly over the same time period.

That makes sense when you consider that Facebook has made a concerted effort to limit misinformation on its platform since early 2017, whereas Twitter has not. (Twitter’s enforcement efforts have focused on bots, spam, harassment, and hate speech, not false news stories.) This indicates that Facebook’s apparent gains are not simply a result of fake news sites giving up, winding down their operations, or changing their domain names.

“While this evidence is far from definitive,” the authors write, “we see it as consistent with the view that the overall magnitude of the misinformation problem may have declined, at least temporarily, and that efforts by Facebook following the 2016 election to limit the diffusion of misinformation may have had a meaningful impact.”

The study, which has not been peer-reviewed, was conducted by economists Gentzkow and Chuan Yu of Stanford and Hunt Alcott of New York University and Microsoft Research. Facebook did not fund or participate in the study.

“It’s an amazing—and somewhat surprising—result,” said Dipayan Ghosh, a fellow at Harvard University’s Shorenstein Center on Media, Politics, and Public Policy, when I contacted him to discuss the study. “The researchers have done the diligent work of drawing on data from multiple reputed sources and conducting well-reasoned analysis.”

It’s an amazing—and somewhat surprising—result. — Dipayan Ghosh

Ghosh, who worked on privacy and public policy at Facebook before leaving in the wake of the 2016 election, has since emerged as a critical voice on the problem of social media misinformation. In January 2018, he wrote in Time that disinformation was “becoming unstoppable,” a claim that the study’s authors cited as part of the motivation for their inquiry. Ghosh told me on Friday that his takeaway from the new paper is that Facebook’s detection and suppression operations appear to be having at least some positive impact, after all. If true, he added, that “bodes well for the long-term impact of disinformation operations, at least on Facebook.”

It would be a mistake, however, to conclude that Facebook has solved its misinformation problem, or that Twitter’s is worse. According to the study’s methodology, the scale of the problem on Facebook was much greater to begin with, and remains much greater today. The ratio of Facebook engagements to Twitter retweets on the 570 sites has fallen from about 40-to-1 to about 15-to-1, which implies that Facebook is still by far the larger fake news source. And the sites included in the study are still racking up some 70 million Facebook engagements per month. That’s far less than the 200 million per month they were receiving at the height of the 2016 election, but it’s still an awful lot.

For context, the authors compare engagement on stories from the “fake news” sites with that on stories from a list of 38 large, established news sites. In 2016, the 570 misinformation sites in the study were receiving about as much Facebook engagement as the 38 mainstream news sources. They’re now receiving less than half as much Facebook engagement, as the large news sites’ engagement has held relatively steady.

A second major caveat is that the definition of “fake news” is fuzzy, and the one used by this study is decidedly unscientific, Gentzkow acknowledged. The authors assembled their list of 570 sites by combining all the sites identified as purveyors of fake or misleading news sources in a set of five previous, mostly informal analyses by various sources. Some of the lists in those previous studies have been controversial, including one compiled in 2016 by a Merrimack College professor that lumped in in established partisan politics sites such as Breitbart and Red State with outright hoax sites.

Somewhat reassuringly, Gentzkow said that the authors’ analysis suggested that the “worst actors”—sites identified as misinformation sources on multiple previous lists, rather than just one—have seen the greatest drop-off in Facebook engagement. The authors tested various ways of slicing the data set in an online appendix to their paper, which is well worth diving into for those interested.

One limitation that the study could not address: the likelihood that important new sources of misinformation have cropped up since 2016, including some designed to evade Facebook’s crackdown. The sites in the study were mostly known to Facebook at the time it began its anti-misinformation projects, so it shouldn’t have required rocket science for Facebook to limit their news feed reach. The real test is whether Facebook can keep up with the evolving techniques of determined propagandists and trolls. Gentzkow said his study can’t tell us that.

A major caveat is that the definition of “fake news” is fuzzy.

A Facebook representative on Friday welcomed the study as “encouraging,” while acknowledging that further study is needed. Communications manager Lauren Svensson pointed out that the study looked at engagement on all articles from a given website, whereas Facebook has focused much of its effort on limiting specific stories that its users or third-party fact-checkers have flagged as false. In other words, Facebook isn’t trying to throttle all the content from the sites examined in the study, so we shouldn’t expect their content to disappear from the news feed.

Overall, Svensson said, the progress shown in the study “is commensurate with what we’ve been seeing. But we know we have more work to do, as always.”

Asked for comment on the study, a Twitter representative noted that the company has deliberately avoided setting itself up as an “arbiter of truth.” Twitter’s stance has been that the site itself can function as an antidote to the spread of misinformation, because its public nature allows false claims to be debunked. It’s a stance that Vijaya Gadde, the company’s lead for legal, public policy, trust and safety, articulated on a recent episode of Slate’s If Then podcast. (This approach was called into question by a March study that found lies travel faster on Twitter than the truth.)

It would be premature to conclude from this study that Facebook is winning the war against fake news. Even if there were a widely agreed-upon definition of misinformation—which there certainly isn’t—more research would be needed to establish that Facebook is stemming its flow. But the study’s findings suggest that, at the very least, Facebook’s efforts are having a significant impact on their intended targets.