When choosing the article to apply the Breaking News Label to, publishers will naturally tend towards articles they consider to be of high interest to their audience. This introduces selection bias, since these articles are not equally drawn from the population of all articles, but rather from a subset of high-potential articles. To avoid our results being distorted by this selection bias, we needed to identify shares for which the Breaking News Label has not been used but which are still representative of the kind of high-interest article a publisher might have used the label for.

To do this, we used our AI’s prediction of how well each article was likely to perform on social media, based on a large set of various factors. Combining this prediction and the actual performance after being shared allowed us to measure by how much each share outperformed our initial prediction. Figure 1 visualises this for shares with the Breaking News Label (orange) and those without it (blue). We observe that on average articles with the Breaking News Label outperform our initial predictions by a far greater degree than articles without the label. On average, shares with the Breaking News Label outperform our AI’s initial prediction of the share’s performance by 51%.

Figure 1: Outperformance comparison of shares with Breaking News Label vs. those without.

To test whether this increase could have occurred randomly, we drew 3,000 samples of 200 shares each from all shares without the Breaking News Label and measured how often a random sample would exhibit a comparable difference in performance. The observed variance in the data indicates a probability of less than 2% that the Breaking News Label posts outperformed the other posts by chance. This is strong evidence that individual shares with the label do indeed generate more clicks than they would have otherwise.

However, we do not find any impact on overall traffic. If shares labelled as breaking news benefited without any negative impact on other posts, a medium-sized publisher, with 25 shares to Facebook per day, could theoretically expect a ~4% increase in monthly social traffic by using the full monthly quota of Breaking News Labels. However, the same publishers for which we just showed the significant positive impact of the Breaking News Label on individual share performance actually saw an overall decrease of 2.5% for clicks, similarly for reshares, likes and comments after they started using the Breaking News Label (Figure 2). Additionally, when considering other publishers, we have found no evidence supporting the expectation that the Breaking News Label benefits overall page performance.