"Dark social" could become less of a problem as Facebook is working on a fix. Flickr CC/Christopher So-called "dark traffic" is a huge problem for publishers and website owners.

It's useful to be able to look at your analytics, but when a huge proportion of referral traffic is listed as "direct" (and aside from a homepage, it's very unlikely users are typing full URLs into their browser to go directly to a page) it's difficult to know where to focus to boost your audience.

Now new research from the analytics firm Chartbeat, as well as confirmation from major publishers, shows that Facebook's mobile apps are largely responsible for the swathes of dark traffic being directed toward websites. We already knew Facebook was the principal source of social referral traffic for most digital publishers. But Facebook has been underselling itself.

Fortunately for publishers and website owners, it appears that Facebook is working on a fix.

The term "dark social" was coined by Alexis C. Madrigal, a senior editor at The Atlantic, who found that more than half of the website’s referral traffic was seemingly untraceable. Dark traffic refers mostly to when links have been shared via an online chat, email, or app rather than through a browser or a specific social app through which referrals can be easily tracked. Chartbeat has found in some cases dark social account for 65% of a website's traffic, averaging at about a third across its network.

Most analytics platforms (Google Analytics, Adobe Omniture, Chartbeat) have a huge section in their reports called "direct," which counts all those apps that users use to click on links "direct to a site." Facebook is one of them — alongside sources like Reddit, Gmail, and IM apps. Twitter, on the other hand, has a special "t.co" URL shortener, which means Twitter referrals are always easily trackable.

The problem with Facebook not using a shortener like this is that a huge proportion of its users access the site via only the mobile app. We know 703 million people visit the site via their mobiles daily, and presumably the majority of those do so via the app. That's a massive wave of referral traffic going unidentified.

Chartbeat, which this week began tracking Facebook users more effectively using a different method (more on that later), has published a blog post that clearly marks out the Facebook/dark social impact.

Last month the analytics company began to look at time series data for specific articles to identify patterns in traffic and correlate that with the "dark social" number.

Take a look at the chart below. The dark traffic spikes at the same time the article began going big on Reddit, and then again when it started picking up pace on Facebook.

Madrigal wrote in a new blog post published this week that he too conducted a test earlier this year. He set up an Atlantic post "deep in the sand of time" so only people who were part of the test would find it. He then posted a link to the story on his Facebook page and told his friends to click to it only if they were using the mobile app. He then looked at the referrers (of which he knew 100% were coming from Facebook). A smattering of people showed up on his Adobe Omniture analytics report as coming from Facebook.com, but the rest showed up as "type/bookmarked."

For the most part, those who had been clicking on their mobiles had gone untraced:

Figuring that those numbers would be tightly correlated with the overall number of Facebook mobile visitors, Madrigal compared Facebook mobile referrals with dark social.

The correlation was pretty clear.

As a result of its tests, Chartbeat has started looking more closely at the "user agent," a tag of code users leave when they visit a website that identifies the type of browser they use and their operating system. It also usually reveals some information about where they originated from. Most analytics systems don't dig deeply into this type of information.

Some publishers have developed methods to dig further into their own website analytics to uncover the "user agent," which reveals Facebook as the source of the direct referral traffic. Both The Guardian and BuzzFeed said on Twitter on Thursday night they had been digging further into user agent data for a while.

But that takes both time and expertise to set up that most publishers don’t have.

Trinity Mirror (owner of the UK newspapers The Daily Mirror and The People) digital director Malcolm Coles (who is soon to take up the same role at the Telegraph Media Group, owner of The Daily Telegraph) told Business Insider that analytics companies had told him Facebook was "finally" fixing the problem.

A separate tweet from the digital analytics company Parse.ly has also suggested Facebook has a fix in the latest updates to its mobile apps. Business Insider contacted Facebook for further details, but the company said it had nothing to share.

Tanya Cordrey, chief digital officer at Guardian News and Media, told Business Insider the Guardian noticed its "dark traffic" audience had dropped around 20% on November 12, and on the same day there was a similar rise in Facebook referral traffic. That was also the same day Facebook released an iOS app update, so it appears the company is already working on the referral issue. As a result of those changes The Guardian's average dark traffic percentage has fallen from between 10-15% to around 7-10%, Cordrey adds.

"The main effect has been that people have underestimated the impact Facebook has on their traffic and that people have maybe been overestimating the impact of Twitter," Coles said.

But he adds that until every app owner — Reddit, Whatsapp email browsers, and so on — does the same thing by creating bespoke automatic URL shorteners, the problem of "dark social" will not go away any time soon.



NOW WATCH: Your Facebook App Is Quietly Clogging Up Your iPhone