tl;dr:

Facebook understands what’s on your photos. It uses this data to cluster similar posts together on your newsfeed. (but not always successful)

Let me start by the following overview. Everytime you load your newsfeed, Facebook needs to decide which status updates to show you. It also needs to decide how to position them. It seems this placement is more advanced than you might expect.

It’s no surprise that Facebook reads the caption, descriptions & comments for each status update. The most basic clustering just looks at the textual info that’s available. If two of your friends share a similar news story from a different newspaper, Facebook will notice that they are about the same topic. Often you see that these 2 posts will appear just next to each other. Do note Facebook doesn’t need the exact same words. Synonyms or related subjects also are taken into account.

Facebook knows what’s on your photos.

Over the last 2 months I noticed that Facebook tried to analyse what is on my friends’ photos. Let me show you some examples:

I’ve blurred my friends’ names & comments. All these posts were placed right below each other on my own newsfeed.

An article from DeCorresepondent.nl unfortunately matched with a ‘rapist’ picture.

Take the example here on the left. Both pictures in these status updates are visually close to each other, close enough for an algorithm to think they are related.

On top of that the first post is shared at 2:54pm, the second post at 2:59pm. With less than 5 min in between both posts, Facebook would guess both are related and decides to put them next to each other.

In reality both posts are unrelated, but Facebook did at least try ☺

I’ve listed more examples of this behaviour (2 posts, 1 always direct after the other).

What does Facebook see? 2 people, looking at each other, kissing,…

Both photos are very similar in terms of composition but in reality completely unrelated. Again, nice try Facebook☺

What does Facebook see? Bold man with a small beard, looking straight in the camera,…

What you see here is not only a similar composition but also a similar color palette (a lot of dark green, black,…).

Are the following posts found together on my newsfeed coincidence?

If 1 in 5 people posts a baby or food plate picture, you just know that every other day 2 very similar posts will be close to each other. I tried to exclude such cases. But sometimes I’ve the feeling that there must be an algorithm at work.

Red background, white letters?

Skeleton?

Person, angle, stripes, contrasts,…

Similar color accent & palette, street view

often 1 recent post + 1 older, but relevant post are grouped:

As you can see on the dates & timestamps, Facebook often shows a status update that’s very recent (just a couple of hours ago) next to a status update from a while back. (Could be days ago). Out of all the hundreds of posts from days back, Facebook picks an image that ‘feels’ the same and is potentially related.

A theatre or conference?

Both composition as context & color palette.

Wood, grey, fighter jet? ☺

Similar results on iPhone & iPad

Sketched people & outlines, desaturated colours

This “hottest women” example below is more challenging. Although the colours are very similar, you could imagine Facebook would be able recognize the type of woman that’s on there. If Facebook would recognise more details like windows, legs,… it could trick the algorithm to ‘think’ both shots are from the same context or event. This example could be a false positive.

Look what Stanford & Google do already:

Source: Stanford.edu

A. Recently @chrisfahey shared this Tennis Girl analysis from an image recognition algorithm developed by Stanford.

“Woman in white dress standing with tennis racket, two people in green behind her.”

(read more on this research).

Source: Google Research Blog

B. This other photo is captioned by Google & its partners as

“Two pizzas sitting on top of a stove top oven”

More info about their progress can be found here.

If Google can pull this off, we can safely assume that Facebook has similar capabilities. Often people only talk about the extremely accurate facial tagging by Facebook, but there is a lot more info that can be seen in our social media images.

Why Facebook clusters (visually) similar posts:

Goal 1: More structure in your “random” newsfeed

My guess would be that Facebook tries to cluster discussions & news items that are related to bring more structure to the random selection on your newsfeed. Already Birthday wishes and other common posts are grouped.

A. Just look at the example below. On the left you see a photo shared from a local Belgian news site (with a description in Dutch).

B. On the right you see a video shared from the well known vimeo.com platform (with a description in English).

Both the language as the source is completely different, but there are 2 words that are the same: “Knack” & “Roeselare”. Next to these words, both images look very similar. Probably these (and other) indicators tell Facebook that both status updates must be related and groups them together. Smart ☺

Goal 2: Real-time advertising placed next to relevant content

The screenshots I took recently were not limited to status updates by friends. ‘Sponsored posts’ seemed to be part of the same experiment. I’ve added an example below.

Sponsored content placed to similar organic content

The post on the left appeared on my newsfeed because it was ‘liked’ by one of my friends. Yes, it’s a marketing image, but this one appeared organically on my newsfeed. The image on the right is a sponsored (read: paid) status update by Skoda Belgium. Facebook decided to place this paid post after a post with similar content.

Sponsored content placed to similar organic content

Also in this case above the paid Uber status (right), was placed next to visually similar content (left). I must admit I didn’t see too many occurences of this one yet. This could alse be due to the lower number of sponsored posts that are available vs organic posts.

Goal 3: Sponsored posts should blend in (think: native advertising)

A last driver I see why Facebook needs this kind of sorting aligns again with their ad-based business model. Facebook wants to show you as many ads & paid posts as possible, without pushing you away or ruining the overall experience. If the look & feel of advertising fits with the normal status updates you might not even notices that you’ve seen a paid post.

From fun to creepy?

Every single day Facebook processes close to 1.5 Billion photos. (Estimated based on the chart below from months ago. This includes Instragram & Whatsapp uploads.) This number is too big to grasp. Don’t even try to imagine the amount of data & knowledge that could be extracted from such a dataset.

Source: KPCB estimates based on publicly disclosed company data, 2014 YTD data per latest as of 5/14

Today we can safely assume that Facebook can do a lot more than Face-tagging. Smarter sorting to enhance the Facebook experience for users while tweaking their ad-based business model makes a lot of sense.

What other info could be extracted in a couple of years from now? What can Facebook do with algorithms that:

Detect if someone is lying or is dishonest (emotional state)

Detect if you hate or love someone?

Detect how images are altered or modified?

Build a 3D mock-up of your interior based on your indoor pics?

…

Personally I’m very excited about all of this. New tech has the potential to pull people out of their comfort zone. Yes, it might be scary at first but often technology leads to more positive possibilities later on.

Did you see similar behaviour on your Facebook newsfeed recently?

Feel free to send me more examples via @nickdemey on twitter