Netflix recommendations are broken… There’s an alternative.

The Percent Match Score is a fraud.

Last year, Netflix has caused a lot of turmoil by changing its recommendation system.

https://media.netflix.com/en/company-blog/goodbye-stars-hello-thumbs

Immediately, a lot of members were upset that Netflix had replaced the stars rating system with a less nuanced thumbs rating system. It took a little longer to realize that the most dramatic change was in fact replacing the predicted rating with a percentage. Netflix calls it the “Percent Match Score”, and it’s very important as it determines which shows will be displayed prominently. The problem is that nobody understands how it works. There has been a lot of debates on the internet, for example this one:

Like the man from the video, when you see the percentage given to a show, you assume that it represents how much you’ll like the show. The higher the percentage, the more you’ll like it. That’s what I thought too, and all the persons around me thought the same. After all, that’s how the predicted rating has always worked. Bad news, we’re all wrong, it’s not at all what the percentage means. A percent match score isn’t the same as a predicted rating. It’s a completely different thing, and the man from the video above is comparing apples and oranges, hence the weird results.

Here’s what Netflix says about the Percent Match Score:

“This score indicates how likely we think you are to like that title”.

As you can see, it’s not about how much you’ll like the title, it’s about how likely you are to like it. In other words, it’s the probability that you’ll give it a thumbs up. If you turn the definition around, the lower the percentage, the more likely you are to not like the show. For example, saying that you’re 70% likely to like a show also means that you’re 30% likely to not like it. It’s very far from being a predicted rating! Not only you don’t know if the show will be excellent, or just very good, or just good (goodbye stars, hello thumbs), but when the percentage is lower than 100%, there’s also a possibility that the show will be bad. Thank you Netflix, we’d be lost without your precious help!

The closer to 100%, the more Netflix is confident that you’ll like the show. Conversely, a percentage lower than 100% reveals an uncertainty. For example, a show with 80% match score means that Netflix recommends you to watch that show, but there’s 20% chance that it’s an error to recommend it to you. You want to know how much you’ll like a show, but instead Netflix tells you how likely its prediction could be wrong!? It’s so idiotic that there has to be a problem with the definition given by Netflix.

That doesn’t make sense… It’s not as if Netflix was trying to accurately predict how much you’ll like a show. Netflix is only trying to predict if you’ll like it (even a little) or not. For such a binary prediction, and assuming that Netflix is at least a little good at understanding your tastes, there should be many cases where there’s little doubt whether you’ll like or not. If Netflix understood your tastes perfectly, there would be only match scores of 100% (Netflix is sure you’ll like) and match scores of 0% (Netflix is sure you won’t like). How far the percentage is from 100% can be interpreted as a measure of how badly Netflix understands your tastes.

We can observe percentages as low as 55%, which would indicate a terrible understanding of your tastes. For example, I’m surprised that Netflix is only 73% confident that I’ll like the Silence of the Lambs, one of my all time favorite movies. The old system was able to predict that I’ll LOVE that movie, but the new system isn’t even sure that I’ll LIKE it? Seriously? Considering how Netflix has proven in the past that it could be very good at understanding our tastes, the only remaining explanation is that the percentages must represent something else, but what?

The fact that 55% is the lowest percentage given to a show seems to confirm that this thing really is a percentage of probability. It never goes as low as 50% probability because it would be as accurate as flipping a coin (1 chance out of 2 that you’ll like). By the way, not showing any percentage lower than 55% is a serious issue with this new system, because you can’t tell if a show doesn’t have a percent match score because it’s too bad or because the system wasn’t able to make a prediction. A percentage lower than 50% could have warned you that you’re more likely to dislike a show rather than like it, but Netflix has chosen to hide all the low percentages. No more bad shows, the new system is so corrupted that the bad shows aren’t shown as bad anymore. That’s certainly better for the Amy Schumer stand-up special!

Furthermore, everybody noticed that the Netflix Originals get high percentages and as a consequence they monopolize the top spots. At least we have a proof that the percentages are not completely random 😈 In fact, the more we look at it, the less the percentages look random. Netflix seems to have the ability to manipulate the percentages to promote the shows it wants. Some excellent movies are given unfair low scores, while some average Netflix Originals are given undeservedly high scores.

I’ve read dozens of discussions on the subject, and apparently nobody in the world understands what those percentages mean. It’s ironical considering that one of the reasons that Netflix gave to justify the change was precisely that users didn’t understand the previous prediction system.

What we know:

- The percentages are different from the predicted ratings.

- The percentages probably are percentages of probability.

- The percentages are boosting Netflix Originals.

- The percentages can’t be what Netflix say they are.

What we don’t know:

- What the percentages really are.

A clarification from Netflix would be much welcome!