June 29th, 2016

Update, 10/3/2016: Amazon BANS incentivized reviews two weeks after our video goes viral:

If you’ve read reviews on Amazon within the last few years, you’ve surely noticed a disclaimer at the bottom of many that look like this:

“I received this product for free or at a discount in exchange for my honest, unbiased review”

At ReviewMeta, we call these “Incentivized Reviews”.

Consumers have growing distrust and even disdain for incentivized reviews, especially when it seems every single one is a glowing 5-star review. We wanted to confirm or deny this seemingly anecdotal opinion, so we analyzed 7 million reviews.

After looking at over 7 million reviews, here’s what we see

We found that reviews containing language that would indicate the reviewer received the item for free or at a discount in exchange for a review (incentivized reviews) on average rate the product .38 stars higher than reviews that did not contain this disclosure (non-incentivized reviews).

Non-Incentivized:

5,071,232 reviews (70%), 4.36 average rating

Incentivized:

2,174,802 reviews (30%), 4.74 average rating

With a 0.38 star difference, the impact may appear marginal, however the actual boost in average product rating can be substantial:

Considering that the average product on Amazon is rated around 4.4 stars, a boost from 4.36 to 4.74 stars can mean the difference between a mediocre product and a top rated product. The graph below shows the percentile of products for each average rating:

The distribution of ratings may be even more disturbing:

Incentivized reviewers are 12 times less likely to give a 1-star rating than non-incentivized reviews, and almost 4 times less likely to leave a critical review in general.

Non-Incentivized

Incentivized

Controlling for product still shows a bias

An obvious criticism to the above data is that these reviews are for different products. However, even when controlling for the product we see a similar trend.

We took a look at products that have at least 10 incentivized reviews AND 10 non-incentivized reviews (251 products with a total of 609,766 reviews). On average, the incentivized reviewers gave the exact same product 0.29 stars higher than the non-incentivized reviewers:

86% of products received a higher average rating from their incentivized reviews than non-incentivized reviews:

Here we can see the distribution of discrepancies between the incentivized average rating and the non-incentivized average rating:

Non-Incentivized Higher

Incentivized Higher

Here are some products that exemplify incentivized reviewers giving a much higher rating than the non-incentivized reviewers:

We also took a closer look into a counterexample (a product that received a lower rating from its incentivized reviews than its non-incentivized reviews) and noticed that when this happens, it can be caused by the non-incentivized reviews being even more biased than the incentivized reviews.

When Incentivized Reviews are Lower than Non-Incentivized: A Possible Warning Sign

Those who write incentivized reviews tend to be less critical

We also examined the habits of reviewers who participate in these “free product for review” programs. We only looked at reviewers who have written at least 10 reviews total. We found that on average, reviewers who have written at least one incentivized review give an average of 4.56 stars, while reviewers who have never written any incentivized reviews give an average of 4.27 stars.

The chart below shows the percentage of reviewers for each average rating. For example, 14% of reviewers with at least one incentivized review give an average of 4.9 stars for all products they review, while only 6% of reviewers with no incentivized reviews give an average of 4.9 stars for all products they review.

Reviewers With No Incentivized Reviews

Reviewers With At Least One Incentivized Review

Furthermore, reviewers with at least one incentivized review have written an average of 232 reviews each while reviewers who have never written an incentivized review have written an average of 31 reviews each. There appears to be a trend of high-volume reviewers who rate almost everything 5 stars. Here are a few example profiles and current stats as of the time of writing:

myreview2 : 845 reviews, every single one is 5-star.

Notavailableonhere : 758 reviews, every singe one is 5-star.

Amazon lover : This reviewer has 4,379 reviews. The first review was written on March 23rd, 2015, so just shy of 10 reviews per day, every day for more than a year. All are 5-star reviews with the exception of 9 that are 4-star, and 2 that are 3-star. (The 3-star reviews did not even contain an incentivization disclosure!)

Volume of incentivized reviews is increasingly sharply

While we’ve only examined a sample of 7 million reviews, we’re still seeing that incentivized reviews are increasing in popularity at an alarming rate. This graph shows the review volume during the last 48 months between May, 2012 and April, 2016:

Non-Incentivized

Incentivized

2 years ago, incentivized reviews accounted for less than 2% of new reviews. Since February of this year, they make up the majority of all new reviews on Amazon.

What about Amazon Vine?

Update 7/1/2016 – After several questions, we wanted to make a clarification: Amazon Vine is Amazon’s in-house free-product for review system. It is invite-only, often focuses on larger-value items and can even require the reviewers to return the item. Amazon Vine reviews are NOT included in our incentivized group because they are NOT required to leave the “free product” disclaimer since their reviews are already labeled with the “Vine” badge.

Amazon Vine reviews make up roughly 4% of the reviews in our dataset, and give an average of 4.2 stars (compared to 4.74 from incentivized reviewers and 4.36 from non-incentivized), so at first glance, it seems to be much better controlled than their “review club” counterparts. We’ll be doing an in-depth analysis on Vine reviews shortly, so stay tuned by subscribing to our mailing list, following us on Twitter and liking us on Facebook.

Where do these reviews come from, and why are they biased?

Incentivized reviews are a new and rapidly growing breed of review. While some people label them as “fake”, incentivized reviews don’t technically deserve it because they are likely from real people who had a real experience with the actual product. However, the data still shows they still have a bias – despite their claims to the contrary. Related: Unnatural Reviews and Why We Don’t Use the Word “Fake”

Dozens of “review clubs” (3rd party services) have popped up in the last few years to facilitate the exchange of free or discounted products for “honest” reviews. This is fair game according to Amazon’s Terms of Service as long as the reviewer discloses this fact.

No matter what explanation you choose to believe, it’s hard to argue with the data. Some might try to debunk our entire analysis by citing evidence that they (or a friend/family member) accept free products in exchange for reviews, and they always leave 100% honest feedback. We aren’t calling them a liar; of course there will be “honest” reviewers who participate in these programs. But this is the very definition of anecdotal evidence, and our analysis is on the bigger picture.

Now that we’ve established a clear bias in the data, we’re ready to dive into Part II: The Fundamental Problem with Review Clubs