In May, Yelp issued 59 new Consumer Alerts, which are notices it puts on a business’s page that it has been caught trying to pay for better reviews. Among those cited were a Beverly Hills plastic surgeon and an emergency room in Humble, Tex. Lifehacker.com recently took on Rotten Tomatoes and Metacritic, arguing their way of compiling reviews was “fundamentally flawed.” FiveThirtyEight.com reported that “men are sabotaging the online reviews of TV shows aimed at women.” (Why? Because they can.)



Bart de Langhe, an assistant professor of marketing at Leeds School of Business at the University of Colorado, used to see numerical reviews online and accept them implicitly. Then, when his son was born three years ago, he needed to buy a car seat. Mr. de Langhe noticed that the seat rated lowest by Consumer Reports got a high rating on Amazon, and the one rated highest by Consumer Reports received a low rating on Amazon.

The more popular seat on Amazon was also more expensive. Were reviewers, he wondered, paying more attention to things like price and brand than the objective, measurable ability of the seat to protect its occupant? With two other researchers, Philip Fernbach and Donald Lichtenstein, Mr. de Langhe began a study that compared online reviews for items like air-conditioners and car batteries with the evaluations in Consumer Reports.

“Navigating by the Stars” was published in April in The Journal of Consumer Research. After analyzing 344,157 Amazon ratings of 1,272 products in 120 product categories, the researchers found “a substantial disconnect” between the objective quality information that online reviews actually convey and the extent to which consumers trust them.

In other words, the consumer saw a number — 4.6 stars out of 5 — and took it much more seriously than it merited.

Nearly half the time, Amazon reviewers and the Consumer Reports experts disagreed about which item in a random pair was better. Moreover, average user ratings did not predict resale value in the used-product marketplace, another traditional indicator of quality.

Julie Law, a spokeswoman for Amazon, said “Amazon customer reviews reflect the feedback, tastes and concerns of real customers, not professional reviewers. That’s what makes them powerful.” She also said the company was now giving more weight to the most recent helpful reviews from purchases that were verified.