How do you decide which salt-and-pepper shakers to buy? Do you choose the set with 2,500 reviews and 4.5 stars, or the set with 500 reviews and 4.9 stars?

According to a new study published in the journal Psychological Science, people tend to choose products that have a greater number of reviews–even if those same products have lower ratings. And since the researchers also found that there isn’t any relationship between a product’s ratings and the number of reviews it gets, the study means that people are making decisions based on popularity, rather than quality.

For Derek Powell, a post-doctoral scholar at Stanford who was the first author on the paper, this is indicative of the changing state of “social learning”–our collective ability to learn from other people’s experiences–in the age of the internet. With the vast quantity of information online, it turns out that people might not be equipped with the decision-making skills to process it. This insight points to a potentially deep flaw in any ratings-based interface, from e-commerce sites like Amazon to food sites like Yelp: if people aren’t making informed decisions based on star-rating systems because of cognitive bias, then perhaps those interfaces need to compensate, offering ratings information in a way that’s more clear and leads people to make better decisions.

“Greater connectivity has the potential to supercharge one of the most powerful learning mechanisms afforded by evolution and culture,” Powell writes in the paper. “Yet it also places new demands on people’s abilities to translate numbers into meaningful social cues.”

He calls that process of making decisions based on numbers–like the number of reviews or star rating–“intuitive statistics.” To test whether people’s intuitive statistics matched up to actual statistical models, Powell and his colleagues ran an experiment where they asked 132 participants to choose between two similarly priced phone cases that had different ratings and numbers of reviews. The researchers found that people consistently chose the phone case with the greater number of reviews, even if the overall rating wasn’t as good.

“It is persuasive when a product has 1,000 reviews,” Powell says, even if it’s not as highly rated. Rationally, that would suggest that more people thought less of the product, but the results show that people tend to think the popularity of a product is a better reason to buy it than the average rating. There’s definitely truth to it from my perspective–I hadn’t realized it, but I do take the number of reviews into account when choosing something to buy online. Of course, I also look at more than just the rating and the popularity, something the study doesn’t take into account.

While the study only examines people’s decisions based on two numbers and doesn’t look into the impact of other factors like the written content of reviews, it does have implications for both consumers and retailers. As a customer, it’s helpful to know about an internal cognitive bias toward more popular items.