My thanks to @damonism for an insightful discussion which helped to shape my thoughts for this post.

The ethics of product reviews have been in the news again this week, after Amazon recognised that some practices were causing biased reviews and decided to stop it. For this week's post, I decided to return to a favourite old topic of mine and dig a little deeper into the ethics of product reviews in the fountain pen community.

Amazon

The news came to my attention via a tweet from Ed Jelley, who shared a news story saying that Amazon had changed their rules to ban firms from supplying free or discounted product to reviewers. They did this after a study was released which (not surprisingly) appears to show that the 'incentivised' reviewers had a significant bias in favour of the products, and consequently these products were selling more.

The ReviewMeta study itself is fairly straightforward if you're interested in having a read. The authors gathered data from more than seven million reviews and separated the incentivised reviews from the rest. The comparison found the incentivised reviews were more likely to leave a five-star rating and much less likely to leave a one- or two-star ratings. It also found incentivised reviewers were more active than others (an average of 232 vs 31!).

One of their findings caught my eye and presents a potential issue with the study: some incentivised reviewers -- lets call them cheerleaders -- were leaving five star reviews for almost every product. One imagines these cheerleaders were motivated to get as much swag as possible and, while including them with this analysis makes sense, I would've liked to see how the analysis changes they are excluded. In other words, a comparison between normal reviewers and the incentivised reviewers who weren't exclusively chasing swag. While this might change the degree of bias, I doubt that it would cancel it out completely.

An interesting question which wasn't explored in the study is the degree to which reviewers were aware of their bias. It's hard to believe that the cheerleaders were ignorant of their bias (some were posting so frequently that it's hard to believe whether they even used the products) but one wonders if the average reviewer felt like they were giving an honest opinion or not.

After all, bias is not an emotion like love or hate which we can easily detect and (try to) control. I've been teaching for four years now and it was at least two years before I became aware that I might have some biases in marking. It's rarely as simple as liking a student and wanting to give them extra marks; it's much more subtle than that, a lot of very minor decisions start to add up and yield a favoured student with an inappropriate advantage. Sometimes, I might give certain hard-working or likeable students the benefit of the doubt which wouldn't be extended to everyone. Other times, I might selectively choose to double-check the marking because a grade seems oddly high or low (and when you're deliberately looking for reasons to give more marks, you tend to find them). Each decision might only mean a half-mark difference but it adds up.

Bias is often a very natural thing, almost instinctive, it doesn't feel wrong and takes time to realise you're even doing it. It's definitely not something you can control without being aware it exists. Although I'm well aware that I can be biased and actively try to control that, I'm not sure I could ever say a piece of marking is completely unbiased.

It's easy to understand why Amazon would want to remove biased reviews from the site. They know that potential buyers spend a heap of time researching products and would prefer to avoid this. By providing a large review database, buyers can find information and make purchases all on the one site. Amazon captures more buyers and those buyers are more likely to be satisfied with their purchases. It's win-win.

Biased reviews undermine this. Buyers acquire products which aren't suited to their needs and don't meet their expectations. Over time, they lose faith in the Amazon reviews and have to spend more time roaming around the internet looking for reliable information. They are less likely to buy from Amazon and have to invest more time into research, which makes them less likely to make a purchase anywhere. So this rule change is obviously intended to restore reliability to the reviews and prevent more customers from leaking out to other sites.

Bias and the FP Community

It's undeniable that some manufacturers and retailers are actively trying to shape popular views about their business or products, and are willing to toe an ethically questionable line to do so.

One recent example is an ink manufacturer who gave free product to an FPA member in exchange for them posting a review. I don't believe there was any explicit suggestion that the review had to be positive and so it appeared innocuous enough. But there was also a suggestion of more product if the review was satisfactory and a request that the arrangement be kept private. In my opinion, the lure of more freebies is enough to elicit some bias and the request to keep quiet compounds that. In my opinion, this conduct definitely falls into the category of ethically questionable.

But what really concerned me was that the reviewer was not yet 18: not old enough to drink, not old enough to vote, and certainly not old enough to parse the ethics of posting positive reviews for free stuff. This definitely crosses a line from questionable to wrong, and it's inconceivable that the manufacturer could have failed to see any ethical problems in what they were doing.

Of course, most issues in our community are not this flagrant. I doubt that any reviewer actually feels like they're selling out and shilling product which they know to be bad. Instead, it's the same subtle problems of bias which I experience in grading. Every product review involves a thousand small decisions about what to include and how to frame things, and this is where bias occurs: it can nudge each of those decisions in a particular direction and adds up to a review which is much more positive (or much less negative) than it should be.

Recently, I've noticed that some reviewers seem to have amassed half a dozen sponsors (or more) and it's hard to think that this doesn't compound the problem. If you have a single sponsor, you might receive some very moderate benefits, and a negative review may end that relationship. The cost is not large. But if you have a half-dozen sponsors, the benefits from reviews are greater, and so are the costs of spooking them all with an overly honest review. In this case, bias isn't the decision to say good things about a bad product, it's the moment where a reviewer pauses and thinks, 'Do I really want to say that? What will be the consequences?'

(Also of interest here is that sponsorship may cost retailers nothing. They typically provide store credit instead of cash, which most reviewers use towards expensive purchases. If a reviewer has $200 credit and uses that for a $500 pen, they get a good deal. For the retailer, if the pen only costs them $250, they are making $50 profit on a sale they otherwise would not have made. There's no real or opportunity cost, it's very close to free advertising and this probably explains why there's way more sponsorship than two years ago. It also means bias issues are potentially more severe than they were at that time.)

Does Bias Matter?

If buyers are relying on biased reviews, it's a problem because they might end up making purchases which they wouldn't with more reliable information. In econ jargon, this represents a loss of utility. In plain English, it means that a biased reviewer is acting in a way which benefits themselves at someone else's expense.

On the other hand, it's possible that reviews don't have much effect on purchases. If you're an old cynic (like me!), you're probably not relying all that much on the subjective content of reviews but read them for the imagery and technical details (size, weight, etc.). In that case, reviewer bias is not really a problem because the review itself isn't actually being read.

Another possibility is that fewer people are influenced by product reviews nowadays, and are more swayed by what they see on Facebook, Instagram, reddit, and other social media. I suspect brands will try to increase the visibility of their products on those channels through freebies and samples, much as we saw with the ink manufacturer on FPA (who seems to be getting plenty of exposure lately, without much disclosure about whether products were provided for free).

A third possibility is that most people are experiencing products firsthand, through pen shows and meets, and the reviews are just a form of entertainment or an opportunity to broaden one's product awareness (like reading a catalogue). In that case, a biased reviewer is not important because the reader is not gathering factual information for future purchases but just looking for something to read on their commute or downtime. In this case, more extreme bias -- Comical Ali-style -- might even make reviews more interesting!

It's entirely reasonable to make these arguments but each of them have implications for reviewers. The first two cases suggest that the review itself -- which takes quite a lot of work to put together -- is unimportant, and their blogs would be just as useful if it was just pictures and technical information. The third case suggests that reviewers don't actually need to review pens at all, they just need to write something which readers find enjoyable or interesting. Perhaps there's something to this argument, since it would explain why my blog has an audience and why I still get a heap of views for some old reviews (about the Lamy 2000 and Visconti Homo Sapiens), neither of which had much useful information. It would also explain why Youtube series like the Goulet Q&A and Pen Habit's Currently Inked draw so many viewers.

Conclusion

But while there's an argument to be made for the other explanations, my own view is that reviews still matter and so reviewer bias still matters. People considering large purchases will seek out the views of others in the community and it is in the community's interest that they can access accurate, reliable, and unbiased information. The community is harmed whenever someone acts on unreliable information and is consequently disappointed by the products they acquire.

There's no easy or obvious solution to these problems. Manufacturers and retailers exist to sell product and they're going to find innovative ways to do that. That means they're going to try and make it worth the while of reviewers to put out good press. The only thing standing in the way of this becoming a bad practice is making reviewers aware of the potential for bias, setting out the ethical implications of their decisions, ensuring that they disclose commercial relationships, and making sure the community's expectations are upheld.