Is there a bias to user reviews/ratings?

Sometimes I wonder whether XKCD is just better at commentary than all of us. Today’s cartoon demonstrates a great understanding of equilibrium combined with a real issue with regard to consumer incentives. It raises the puzzle that if you like, say, a hotel you have an interest in not saying so because that either reduces vacancies or increases price. You have a similar issue with respect to negative reviews as you would like to steer other consumers there and away from potentially better hotels you would want to stay at. At the very least, one wonders why you would bother to write a review at all. Also, note that ratings of reviewers by other users doesn’t help. If you read a negative review and it turns out the hotel was good, you don’t want to rate the first reviewer poorly because they will help you steer customers away from the gem you have discovered.

Ratings appear to matter for these things — as I wrote about the restaurant evidence previously here. So if reviews were biased this could set in motion a poor equilibrium outcome as xkcd suggests. That said, if the system were explicitly broken, consumers would realise it. Ratings just wouldn’t matter well before the bad drove out the good.

But there is a more subtle bias I have wondered about; one that arises if we supposed consumers only wrote a review once. Suppose they write it honestly. If they have a bad experience, they write a bad review and move on to another hotel. If they have a good experience, they write a good review and return to the hotel. But they don’t review it again. That means that as we count up ratings and average over number of reviews rather than number of hotel stays, then rating averages will be biased downwards. That is, repeat customers soak up some capacity and one consequence of that is that they reduce the probability of an additional review. Those very popular hotels who are always at capacity would be hit by this bias more than unpopular ones and so the variance of average rating scores will be less than it should be. I don’t think this will break the system but it is interesting to ponder.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s