How Not To Sort By Average Rating
Summary of post
Problem: We have a list of items each with a rating by users. We want to put the highest-rated items at the top and lowest-rated at the bottom. We need some “score” to sort by.
Wrong solution 1
score = (positive ratings) - (negative ratings)
Magnitude (# of ratings) outweighs percentage of ratings.
Wrong solution 2
score = average rating = (positive ratings) / (total ratings)
Percentage of ratings outweighs magnitude of ratings.
Correct solution
score = lower bound of Wilson score confidence interval for a Bernoulli parameter
Balances the proportion of positive ratings with the uncertainty of having a small number of observations.
Given the ratings I have, there is a 95% chance that the “real” fraction of positive ratings is at least what?
insert complicated equation here
Other applications
Useful to know with confidence what percentage of people took some sort of action. Examples:
- detect spam/abuse: what percentage of people who see this item will mark it as span?
- “best of” list: what percentage of people who see this item will mark it as “best of”?
- “most emailed” list: what percentage of people who see this page will click “email”?
In top rated lists, it can be useful to sort on positive ratings per page view, download, or purchase, rather than per rating, since many people will not rate if satisfied with an item.