# How Not To Sort By Average Rating

*Summary of post*

**Problem**: We have a list of items each with a rating by users. We want to put the highest-rated items at the top and lowest-rated at the bottom. We need some “score” to sort by.

#### Wrong solution 1

`score = (positive ratings) - (negative ratings)`

Magnitude (# of ratings) outweighs percentage of ratings.

#### Wrong solution 2

`score = average rating = (positive ratings) / (total ratings)`

Percentage of ratings outweighs magnitude of ratings.

#### Correct solution

`score = lower bound of Wilson score confidence interval for a Bernoulli parameter`

Balances the proportion of positive ratings with the uncertainty of having a small number of observations.

*Given the ratings I have, there is a 95% chance that the “real” fraction of positive ratings is at least what?*

**insert complicated equation here**

## Other applications

Useful to know with confidence what percentage of people took some sort of action. Examples:

- detect spam/abuse: what percentage of people who see this item will mark it as span?
- “best of” list: what percentage of people who see this item will mark it as “best of”?
- “most emailed” list: what percentage of people who see this page will click “email”?

In top rated lists, it can be useful to sort on positive ratings *per page view*, *download*, or *purchase*, rather than *per rating*, since many people will not rate if satisfied with an item.