384 views

Skip to first unread message

Nov 29, 2002, 11:07:29 AM11/29/02

to

Here's an interesting puzzle -- at least I'm puzzled with this! :-)

I see the Top 250 movies in the IMDB (Internet Movie DataBase),

and I'm curious about the formula for the weighted ranking, which

they call a "true Bayesian estimate".

People vote for a movie, giving 1 to 10, and then they rank the

movies with this formula, that considers the average of votes

and also the number of votes:

v m

------- * R + ------- * c

v + m v + m

Where:

v = number of votes

m = minimum number of votes to be considered (currently 1250)

R = average "vote" given to the movie

c = average "vote" for all the movies (currently 6.9)

(when I put "currently", I mean that that is what they list in

their page, where they describe the formula)

I understand part of the "intent" of that formula, but I'm having

trouble understanding another part...

I see how it conveniently gives better ranking to a movie with

higher number of votes for the same average vote (if 10 people

say that a movie is 8/10, that's not as good as 1000 people

saying it is 8/10). So, the estimate gives certain weight to

the movie's ranking, but then it also considers the overall

average -- kind of an "in case of doubt, I'll assume you are

in the overall average" -- which translates to "in case of

insufficient information, I'll assume that probabilistically

speaking, you'll approach the overall average".

Obviously, the fraction that considers the movie's average vote

approaches 1 when the number of votes approaches infinity,

and the other fraction (the one that considers the overall

average) approaches zero when the number of votes approaches

infinity.

My doubt is: what is that magical number 1250? Is it arbitrary?

If it is, then how can they call this a "true Bayesian estimate"?

A true estimate tells me that they are estimating what is the

value with maximum likelihood to be the true value of the movie's

average vote, given that we don't count on an infinite number

of votes (which is what we would require to obtain the true

average vote). But this value, this formula is, the way I see

it, "tainted" by this mysterious 1250...

Let me put my doubt this way: We have to determine things like

the following: we have two movies: move 1 has 2000 votes and

they average 9/10; movie 2 has 4000 votes and they average 8/10;

which movie is better?? (defining "better" as the movie for

which an infinite number of voters would give a higher average).

Movie 2 may have less average, but it carries more weight, as I

have higher "confidence" in that value, since a lot of people

have voted, whereas movie 1 has higher average, but it is

"doubtful" -- not a lot of people have voted...

So you see that if I make m = 10, I obtain one value, but if I

make m = 2000, I obtain something very different. So, is that

m number really arbitrary? Or does it obey some mathematical

formula depending on the probability distribution of the votes?

Thanks,

Carlos

--

Nov 29, 2002, 4:00:19 PM11/29/02

to

>Subject: Determining weighted rankings

>From: Carlos Moreno moreno_at_mo...@xx.xxx

>Date: 11/29/2002 10:07 AM Central Standard Time

>Message-id: <3DE790C1...@xx.xxx>

>From: Carlos Moreno moreno_at_mo...@xx.xxx

>Date: 11/29/2002 10:07 AM Central Standard Time

>Message-id: <3DE790C1...@xx.xxx>

By what criteria do you claim that not a "lot" of people voted for movie 1? It

seems to me that the function of m is to determine what constitutes a "lot".

In your example, 2000 > 1250, so why should it be doubtful?

Plugging in the numbers gives rankings of

v=2000 R=9 m=1250

8.192307692 for movie 1

and

v=4000 R=8 m=1250

7.738095238 for movie 2

so yes, movie 1 is indeed "better" than movie 2.

One way to look at m is to ask: how many votes of 9 does a movie need to be

considered "better" than movie 2?

v=831 R=9 m=1250

7.738587218

Thus, with only 831 votes required when we actually have 2000, there is no

reason to be doubtful that movie 1 is better.

Another question that can be asked is what if m were smaller, say 625 instead

of 1250? The ranking formula will result in a value that falls between c and R.

Higher values of v bring the result closer to R. The parameter m determines how

quickly it approaches R. Lower values of m mean greater signifigance to each

vote.

So we can ask: if m were 625, how many votes of 9 would a movie need to equal

4000 votes of 8?

v=4000 R=8 m=625

7.851351351

v=518 R=9 m=625

7.851706037

Note that with a lower value of m, we only need 518 votes instead of 831 to

equal movie 2. Thus, each vote of 9 carries more weight when m is lower.

>

>So you see that if I make m = 10, I obtain one value, but if I

>make m = 2000, I obtain something very different. So, is that

>m number really arbitrary? Or does it obey some mathematical

>formula depending on the probability distribution of the votes?

I doubt that m is arbitrary, but I do not how how they arrive at it.

>

>Thanks,

>

>Carlos

>--

Dec 2, 2002, 3:20:55 PM12/2/02

to

Carlos Moreno <moreno_at_mo...@xx.xxx> wrote in message news:<3DE790C1...@xx.xxx>...

I agree with you. "A true Bayesian estimate" implies to me that the

estimate assumes a plausible model of the population of voters and

uses that model to derive the best possible estimate. I can't guess

what model that would be.

> A true estimate tells me that they are estimating what is the

> value with maximum likelihood to be the true value of the movie's

> average vote, given that we don't count on an infinite number

> of votes (which is what we would require to obtain the true

> average vote). But this value, this formula is, the way I see

> it, "tainted" by this mysterious 1250...

>

> Let me put my doubt this way: We have to determine things like

> the following: we have two movies: move 1 has 2000 votes and

> they average 9/10; movie 2 has 4000 votes and they average 8/10;

> which movie is better?? (defining "better" as the movie for

> which an infinite number of voters would give a higher average).

> Movie 2 may have less average, but it carries more weight, as I

> have higher "confidence" in that value, since a lot of people

> have voted, whereas movie 1 has higher average, but it is

> "doubtful" -- not a lot of people have voted...

The 1250 value is unrelated to the usual statistical confidence

levels. With population sizes of 2000 and 4000 and all ratings

limited to the range 1/10..10/10, we can be very confident in the

significance of the difference between the two means of 9/10 and 8/10

(we're talking on the order of 10 standard deviations).

Maybe the rating formula is based on the assumption that the

moviegoers are voting with their feet. You could consider the

decision not to see a movie as a tentative thumbs-down on that movie

based on its reputation, or lack of same. But the formula isn't

directly based on this consideration, either, since the formula is

nonlinear in the number of voters -- it treats the difference between

2000 and 10000 votes as being much more important than the difference

between 10000 and 80000 votes.

You could rightly object that a linear model based on the

voting-with-their-feet assumption is too simplistic. A person might

prefer to see movie 1 but ends up seeing movie 2 instead just because

movie 2 happened to be the one showing at the local theater. Any

model you choose will be incomplete, since you can't model all the

elements that determine how many people will end up rating a given

movie on IMDB.

So the actual formula used by IMDB seems like a reasonable hack. Such

hacks would probably offend an anti-Bayesian, and if the enemy of your

enemy is your friend, maybe that makes it a Bayesian hack. Or maybe

the writer just thought that "a true Bayesian estimate" sounded more

scientificky than "a weighted average".

Reply all

Reply to author

Forward

0 new messages

Search

Clear search

Close search

Google apps

Main menu