Best non-predictive rating systems
James Gibson
OK, I think most of us agree that the RPI is a poor system for ranking
teams. However, it does not seem that using predictive ratings is
going to be a very popular way to go for ranking NCAA tournament
teams. I think, though, most of us agree that the predictive ratings
are the most accurate ratings. So to find a non-predictive system
that had more accuracy, in terms of correlation to the predictive
ratings, I did a crude least squares study on the ordinal rankings
comparing a few non-predictive ranking systems against predictive
rating systems.
For the three predictive rating systems, which we consider to be the
most accurate, I chose the Sagarin Predictor, the Dolphin predictive
ratings, and Ken Pomeroy's pythagorean Log 5 rating system. I think
that helps us span a few different ways to make predictive ratings.
For non-predictive rating systems, I chose Sagarin's ELO-Chess,
Sagarin's overall ranking, Dolphin's standard ranking, the improved
RPI from the Dolphin ratings page, and the RPI according to Pomeroy.
Assuming Sagarin Predictor is the most accurate, the least squares
rankings of the non-predictive systems are:
1. Sagarin Overall 234
2. Dolphin Standard 382
3. Sagarin ELO 444
4. Improved RPI 556
5. Actual RPI 608
Assuming Dolphin Predictive is the most accurate, the least squares
rankings of the non-predictive systems are:
1. Sagarin Overall 303
2. Improved RPI 311
3. Dolphin Standard 392
4. Sagarin ELO 476
5. Actual RPI 623
Assuming that Pomeroy's Pythagorean Log5 methods is the most accurate,
the least squares rankings of the non-predictive systems are:
1. Dolphin Standard 252
2. Improved RPI 279
3. Sagarin Overall 403
4. Sagarin ELO 552
5. Actual RPI 613
If these predictive ratings are accurate, then it's clear the RPI
needs to be junked. I am surprised at how poorly Sagarin ELO came out
in these comparisons. Of these systems, both Sagarin Overall and
Dolphin standard come out well.
Michael Sullivan
> If these predictive ratings are accurate, then it's clear the RPI
> needs to be junked. I am surprised at how poorly Sagarin ELO came out
> in these comparisons. Of these systems, both Sagarin Overall and
> Dolphin standard come out well.
Both Dolphin standard and Sagarin Overall use predictive data, so that's
not really much of a surprise. If median likelihood completely ignored
MOV information even in determining SOS, then it might be a good
candidate.
Certainly it looks like Improved RPI dominates RPI, even if it retains
one of the primary problems (that it greatly overweight schedule
strength -- at least it seems to do a reasonable job of measuring the
schedule strength before it dramatically overweights it).
I'm surprised that Sag ELO is worse than improved RPI though. Of
course, Sagarin never discusses exactly what he does to make that
ranking, so it's hard to know why. A real maximum likelihood method
oughht to be a better predictor than any ad hoc RPI style formula.
Michael
--
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
James Gibson
> James Gibson <james.m.gib...@gmail.com> wrote:
> > If these predictive ratings are accurate, then it's clear the RPI
> > needs to be junked. I am surprised at how poorly Sagarin ELO came out
> > in these comparisons. Of these systems, both Sagarin Overall and
> > Dolphin standard come out well.
>
> Both Dolphin standard and Sagarin Overall use predictive data, so that's
> not really much of a surprise. If median likelihood completely ignored
> MOV information even in determining SOS, then it might be a good
> candidate.
Well, I knew correlating Sagarin overall to Sagarin pure points would
severely favor Sagarin, considering it's using part of its own rating
system. I think something like Dolphin may be acceptable because it
doesn't encourage RUTS for your own rating, but uses the points to
figure out how strong your opponents are.
> Certainly it looks like Improved RPI dominates RPI, even if it retains
> one of the primary problems (that it greatly overweight schedule
> strength -- at least it seems to do a reasonable job of measuring the
> schedule strength before it dramatically overweights it).
>
> I'm surprised that Sag ELO is worse than improved RPI though. Of
> course, Sagarin never discusses exactly what he does to make that
> ranking, so it's hard to know why. A real maximum likelihood method
> oughht to be a better predictor than any ad hoc RPI style formula.
I want to do this with a few others. I saved Massey's page from
yesterday so I have all the pre-tournament rankings to compare to the
predictive ones I have. First glance tells me that Wilson, which is
an ad hoc formula looks ok, while the Colley Matrix looks almost as
bad as the RPI.
stat monkey
James Gibson
Ok. I am going to add LRMC pure points and Greenfield to my set of
predictive rankings. For the non-predictive rankings, I will add LRMC
capped, LRMC(0), as well as the Colton Index, Massey BCS, the Score
Card, the Colley Matrix, and DWHoops. Anybody have any others they
want me to add on either side? I can't promise quick results, not
with all the actual basketball I need to watch over the next few weeks.