Player Ratings and Statistics
Lowell....@gmail.com
) got me thinking about statistics and ultimate. If your goal with
these statistics is to accurately rate a player, things like assists
and number of D's just doesn't cut it -- someone could do well in both
of those areas, yet still be a detriment to their team -- for example,
if the player in question had a lot of turns in addition to the
assists, or if their D's came off of risky poaches. Alternatively, it
is easy to imagine an excellent player who completely shuts their man
down and never gets a D, or a handler who gets the offense moving, but
for one reason or another doesn't throw many assists.
I think the best measure of a player's ability is simply whether their
team is more likely to score if they are on the field, adjusted for
the strength of their teammates on the field, the strength of the
opponent, etc. (Think adjusted plus-minus if you follow basketball
statistics. See http://mmackey.blogspot.com/2009/02/howstat-working-out-for-you.html
for a few links.) This is pretty much the definition of a players
abilities, and it turns out not to be too difficult to calculate
(though not as simple as counting goals/assists/turnovers/D's).
Of course, the down side to this approach (and perhaps something that
the traditional stats can help with) is that this method doesn't give
any clue as to _why_ someone is good (or bad), or what they need to
improve on -- it just indicates how much more likely the team is to
score with them on the field.
During some down time over the winter, I got my geek on and wrote a
program that does this sort of analysis. It takes into consideration
(only) which players on the field, who the opponent was, and which
team scored, and from this calculates what effect each player has on
the team's probability of scoring (and, incidentally, the strengths of
all of the opponents and how much advantage the offensive team has).
For what its worth, based on the results from a one-day not-at-all-
serious tournament, our team's best D-line player had several flashy
plays - like big layouts and hucks - over the course of the
tournament, while the best O-line player was exceedingly boring (in
the good, chilly, way).
What does everyone else think? Are other statistics more useful?
More interesting? Have I completely reinvented the wheel here? Is
worrying too much about statistics detrimental to the fun of the
game? Am I too geeky to play Ultimate?
If anyone is interested, send me an email and I will send you a brief
writeup and my source code. (Note that you will need some
understanding of probability and you will have to know your way around
a compiler for this to be of any use to you.) If there seems to be a
general level of interest, I will try to get my program into a form
that is usable without any specialized knowledge. And if you have
detailed statistics of the type mentioned above that you are willing
to share, I might be willing to do some analysis for you in order to
test my code (no promises though).
-Lowell
bww
somewhat flawed. If you are say, Tim Gehret on Florida, and play
every point of nationals (which i believe he did), his probability of
scoring is most likely less than a scrub O-Line player on the same
team, because the D-Line is guaranteed to score less often. Your
model would work beautifully if a team has a perfect divide between O
and D lines, but often players play both, and im interested in how (if
you did) account for that.
Frankly, nobody takes good stats in ultimate because there are too
many things to keep track of, we're at a similar stage to when Sacks
weren't a stat in football. But, for fun's sake, and to avoid reading
an excedingly boring essay, lemme think of a list of stats that might
accurately reflect how well a player does.
Scores
Assists
Callahans
Legit Ds
Throw Aways
Punts
Forced Throw Aways
Drops
Throws (or would catches be better, they represent almost the same
thing, but only almost)
Marks Broken on O
Defensive Marks Broken (the times your mark was broken... gotta be a
better way to say it)
Handblocks
Covered Opponent's Catches
Pulls
Pulls OB
Greatests
Calls
Calls Against
Calls Overturned
TMFs
Im assuming this is an incomplete list. But i had been thinking about
this for a while as well. I hate ultimate stats.
Jeff
Stats Allstar article in the NY Times this Sunday ... or why Shane
Battier could be his teams MVP ...
http://www.nytimes.com/2009/02/15/magazine/15Battier-t.html?_r=1&ref=basketball
Joe Seidler
<Lowell.L.Ba...@gmail.com> wrote:
> Reading a few blog posts (likehttp://cultimate.blogspot.com/2008/10/defensive-brilliance-is-it-show...
> ) got me thinking about statistics and ultimate. If your goal with
> these statistics is to accurately rate a player, things like assists
> and number of D's just doesn't cut it -- someone could do well in both
> of those areas, yet still be a detriment to their team -- for example,
> if the player in question had a lot of turns in addition to the
> assists, or if their D's came off of risky poaches. Alternatively, it
> is easy to imagine an excellent player who completely shuts their man
> down and never gets a D, or a handler who gets the offense moving, but
> for one reason or another doesn't throw many assists.
>
> I think the best measure of a player's ability is simply whether their
> team is more likely to score if they are on the field, adjusted for
> the strength of their teammates on the field, the strength of the
> opponent, etc. (Think adjusted plus-minus if you follow basketball
Here is one way to rank players:
http://www.ultimatehistory.com/specialaccomplishments
mkt
> The probability that Team A scores with Player Y on the field is also
> somewhat flawed. If you are say, Tim Gehret on Florida, and play
> every point of nationals (which i believe he did), his probability of
> scoring is most likely less than a scrub O-Line player on the same
> team, because the D-Line is guaranteed to score less often. Your
> model would work beautifully if a team has a perfect divide between O
> and D lines, but often players play both, and im interested in how (if
> you did) account for that.
The last issue can be accounted for in adjusted plus/minus statistics,
which can take into account not just who a player played with, but
whether he was on the O-line or D-line, and in basketball, homecourt
advantage (perhaps the closest analogue to O- and D-lines, although it
pertains to entire games and not to single points).
However the first issue, players who play almost all of a team's
points, is a major problem for adjusted plus/minus ratings. They
basically work by comparing how a team does when a player is on the
field compared to when he is off the field. But if there are few
points when a player is off the field (or if they are unusual points,
e.g. garbage time in basketball games), then plus/minus has a hard
time giving accurate estimates.
A similar issue arises with groups of players: if Tim Duncan and Tony
Parker spend most of their minutes on the floor together (and likewise
are mainly on the bench at the same time), and if we observe that the
Spurs tend to outscore their opponents when Duncan and Parker are both
on the floor, then we know that the Duncan/Parker duo is helping the
Spurs -- but we cannot estimate how much of that is due to Duncan and
how much is due to Parker.
Bottom line: adjusted plus/minus stats are a cool idea, but they are
not precise; they have large standard errors. Analysts looking at NBA
statistics will sometimes try to use three seasons' of data instead of
just one, in order to get larger sample sizes and smaller standard
errors. The problem there of course is that players' qualities will
change over three years, especially if they are a developing young
player, or an aging old player.
> Frankly, nobody takes good stats in ultimate because there are too
> many things to keep track of, we're at a similar stage to when Sacks
One nice thing about plus/minus stats is that the only thing you need
to track is who was on the field during the point (hmm, I'm not sure
how injury subs would be handled), and who pulled. Which goalline
they were attacking would probably be good too, to try to account for
wind direction. I doubt that homefield advantage counts for much but
you could record that too.
That's just for doing adjusted plus/minus, which in the assessment
field would be called a "summative measure". Ben's list of the stats
on the plays that a player makes, ("boxscore stats" in the lingo of
basketball analysts; "formative measures" in the lingo of assessment)
would be used for a different way of trying to measure players'
contributions. Some analysts of NBA stats have attempted to combine
the boxscore stats with adjusted plus/minus: with boxscore stats, we
don't know how relatively important each stat is to victory nor how
much of the stat should be credited to the individual player vs how
much of it was due to team defense or a teammate's help. Adjusted
plus/minus stats don't have those problems because they directly
measure which team scored, and in theory measure how much of that
contribution came from each individual player. Their big disadvantage
is that they have large standard errors, i.e. low reliability.
Would a season's worth of Ultimate stats yield enough data to overcome
the large standard errors? My guess is probably not, NBA players play
82 games in which typically 100 points (the equivalent of about 50
field goals) are scored whereas college Ultimate players typically
play fewer games than that, and only to 15 points (or 13 or 11). Plus
there are 7 teammates on the field, vs. only 5 on a basketball court,
so there are more individual player impacts that need to be estimated
from each point played. I'm not sure how many games a club team plays
in a season, I can imagine it can get close to or exceed 82 games but
probably not by much.
All that being said, it's certainly an interesting idea, and I would
be interested in seeing the calculations and the program. tamada
(at) oxy (dot) edu
--MKT
P.S. The NY Times article was interesting, what it mentioned but
perhaps not quite prominently enough is that the Houston Rockets, in
doing summative evaluations of Shane Battier, are almost certainly
using some form of adjusted plus/minus rating, although probably a
proprietary one of their own design. I.e. they know (or at least
believe) that they do better when he is on the floor. The next step
is to look at formative measures, as described in the NY Times article
(and similar to Ben's list, except looking for new, unmeasured aspects
of Battier's play): what exactly is Battier doing that makes his
teams better? From the article, it sounds like the Rockets have
clearly identified some things that Battier does, but are still
looking for the rest.
ad_g...@yahoo.com
Madden Stats. Duh
Rob
Yards gained/touch vs turnovers/touch
The first to give an idea of how offensive minded the player is ie:
are they running or hucking deep vs dump and swing handler
The second to give a measure of how well they achieve the first stat
Rob
joel....@gmail.com
> The probability that Team A scores with Player Y on the field is also
> somewhat flawed. If you are say, Tim Gehret on Florida, and play
> every point of nationals (which i believe he did), his probability of
> scoring is most likely less than a scrub O-Line player on the same
> team, because the D-Line is guaranteed to score less often. Your
> model would work beautifully if a team has a perfect divide between O
> and D lines, but often players play both, and im interested in how (if
> you did) account for that.
I have been using a modified +/- for a while with my team. I
understand your argument and was concerned about it as well. To
account for this, I divide the +/- score by number of points played so
everyone is on equal footing. This gives a number from -1 to 1 with
-1 meaning the other team scores every time this person is on the
field and +1 meaning our team scores every time. For my own use, it
has helped me get a better handle on who is actually playing well as
opposed to who I think (sometimes erroneously) is playing well.
For teams that run fairly standard lines, this approach may not be as
revealing as an individual player may play more points with a better
(or lessor) group of players.
joel
Lowell....@gmail.com
First of all, great post MKT; you answered a lot of questions better
than I could have.
On Feb 17, 10:18 pm, mkt <tam...@oxy.edu> wrote:
I do take into account the fact that it might be easier to play O than
D (and I actually calculate this in the code); however, the method I
currently use is only valid if all of the teams are of roughly equal
strength(*) -- I plan to correct this in the future. You are correct
in that I can not predict a player's ability if they play every single
point (if a good player played every single point, my code would rate
both him and all of the opponents somewhat worse than they deserved).
Similarly, two players who went on and came off the field for the same
points would be given the same rating (which would be somewhere in
between the proper ratings for both of them).
However, the code should work fine (within the statistical limitations
discussed below), as long as every player takes some points off during
the course of a tournament, even if that player plays every single
point for most games. (You just need enough points without them on
the field to figure out how good the other players are).
(*) I currently look at how likely it is the the offensive team scores
vs. the defensive team; however, if the teams are mismatched, the
stronger (winning) team will play more D-points, making the code think
that D is easier than it should be. The correct way to do this is to
look at how likely each team is to win an O point as opposed to that
team winning a D point.
>
> Bottom line: adjusted plus/minus stats are a cool idea, but they are
> not precise; they have large standard errors. Analysts looking at NBA
> statistics will sometimes try to use three seasons' of data instead of
> just one, in order to get larger sample sizes and smaller standard
> errors. The problem there of course is that players' qualities will
> change over three years, especially if they are a developing young
> player, or an aging old player.
You are right about the statistical errors bit. What I have done to
date is to calculate the ratings spread that results from only
considering, say, random subsets of 90% of the points played. It
seems like there is enough data from a two day tournament to get
reasonable (but not great) uncertainties, but (and I have not checked
this yet), I expect the uncertainty to scale with the number of points
like 1/sqrt(N), i.e. that it would take a huge amount of data to get
"good" results.
>
> > Frankly, nobody takes good stats in ultimate because there are too
> > many things to keep track of, we're at a similar stage to when Sacks
>
> One nice thing about plus/minus stats is that the only thing you need
> to track is who was on the field during the point (hmm, I'm not sure
> how injury subs would be handled), and who pulled. Which goalline
> they were attacking would probably be good too, to try to account for
> wind direction. I doubt that homefield advantage counts for much but
> you could record that too.
I do not know how to account for injury subs, my best idea so far is
to give both the sub and the subee 50% of the credit for the point.
And to just give up if the sub gets injured. This isn't really fair,
but I don't think it would make that big of a difference. It is very
easy to include other variables, for example which goalline you are
attacking, but soon you get to the point where you don't have enough
data to estimate all of the variables. For example, I would like to
consider two person interactions, that is how well player A does when
player B is on the field vs. when player B is off the field. However,
for a team of 20, you have (20)(19)/2=190 such interactions, which is
comparable to the total number of points played in a tournament.
>
Their big disadvantage
> is that they have large standard errors, i.e. low reliability.
>
> Would a season's worth of Ultimate stats yield enough data to overcome
> the large standard errors?
Good question, and I do not really know the answer, but I plan to find
out someday.
O-Town
A comprehensive list of boxscore stats might be nice, but some realism
could be useful: You're not going to get those kind of stats anytime
soon. Reflecting on the Battier article, I started thinking about what
stats would change if a good containment defender were marking up
against a offensive star. The star would do less breaking, less
scoring, less assisting, throw on higher stall counts, and generally
be forced to make lower percentage throws. None of those things shows
up in the stat-box for a defender and I highly doubt we're going to
start keeping stats on our opponents like that anytime soon.
Personally, I stick to a simple rubric for grading my own play:
Turnover differential. Did I EARN a D? Plus one. Did I turn the disc
over? Minus one. Ultimate is offensively oriented. If you don't turn
the disc over to the other team, you'll win - providing you win the
flip as well. Eventually the difference in turnovers determines the
score. Why not simply track each player's contribution to that?
Combine that with the plus/minus you've already got going on and I
think the intersection will provide you with some useful info.
Best of all, this system is easily implemented. If you're keeping
track of who's playing on what points already, tallying up Ds and
Turns is a cinch.
- O
Lowell....@gmail.com
> understand your argument and was concerned about it as well. To
> account for this, I divide the +/- score by number of points played so
> everyone is on equal footing. This gives a number from -1 to 1 with
> -1 meaning the other team scores every time this person is on the
> field and +1 meaning our team scores every time. For my own use, it
> has helped me get a better handle on who is actually playing well as
> opposed to who I think (sometimes erroneously) is playing well.
> joel
Has keeping these stats been useful, meaning do you think that knowing
who is playing well has helped you win games? Has it been an issue of
deflating egos, or altering strategy, or just playing the people who
are having a good day? One thing that I am interested in is
determining whether a particular player is more valuable to the team
on an O line or on a D line. You could also break down the data more
and determine (for example) if an individual is more useful cutting or
handling, or playing zone D vs. man D.
Also, you should be able to (relatively easily) correct for the number
of points played (as you have), as well as the strength of the other
players on the field and the opponent. I started from a strictly
probability-based approach, instead of starting at +/- and working
back to a probability (because that is easier for me to understand),
but I expect that the two concepts are basically equivalent. Your
approach might also give more spuriously high or low results due to
statistical fluctuations for someone only playing a few points, while
my method will tend to rank players closer to average in the absence
of much data.
Adam Tarr
I'm interested in what you got. Take out the NOSPAM to e-mail me.
For those interested, here's a slightly out-of-date look at the
adjusted plus-minus stats I compile for my club team.
http://atarr.blogspot.com/2007/10/updated-approach-to-plusminus-stat.html
I've been meaning to update, because I've got a new way of figuring
out the baselines, based on turnover statistics, that gives
(subjectively) better results.
On Feb 17, 6:01 pm, "Lowell.L.Ba...@gmail.com"
<Lowell.L.Ba...@gmail.com> wrote:
> Reading a few blog posts (likehttp://cultimate.blogspot.com/2008/10/defensive-brilliance-is-it-show...
> ) got me thinking about statistics and ultimate. If your goal with
> these statistics is to accurately rate a player, things like assists
> and number of D's just doesn't cut it -- someone could do well in both
> of those areas, yet still be a detriment to their team -- for example,
> if the player in question had a lot of turns in addition to the
> assists, or if their D's came off of risky poaches. Alternatively, it
> is easy to imagine an excellent player who completely shuts their man
> down and never gets a D, or a handler who gets the offense moving, but
> for one reason or another doesn't throw many assists.
>
> I think the best measure of a player's ability is simply whether their
> team is more likely to score if they are on the field, adjusted for
> the strength of their teammates on the field, the strength of the
> opponent, etc. (Think adjusted plus-minus if you follow basketball
> statistics. Seehttp://mmackey.blogspot.com/2009/02/howstat-working-out-for-you.html
Jeff
other teams players in a meaningful enough manner to know how to
encourage the other player/teams to play towards their weaknesses ...
applying those stats to develop a desire set of outcomes against a
player and then measuring how the player/team whose goals is to
acheive them is sucsessful
On Feb 18, 12:26�pm, Adam Tarr <ahtarrNOS...@gmail.com> wrote:
> Lowell,
>
> I'm interested in what you got. �Take out the NOSPAM to e-mail me.
>
> For those interested, here's a slightly out-of-date look at the
> adjusted plus-minus stats I compile for my club team.
>
> http://atarr.blogspot.com/2007/10/updated-approach-to-plusminus-stat....
> > -Lowell- Hide quoted text -
>
> - Show quoted text -
scott...@gmail.com
had the great privelege of serving as the coach for Rare Air for the
last few years, and have spent a good bit of time recording and
analyzing game scoresheets, and looking for better ways to quantify
performance on the field beyond the standard "formative measures",
e.g. goals, assists, blocks, turnovers, drops, etc.
As has been stated, these measures don't always give a good picture of
a player's value to his team. One player's style of defense may
result in a lot of blocks, while another "shut-down" defender might
never show up on the scoresheet, though he might have prevented the
opponent's best player from touching the disc, contributing to
turnovers simply by taking away the opponent's best option.
What we would like to somehow quantify is "influence", i.e. a player's
ability to increase the chances of scoring by being on the field, and
doing all the things that make a difference, even if they don't show
up in the box score. Of course, this acknowledges that even if
someone does everything right, there is a distinct possibility that
the other team could still score, as there is the possibility that
someone could do everything wrong and still be on the field when his/
her team scores. Thus, the measure would become valid only over a
larger sample set, as over the course of time, the short-term
fluctuations would start to be averaged out, and the long-term
"influence" would start to make itself evident. (How big a sample set
is required is certainly a worthy point of discussion.)
Initially, plus/minus looks like a promising measure, but as noted, it
has many flaws. Mostly, it doesn't take into account how difficult
the opponent, and either rewards or punishes players based on the
number of points played. Example: If you win a game against an weak
opponent 15-0, the players who play more (which may not be your best
players) have a higher plus/minus. If you then lose 15-0 to a strong
opponent, those who played more points (perhaps your stronger
players), will have a lower plus/minus.
So, a few years ago, I started toying around with a numeric
measurement called Relative Plus/Minus (RPM), which sounds very
similar to the adjusted plus/minus described previously.
Warning: Those who are not math geeks, or could care less about this
kind of stuff should read no further.
Specifically, with RPM, I believe I have found a way to remove the
skew caused by the number of points played, the skew caused by the
final overall score, and the skew caused by the difference between
starting on offense vs. starting on defense. The result is a number
that represents a player’s plus/minus relative to that of the team
average, calculated separately for offense and defense before
generating the total sum.
Regardless of the final score, you will always have a positive RPM
(even if your total plus/minus was negative) if your team's plus/minus
was better with you on the field. In fact, in any game, because the
RPM is relative to the team average, the sum total of all the RPM
numbers for the team will always be zero, regardless of the margin of
victory or defeat.
Here’s how it works …
For each player, I come up with two sub-totals, called “expected OPM”
and “expected DPM”. The expected OPM is the expected plus/minus for
the player, based on the percentage of points played when starting on
offense and the resulting team plus/minus when starting on offense.
Expected OPM = (individual points played on offense / team points
played on offense) x (total team plus/minus on offense)
So, if the team played a total of 10 points starting on offense, and a
player played 6 of those points, and the team was a total of +5 when
starting on offense, then the expected OPM = (6/10) x 5 = 3.0.
Likewise, expected DPM is the expected plus/minus for the player,
based on the percentage of points played when starting on defense and
the resulting team plus/minus when starting on defense.
Expected DPM = (individual points played on defense / team points
played on defense) x (total team plus/minus on defense)
So, if the team played 10 points starting on defense, and a player
played 4 of those points, and the team was a total of -3 when starting
on defense, then the expected DPM = (4/10) x -3 = -1.2.
I then add the two sub-totals to get the player’s expected plus/minus
for the game. In the above examples, the player’s expected plus/minus
would be 3.0 – 1.2 = 1.8. This is the plus/minus that the player
would expect to have based on the team average and the percentage of
points played on offense and defense. To get the RPM, I just take the
player’s actual plus/minus from the game, and subtract the expected
plus/minus. So, if the player’s actual plus/minus for the game was
+5, then the RPM would be 5 – 1.8 = 3.2.
This number represents the player’s plus/minus relative to the team
average. So, if the RPM is positive, it means that when this player
was on the field, we scored more than we usually did. And if the RPM
is negative, it means that when this player was on the field, we
scored less than we usually did. Whether or not the player had
anything to do with that, or it was just random occurrence, is up for
debate. And, as I mentioned before, because this number is relative
to the team average, the sum total of the RPM ratings for members of
any team will be zero. Also, note that if a player plays every point,
that player's RPM will be zero for that game.
I do feel that while this stat may have some randomness on a game to
game basis, over the course of a season, it might begin to represent
some of the intangibles that don’t show up in other places on the
scoresheet. If someone if consistently positive, perhaps they are
having a positive influence on the game when they are on the field, to
the point that we score more often when they are on the field?
When I look at the numbers for a game, I tend to ignore any numbers
that are in the -2 to +2 range, assuming that this is down in the
noise. However, when I start seeing larger numbers, I begin to think
that perhaps it might be more than just randomness.
For example, in one game, one of our players wound up with a +5.0 RPM
for the game. This is a very large number for a single game. Two
goals and an assist probably had something to do with it. But another
player also had 2 goals and an assist, and wound up with an RPM of
only 0.7. Is it possible that the first player had influence on the
game beyond the story told by the goals and assists? Perhaps it was
her great shut-down D on the dump pass (which does not show up in the
D column)? Perhaps it was her good movement of the disc, good
patience, and great break mark throws (which might not have directly
resulted in a G or A in her stats)?
Regardless, the end result is that with the first player on the field
in that game, we outscored our opponents 7-5. With that same player
on the sidelines, we got outscored by our opponents 8-1. That’s quite
a difference in one game. But you might not know that by just looking
at the stats, without the benefit of the high RPM telling you that
something different happened in that game with that first players on
the field. But was it a statistically random coincidence or not?
I believe that over the course of a season, the RPM numbers will give
an indication of the players who had "influence" over the game, on
both offense and defense. I continue to add the RPM numbers from each
game over the course of a season. Each game exists as its own entity,
which should remove the issue of varying opponents strengths, since
you are only measuring on a game by game basis against the average for
that game. Note that this does not take into account factors like
wind advantage, though in most of these games the defense and offense
tend to track with upwind and downwind, though not perfectly. It also
does not take into account things like "garbage time", where the skill
level of one team changes over the course of a game. I have also (as
stated by someone else previosuly) not completely resolved how to
handle injury substitutions during points. I tried halving the point,
but it made the calculations more cumbersome, so I have just gone back
to giving credit to the person who is on the field at the time the
goal is scored. Again, not perfect.
And, the final disclaimer, as with all statistics, the RPM numbers can
be used as an aid, but must continue to be looked at with a critical
eye, as they do not usually tell the whole story.
Hope that helps ...
- Scott Gurst