voting/rating techniques
Curt Siffert
I've got a question about voting. I know this subject has
been covered, abused, and beaten with heavy objects in past
years, but I'm still confused about one thing.
Some people profess a fondness for the strategy of grading
on a curve. That is, giving a 10 to their favorite game, a
1 to their least favorite, and filling in the gaps from there.
I think Cadre's strategic-voting practice (artificially inflating
the score of a game that he sorta likes but is sure will be
under-appreciated by others) is a variant of this. It also
makes me wonder how much money he loses on the stock market. :)
The problem is that these people usually only a play a subset
of the games. It's my contention that this mucks up
the ranking of the games. I think my thesis is that this
strategy only works if you play/rate ALL the games.
I can just imagine cases where it would affect the standings
at the end - where one game in fourth place is given a high
score by a person who wouldn't have rated it so highly if he
had played the one in third, resulting in the placement order
being switched. I was thinking this wouldn't matter so much
if we figured the results purely through rankings than
calculating/averaging the actual number ratings, but it turns
out it's vulnerable to the same problem.
So - even if this ends up purely theoretical - is there a way
to calculate the votes that weigh all the rating sheets depending
on which games they do and don't have included? I suppose there's
weighting the rating sheets depending on how many games they rate,
but that doesn't seem quite fair... or maybe it's judging each game
by how many games the person liked better than it, and seeing if
those other games are also on the sheet of other people who rated
the same game.
Anyway, is there any chance we can be given the rating sheets
(without name/email address) at the end so we can experiment
with our own figuring algorithms?
Danks and Murkies,
Curt
Norman Perlmutter
<sif...@museworldSPAMSUCKS.com> wrote:
>The problem is that these people usually only a play a subset
>of the games. It's my contention that this mucks up
>the ranking of the games. I think my thesis is that this
>strategy only works if you play/rate ALL the games.
I agree, with the provision that it should be acceptable if
they can't play a few games because of platform problems.
Norman Perlmutter
Adam Thornton
We've had this discussion here before.
To summarize:
This may be feasible with a dozen games.
In a field of 52 games, where you're supposed to have up to 2 hours to
evaluate each one, well, that's 2-1/2 work weeks there. Many of us have
real jobs and are simply not going to be able to do that. The consensus
is, best to play what you can--although it helps if you use a
randomizer, like Comp01.z5, so that people don't all end up playing
games by the ten authors they've heard of--and rate those.
Me, I'm guilty of, um, playing games by people I've heard of this year.
But then, I'm not eligible to vote, so it doesn't really matter.
Adam
OKB -- not okblacke
>Some people profess a fondness for the strategy of grading
>on a curve. That is, giving a 10 to their favorite game, a
>1 to their least favorite, and filling in the gaps from there.
>I think Cadre's strategic-voting practice (artificially inflating
>the score of a game that he sorta likes but is sure will be
>under-appreciated by others) is a variant of this. It also
>makes me wonder how much money he loses on the stock market. :)
>
>The problem is that these people usually only a play a subset
>of the games. It's my contention that this mucks up
>the ranking of the games. I think my thesis is that this
>strategy only works if you play/rate ALL the games.
The consensus seems to be that almost no one is going to play every game,
and so we all rely on a few equalizers (randomness provided by Comp01.z5,
platform differences, personal preferences/aversions) to balance it out.
>I can just imagine cases where it would affect the standings
>at the end - where one game in fourth place is given a high
>score by a person who wouldn't have rated it so highly if he
>had played the one in third, resulting in the placement order
>being switched.
I think that, in this situation, worrying about how you rated the games
you did play as opposed to how you might have rated the ones you didn't isn't
really going to be helpful. We seem to have enough judges that no one vote set
like the one you describe is going to actually be a deciding factor in the
final standings any more than it should be.
>So - even if this ends up purely theoretical - is there a way
>to calculate the votes that weigh all the rating sheets depending
>on which games they do and don't have included? I suppose there's
>weighting the rating sheets depending on how many games they rate,
>but that doesn't seem quite fair... or maybe it's judging each game
>by how many games the person liked better than it, and seeing if
>those other games are also on the sheet of other people who rated
>the same game.
Basically what a system like this is trying to do is simulate actual
voters opinions -- you're trying to guess how the game would have done if
everyone had rated it. This is not necessarily bad, except that many of those
who DO vote but DON'T vote on the game in question are going to resent their
votes being "synthesized" in this way. Put simply, there really isn't a way to
factor in what games Jack DID vote for without implicitly factoring in what
games Jill DIDN'T vote for.
I still have a little hope that maybe we'll stumble on some really good
formula that will really make everybody happy, but I'm basically content with
what we have now. For one thing, we have enough judges voting with enough
different schemes (curving, rigid point allocation, freewheeling emotional
response) that there's a fair amount of balance. Perhaps more importantly, the
current system is very simple.
I basically think of it like this: the winner of the Comp is not defined
as the most-liked game -- it's defined as the game with the highest average
score. At least with the current system we're clear on this. With some really
complex system, we'd be less clear on the mathematical basis, and it's a good
bet we STILL wouldn't agree that the new system would give 1st place to the
most-liked game.
--OKB (Bren...@aol.com) -- no relation to okblacke
"Do not follow where the path may lead;
go, instead, where there is no path, and leave a trail."
--Author Unknown
Adam Cadre
> Some people profess a fondness for the strategy of grading
> on a curve. That is, giving a 10 to their favorite game, a
> 1 to their least favorite, and filling in the gaps from there.
> I think Cadre's strategic-voting practice (artificially inflating
> the score of a game that he sorta likes but is sure will be
> under-appreciated by others) is a variant of this.
Did I say I do that? I don't think I do. I *have* in previous years
used the curve you mention, and I won't give more than a 3 or 4 to a
game I don't personally enjoy, no matter how well-executed it is, so
an 8 from me does count a tiny bit more than an 8 from someone who gives
6s and 7s to such games. But I don't think that's quite the same thing.
-----
Adam Cadre, Brooklyn, NY
http://adamcadre.ac
Curt Siffert
two years ago on the group. Definitely a hazy memory. ;-)
Sorry if I was off base.
Alexander Deubelbeiss
>
>Some people profess a fondness for the strategy of grading
>on a curve. That is, giving a 10 to their favorite game, a
>1 to their least favorite, and filling in the gaps from there.
"Some people"... so what's the alternative? Giving a hypothetical
10 to their favourite game ever and a hypothetical 1 to the worst
they've ever played, and _then_ filling in the blanks?
OKB -- not okblacke
>"Some people"... so what's the alternative? Giving a hypothetical=20
>10 to their favourite game ever and a hypothetical 1 to the worst=20
>they've ever played, and _then_ filling in the blanks?
Um, yes. Or doling out and taking away points based on specific things.
Good writing=+1 point, cool story=+1 point, bug=-1 point, that kind of thing.
Or just going with your gut reaction ("this one feels like a 7").
Paul O'Brian
Actually, there are plenty of alternatives. The strategy Curt describes
(as I read it) posits that one game gets a 10, one game gets a 1, and the
rest get placed inbetween. The strategy you describe is close to my own
approach -- I'd give a 10 to a game that seemed completely flawless to me,
and I've never yet given one, though I tend to rank out to .1 of accuracy
and round up 9.5 and above to 10 for my actual votes.
--
Paul O'Brian obr...@colorado.edu http://ucsu.colorado.edu/~obrian
FINALLY, an alternative to EXPENSIVE therapy! It's SPAG, the text
adventure magazine! Check it out at http://www.sparkynet.com/spag
Issue #26 has just arrived!
Richard Bos
> Alexander Deubelbeiss deub...@gmx.net wrote:
> >"Some people"... so what's the alternative? Giving a hypothetical
> >10 to their favourite game ever and a hypothetical 1 to the worst
> >they've ever played, and _then_ filling in the blanks?
>
> Um, yes. Or doling out and taking away points based on specific things.
> Good writing=+1 point, cool story=+1 point, bug=-1 point, that kind of thing.
> Or just going with your gut reaction ("this one feels like a 7").
A bit of both. I start with an off-hand value ("Oh, I like this one
quite a bit. Give it a 7."), and then I modify it according to whether
it has bugs, or good tricks, or anything like that. Even the
modifications are fairly random, though, not something as binary as
every bug costs a point.
Richard
Cedric Knight
This prompts me to ask whether people generally do try for a "curved" /
"Gaussian" distribution of their scores.
I'm scoring intuitively against ideal "1"s and "10"s as I go along. So
after 21 games, I haven't yet had a nine or ten (and only one 1), and am
still hoping to find one.
But at the end of play, I might decide to rank them so the best 5 get a
10, the next best 5 get a 9 and so on (that is instead of the usual
normal distribution) so that the midrange games are more differentiated
from one another. Wouldn't that mean that my vote is more likely to
affect the result, as the standard deviation of those
"continuous"/"random" scores would be bigger than a more "natural"
Gaussian distribution? And would that be a good thing, a bad thing, or
indifferent?
There's interest in the s.d. of votes for particular games, but not by
individual judges.
I'm sure Olympic committees must have to deal with this as well.
Cedric
David Thornley
Cedric Knight <ckn...@gn.apc.org> wrote:
>
>This prompts me to ask whether people generally do try for a "curved" /
>"Gaussian" distribution of their scores.
>
There's an unfortunate amount of 1s and 2s among about twenty
rated games, and the scores get scarcer as they get higher. So
far, I've awarded one 10, no 9, and one 8. I'm not deliberately
aiming at any distribution, except that I will always award
at least one 10 and one 1 in the end, to maximize the statistical
effects of my voting.
>I'm scoring intuitively against ideal "1"s and "10"s as I go along. So
>after 21 games, I haven't yet had a nine or ten (and only one 1), and am
>still hoping to find one.
>
Only one 1? Either you're luckier in your first-half randomization
than me or you have a much more lenient idea of what constitutes
wasting the judge's time than I do.
The problem with an ideal 10 is that competition games are
inherently limited in size, since the rules say rate them after
two hours at most. It's harder to write a really good small
game than a really good large one. If I were to write down ten
favorites, the majority would be too large to be good competition
games.
>But at the end of play, I might decide to rank them so the best 5 get a
>10, the next best 5 get a 9 and so on (that is instead of the usual
>normal distribution) so that the midrange games are more differentiated
>from one another. Wouldn't that mean that my vote is more likely to
>affect the result, as the standard deviation of those
>"continuous"/"random" scores would be bigger than a more "natural"
>Gaussian distribution? And would that be a good thing, a bad thing, or
>indifferent?
>
What do you mean, "affect the result"? Assume that most people do
some sort of Gaussian distribution; then you would be slightly
differentiating the midrange and not the ends. What most
people really care about is the top games, and that scheme
would reduce your influence there.
Suppose we have identical picks for best and twelfth best. We
both rate the best game as a 10, but I may be rating the twelfth
best as a 5 whereas you'd be rating it as an 8. This means that
I'm affecting the relative scores more than you.
Let's put it another way. Suppose that my favorite game is
your twelfth favorite, rated 8. Suppose yours is my twelfth
favorite, rated 5. My game gets an average score of 9, yours
of 7.5.
If you have a favorite, and you want to contribute as much as
possible to its placing high, rate it 10 and rate everything
else 1, including the ones you can't play (after all, you
can always say that those people deserve it for not allowing
you to play the game). Obviously, this means you have absolutely
no effect on how anything else scores relative to anything else,
which means that you're lumping in enjoyable games with the one
that displays disgusting images on your screen while it wipes
your hard drive. (Sorry, revealing which one that is would
violate the rule of not discussing specific games during the
voting period.)
Alternatively, if you're interested in giving a certain game
the lowest possible standing, you can give it a 1 and everything
else a 10 (well, you found absolutely nothing wrong with the
ones you couldn't play, right?).
>There's interest in the s.d. of votes for particular games, but not by
>individual judges.
>
What I'd like to see is sanitized listings of ratings so that
somebody else could do correlations and analysis of variance
and write up the interesting stuff for me. (Let's be
realistic; I wouldn't have the time to do it myself.)
>I'm sure Olympic committees must have to deal with this as well.
>
People have studied voting systems for a *long* time, and the
conclusion is that there can be no perfect one. All voting
schemes will fail to reward the candidate most liked, under
certain patterns of voter preference.
My opinion is that telling people to rate them from 1 to 10
and then averaging the votes has the advantage of being simple
and not being demonstrably worse than any other system.
--
David H. Thornley | If you want my opinion, ask.
da...@thornley.net | If you don't, flee.
http://www.thornley.net/~thornley/david/ | O-
Fred M. Sloniker
wrote:
>People have studied voting systems for a *long* time, and the
>conclusion is that there can be no perfect one. All voting schemes
>will fail to reward the candidate most liked, under certain patterns
>of voter preference.
I'm aware that there's no 'perfect' voting system (except, I imagine,
in the case of exactly two things to choose between, in which case
simple majority should suffice), but this seems an odd statement to
make. If there's no way to score a vote that's guaranteed to 'reward
the candidate most liked' under all circumstances... how do you *know*
what the candidate most liked is? (If you knew, after all, you'd
simply say that was the winner of the vote...)
OKB -- not okblacke
>I'm aware that there's no 'perfect' voting system (except, I imagine,
>in the case of exactly two things to choose between, in which case
>simple majority should suffice)
Hardly. You think I'm going to be happy if I pick A and B wins? Maybe it
will indeed choose the one more people like, but it won't be perfect.
>If there's no way to score a vote that's guaranteed to 'reward
>the candidate most liked' under all circumstances... how do you *know*
>what the candidate most liked is? (If you knew, after all, you'd
>simply say that was the winner of the vote...)
Exactly.
Fred M. Sloniker
okblacke) wrote:
>>I'm aware that there's no 'perfect' voting system (except, I
>>imagine, in the case of exactly two things to choose between, in
>>which case simple majority should suffice)
>Hardly. You think I'm going to be happy if I pick A and B wins?
So the criterion for a 'perfect' voting system is that your personal
choices always win? So how's that world dominance plan coming along?
n_n
>>If there's no way to score a vote that's guaranteed to 'reward
>>the candidate most liked' under all circumstances... how do you
>>*know* what the candidate most liked is? (If you knew, after all,
>>you'd simply say that was the winner of the vote...)
>
> Exactly.
Hm. Perhaps my point wasn't clear. If there's no way to know who the
candidate most liked really *is*, how can you know the voting's
failing to select the candidate most liked? Contrariwise, if you
don't know it's failing, how do you know there's no way to know who
the candidate most liked is? It seems like a paradox to me, but
perhaps I'm not understanding the issue...
Cedric Knight
news:cY0z7.5072$Ra3.1...@ruti.visi.com...
> In article <3bcb6114$0$236$cc9e...@news.dial.pipex.com>,
> Cedric Knight <ckn...@gn.apc.org> wrote:
> >
> >This prompts me to ask whether people generally do try for a "curved"
/
> >"Gaussian" distribution of their scores.
> >
> I don't. In this comp, it's more of the right side of a distribution.
> There's an unfortunate amount of 1s and 2s among about twenty
> rated games, and the scores get scarcer as they get higher. So
> far, I've awarded one 10, no 9, and one 8. I'm not deliberately
> aiming at any distribution, except that I will always award
> at least one 10 and one 1 in the end, to maximize the statistical
> effects of my voting.
That seems to be a kind of more limited version of what I suggested. As
one possibility, there are a lot of competent but unremarkable works
bunched together as 5s and 6s; but to maximise the effect of a single
vote (and the s.d. of games within that individual vote), one would
divide equally between 1s and 10s (which of course was not what I'm
proposing to do myself).
But I think that answers my question, thanks.
[snipped some of the stuff I agree with]
> What do you mean, "affect the result"? Assume that most people do
> some sort of Gaussian distribution; then you would be slightly
> differentiating the midrange and not the ends. What most
> people really care about is the top games, and that scheme
> would reduce your influence there.
That is true, although surely it is also possible that the top games of
the comp are merely mid(-to-high) range for me, so it may not reduce my
influence that much. Authors may also care about whether their game
finishes in the top half or bottom half of the results. A scoring
strategy that differentiates average-range games will have more effect
*for the average game* (rather than the top 3 or so which many people
play all of), because on average a game must be pretty, well... average.
> If you have a favorite, and you want to contribute as much as
> possible to its placing high, rate it 10 and rate everything
> else 1, including the ones you can't play (after all, you
> can always say that those people deserve it for not allowing
> you to play the game). Obviously, this means you have absolutely
> no effect on how anything else scores relative to anything else,
> which means that you're lumping in enjoyable games with the one
> that displays disgusting images on your screen while it wipes
> your hard drive. (Sorry, revealing which one that is would
> violate the rule of not discussing specific games during the
> voting period.)
I liked that one. I gave it an eight. (You're joking, I hope).
> My opinion is that telling people to rate them from 1 to 10
> and then averaging the votes has the advantage of being simple
> and not being demonstrably worse than any other system.
I agree, and don't mean to suggest there's anything at all wrong with the
current system.
CK
OKB -- not okblacke
>choices always win? So how's that world dominance plan coming along?
>n_n
My criterion for a perfect voting system is one where everybody's personal
choices always win. (Or, actually, where everybody is always 100% happy with
the results, which might not necessarily be the same thing.)
>Hm. Perhaps my point wasn't clear. If there's no way to know who the
>candidate most liked really *is*, how can you know the voting's
>failing to select the candidate most liked? Contrariwise, if you
>don't know it's failing, how do you know there's no way to know who
>the candidate most liked is? It seems like a paradox to me, but
>perhaps I'm not understanding the issue...
I understood your point. My point is that this is indeed the basic
problem with choosing a voting system. There needs to be a balance between an
intuitive feeling that such-and-such ranking is the appropriate one, and trying
to mathematically describe that intuition. If we could do this, there'd be no
question of a perfect voting system.
So, yeah, I agree completely! :-)