IFComp scoring guidelines?
the all-original Constitutional Windup Boy
give up on a game before two hours are over (either because it's too
hard, or because it's not fun) but still vote on it?
At one extreme, it seems unfair to judge a game after only five minutes,
but I've already run into one entry that I was deeply reluctant to play
any longer than that.
a
James Mitchelhill
The two hour limit is a maximum. You don't have to play games for this long
if you'd prefer not to.
If a game makes you want to quit after five minutes, then that's certainly
something that should be reflected in your score. As long as you feel
you've given the game a fair chance, then there's no problem with quitting,
rating low and moving on. That's certainly what I do.
To be honest, I don't think anyone plays every game for the whole two
hours. It'd certainly make the prospect of playing some of the games a lot
more chilling than it already is.
--
James Mitchelhill
ja...@disorderfeed.net
http://disorderfeed.net
Stephen Granade
The "two hour" rule is: Judges must base their judgement of each game
on at most the first two hours of play. Note the "at most". There's no
definition of what exactly your 1-10 scale means; all judges decide
that for themselves. If you know how you feel about a game after five
minutes, you certainly may judge it at that point.
Stephen
--
Stephen Granade
stephen...@granades.com
Rifflesby
unofficial ratings, since I'm in the comp):
http://www.sidneymerk.com/comp05/scoring.shtml
Rifflesby
David Thornley
the all-original Constitutional Windup Boy <aa...@eecs.harvard.edu> wrote:
>I've never voted in the IF Comp before-- is it considered bad form to
>give up on a game before two hours are over (either because it's too
>hard, or because it's not fun) but still vote on it?
>
Otherwise, those of us who hope to judge would be committing ourselves
to spending a whole lot of time, including hours spent on material we
found offensive, or spending time playing and replaying somebody's
idea of a joke that's exhaustible in ten minutes.
>At one extreme, it seems unfair to judge a game after only five minutes,
>but I've already run into one entry that I was deeply reluctant to play
>any longer than that.
>
That would be a good candidate for a low score, then, I'd think.
I believe that, by entering a game in the Comp, the author is requesting
time and attention. If the game fails to merit this, rate it as it
deserves.
--
David H. Thornley | If you want my opinion, ask.
da...@thornley.net | If you don't, flee.
http://www.thornley.net/~thornley/david/ | O-
d...@pobox.com
My opinion: you're the the judge; wield that power as you see fit. If
you think penalising games that repel you after 5 minutes is reasonable
then do so. Personally, I think it's completely reasonable.
I woke up with a brilliant idea for scoring this year that I am sorely
tempted to implement:
My favourite game: 10
My next favourite game: 9
My next favourite, after the 2 best: 8
...
and so on until all the remaining games all get a score of 1.
drj
Stebbins
d...@pobox.com wrote:
> My opinion: you're the the judge; wield that power as you see fit. If
> you think penalising games that repel you after 5 minutes is reasonable
> then do so. Personally, I think it's completely reasonable.
>
> I woke up with a brilliant idea for scoring this year that I am sorely
> tempted to implement:
>
> My favourite game: 10
> My next favourite game: 9
> My next favourite, after the 2 best: 8
> ...
> and so on until all the remaining games all get a score of 1.
>
> drj
Aww, now that's just plain mean. However, I myself have only played
two games so far, so it could be entirely possible that there are 35
games in this comp that deserve a 1. I do agree that 10's and 9's
should be awarded sparingly.
dwh...@gmail.com
d...@pobox.com wrote:
>
> I woke up with a brilliant idea for scoring this year that I am sorely
> tempted to implement:
>
> My favourite game: 10
> My next favourite game: 9
> My next favourite, after the 2 best: 8
> ...
> and so on until all the remaining games all get a score of 1.
>
> drj
What happens if there's only one decent game in the Comp? You're giving
terrible games scores of 9, 8, 7, etc. Your 'brilliant' idea also means
that 35 games in the Comp, irrespective of their quality, get a rating
of 1 from you. Now if this was the worst Comp in history, there's still
no way 35 of the games would deserve the lowest possible rating.
d...@pobox.com
On Oct 9, 6:40 pm, dwh...@gmail.com wrote:
> d...@pobox.com wrote:
>
> > I woke up with a brilliant idea for scoring this year that I am sorely
> > tempted to implement:
>
> > My favourite game: 10
> > My next favourite game: 9
> > My next favourite, after the 2 best: 8
> > ...
> > and so on until all the remaining games all get a score of 1.
>
> What happens if there's only one decent game in the Comp? You're giving
> terrible games scores of 9, 8, 7, etc. Your 'brilliant' idea also means
> that 35 games in the Comp, irrespective of their quality, get a rating
> of 1 from you. Now if this was the worst Comp in history, there's still
> no way 35 of the games would deserve the lowest possible rating.
My suggestion was perhaps more tongue-in-cheek than you realised, my
apologies. Your concerns are valid and noted. I hadn't actually
thought that there might be fewer than 9 games worthy of a score of 2
or more. The flip-side, that there would be more than 9 games worthy
of a score of 2 or more did, of course, occur to me. I think some of
the thinking behind my briiilliant idea was that 40-odd games was
rather a lot to review (even if I am happy dismissing some of them
after... let's see, so far my shortest play attempt was... 16 minutes),
and how could I wield my voting power so as to encourage fewer, but
better, games in future competitions?
There is absolutely no sense in which I feel I should a game the score
it "deserves". I give it the score I give it. It's my vote, and my
score.
drj
Gene Wirchenko
[snip]
>There is absolutely no sense in which I feel I should a game the score
>it "deserves". I give it the score I give it. It's my vote, and my
>score.
Upthread, you claim (indirectly) to be a judge. A judge judges.
Sincerely,
Gene Wirchenko
Computerese Irregular Verb Conjugation:
I have preferences.
You have biases.
He/She has prejudices.
Rifflesby
> There is absolutely no sense in which I feel I should a game the score
> it "deserves". I give it the score I give it. It's my vote, and my
> score.
?
Isn't the score you give a game, the score you think it deserves? I
mean, I have to assume you're not rolling dice, or perversely giving 1s
to games you really liked...
--Riff
dwh...@gmail.com
d...@pobox.com wrote:
>
> There is absolutely no sense in which I feel I should a game the score
> it "deserves". I give it the score I give it. It's my vote, and my
> score.
>
No matter how many times I look at that, it still doesn't make any
sense to me.
The score a game "deserves" is the score you think it should have based
on good/bad it is. Are you saying it's wrong to score a game based on
its quality. How else would you score a game? Toss a coin and heads
it's bad and tails it's good? Draw straws? Close your eyes and type a
number on the keyboard totally at random?
Rikard Peterson
"Rifflesby" <riff....@gmail.com> wrote:
The difference is that some people base their score completely on their
subjective experience (I guess drj is in that category) while others use
try in various ways to give an objective score. The times I've voted,
it's been the subjective style, since I couldn't possibly come close to
objective results even if I tried. And I think it's the playing
experience that matters anyway.
Note that I'm not claiming the people trying to be (more or less)
objective to be bad judges. Only different.
Rikard
James Mitchelhill
> In article <1160628309....@e3g2000cwe.googlegroups.com>,
> "Rifflesby" <riff....@gmail.com> wrote:
>
>> Isn't the score you give a game, the score you think it deserves? I
>> mean, I have to assume you're not rolling dice, or perversely giving 1s
>> to games you really liked...
>>
>> --Riff
>
> The difference is that some people base their score completely on their
> subjective experience (I guess drj is in that category) while others use
> try in various ways to give an objective score. The times I've voted,
> it's been the subjective style, since I couldn't possibly come close to
> objective results even if I tried. And I think it's the playing
> experience that matters anyway.
Yeah. My problem with objective scoring systems is that it's not how good a
game's prose is, or how clever the puzzles, or even how well implemented it
is that matters. These things are just tools for authors to use. You can
rate the quality of the tools a game uses on a reasonably objective scale,
but it's much harder to objectively rate how effectively a game uses these
elements to produce the player's experience.
And it's the effectiveness that's the interesting bit. Even a game with
rather poor individual elements can use those elements together very
effectively. This doesn't happen very often, but it does happen. And it's
much more common that a game that is well implemented and well written can
add up to nothing very worthwhile.
quic...@quickfur.ath.cx
> On Thu, 12 Oct 2006 21:39:15 +0200, Rikard Peterson wrote:
> > The difference is that some people base their score completely on
> > their subjective experience (I guess drj is in that category) while
> > others use try in various ways to give an objective score. The times
> > I've voted, it's been the subjective style, since I couldn't
> > possibly come close to objective results even if I tried. And I
> > think it's the playing experience that matters anyway.
>
> Yeah. My problem with objective scoring systems is that it's not how
> good a game's prose is, or how clever the puzzles, or even how well
> implemented it is that matters. These things are just tools for
> authors to use. You can rate the quality of the tools a game uses on a
> reasonably objective scale, but it's much harder to objectively rate
> how effectively a game uses these elements to produce the player's
> experience.
Well, the competition having only a single number to represent the
quality of the game doesn't help in this respect. After judging the
first few games, I realized that my perception of a game's "worth" can't
possibly be represented accurately by a single number, so I've come up
with a crude 6-category system that hopefully provides a better picture
of my playing experience. Of course, I still use a final number for the
purposes of the competition, but my plan is to post the category scores
once the judging period is over.
The categories I have are writing, setting, story, puzzles, technical,
and Brownie points, each scored out of 10 (admittedly still mechanical,
but at least I have 6 degrees of freedom with which to express my
impression of the game). Since some categories may not apply to every
game, I've decided that they can be freely omitted depending on the
game, and do not need to add up (via average or any other scheme) to the
"official" judging score.
> And it's the effectiveness that's the interesting bit. Even a game
> with rather poor individual elements can use those elements together
> very effectively. This doesn't happen very often, but it does happen.
> And it's much more common that a game that is well implemented and
> well written can add up to nothing very worthwhile.
[...]
My Brownie points category accounts for this, somewhat. It's basically a
score representing how much the game pleased me regardless of the other
factors. So far, I've found a number of games where the individual
elements aren't quite the best, but I liked the concept so much that I
gave them lots of Brownie points. A few games had very good scores in
the other areas, but fail to appeal to me, so they didn't earn any
Brownie points.
QF
--
Those who have not appreciated the beauty of language are not qualified
to bemoan its flaws.
Michael Martin
time has been unjustifiably wasted has a maximum score of 3, while a
game that has at least *something* about it that is unreservedly
completely great in my book is going to be getting at least a 6.
(Games that manage to do both at once tend to get 4s.)
By the end of the comps I'm judging, though, my metric has really
become "Am I comfortable giving these games the same ranking?" and
shifting games around until my answer becomes "yes".
--Michael
d...@pobox.com
Well, you mentioned "deserve" first. You said "there's still
no way 35 of the games would deserve the lowest possible rating". Now
either that's true and so you admit that "deserve" means something
other than "what I choose to score it"; or it's false because "deserve"
just means "whatever score I feel like giving them" and I might easily
decide to give 35 games a score of 1. Or the law of excluded middle
doesn't apply.
I think there probably are valid senses in which games deserve scores
that are higher or lower than the ones they get (from me, say). Does a
largely puzzleless work that is well written and structured and will
probably be well received by the IF community "deserve" to get a score
of 1 just because I decided I didn't like puzzleless IF the day I
scored it? Umm, actually I don't know. But I admit there is probably
quite a bit of arbitrariness in my scoring.
This is possibly related to Gene Wirchenko "judgement" post as well. I
am a judge, but unlike real judges I haven't been selected by society
because of my demonstrated ability to make fine judgements that settle
disputes sensibly and fairly (unlike _real_ judges. Yeah), I've been
selected by my ability to select options from 5 or more pull-down menus
on a web page. Does that mean I don't apply my judgement when scoring?
No, of course I apply my judgement. But I probably don't weigh the
pros and cons of each game's score as much as if I was a specially
appointed judge; I'm almost certain I'm more whimsical.
It probably didn't help that I had a cold when I posted that, but I can
hardly claim to be more coherent now.
drj
Eric Eve
news:1160913205.5...@i3g2000cwc.googlegroups.com...
>> The score a game "deserves" is the score you think it should have
>> based
>> on good/bad it is. Are you saying it's wrong to score a game
>> based on
>> its quality. How else would you score a game? Toss a coin and
>> heads
>> it's bad and tails it's good? Draw straws? Close your eyes and
>> type a
>> number on the keyboard totally at random?
>
> Well, you mentioned "deserve" first. You said "there's still
> no way 35 of the games would deserve the lowest possible rating".
> Now
> either that's true and so you admit that "deserve" means something
> other than "what I choose to score it"; or it's false because
> "deserve"
> just means "whatever score I feel like giving them" and I might
> easily
> decide to give 35 games a score of 1. Or the law of excluded
> middle
> doesn't apply.
Well, I obviously can't speak for David Whyld, but I should thought
"deserve" implies some concept of justice or fair play, and thus
excludes "whatever score I feel like giving them" if that's meant in
a purely arbitrary sense.
> I think there probably are valid senses in which games deserve
scores
> that are higher or lower than the ones they get (from me, say).
> Does a
> largely puzzleless work that is well written and structured and
> will
> probably be well received by the IF community "deserve" to get a
> score
> of 1 just because I decided I didn't like puzzleless IF the day I
> scored it? Umm, actually I don't know. But I admit there is
> probably
> quite a bit of arbitrariness in my scoring.
It would certainly offend my sense of fair play if you gave such a
game a score of 1 for such a reason. On the other hand it would seem
to me to be perfectly reasonably to score such a game a bit less
than a more puzzle-intensive game because you enjoyed the latter
more. Of course you're entitled to score games any way you like, but
what you're entitled do is not automatically what's fair and just,
and ISTM that the IF-Comp relies on the majority of judges at least
trying to be reasonably fair and just in order to arrive at a
meaningful result.
> This is possibly related to Gene Wirchenko "judgement" post as
> well. I
> am a judge, but unlike real judges I haven't been selected by
> society
> because of my demonstrated ability to make fine judgements that
> settle
> disputes sensibly and fairly (unlike _real_ judges. Yeah), I've
> been
> selected by my ability to select options from 5 or more pull-down
> menus
> on a web page. Does that mean I don't apply my judgement when
> scoring?
> No, of course I apply my judgement. But I probably don't weigh
> the
> pros and cons of each game's score as much as if I was a specially
> appointed judge; I'm almost certain I'm more whimsical.
This seems to make a rather different claim from saying that you
simply award scores just as you happen to feel like. I think what
you're actually saying is that you don't start out with a set of
formal criteria (so many points for good spelling, so many for
bugless implementation, so many for convincing NPCs or a decent plot
etc.) and then arrive at a final score by totting up the sub-points
for each one. And that's fair enough (I don't work that way either).
But from what you're saying it sounds like you're operating with
some idea of what constitutes a good game, as well as what you
happen to enjoy, even if you haven't consciously formulated it, and
that in practice both elements are factoring into the score you
arrive at (but maybe I'm putting words into your mouth here).
In practice it would be nigh on impossible to exclude a subjective
element from scoring: most judges will probably want their scores to
be some reflection of how much they enjoyed the various games, and
different people will inevitably enjoy different things. If that's
all you're saying, I'd have thought it was relatively uncontentious.
-- Eric
d...@pobox.com
On Oct 15, 1:35 pm, "Eric Eve" <eric....@NOSPAMhmc.ox.ac.uk> wrote:
> Well, I obviously can't speak for David Whyld, but I should thought
> "deserve" implies some concept of justice or fair play, and thus
> excludes "whatever score I feel like giving them" if that's meant in
> a purely arbitrary sense.
Well, yeah, me too. So it seems to me that the more whimsical I am the
more likely I am to give a game a score that it doesn't deserve. I'm
okay with that, I don't feel honour bound to do justice to the game I
score.
>
> > I think there probably are valid senses in which games deserve
> scores
>
> > that are higher or lower than the ones they get (from me, say).
> > Does a
> > largely puzzleless work that is well written and structured and
> > will
> > probably be well received by the IF community "deserve" to get a
> > score
> > of 1 just because I decided I didn't like puzzleless IF the day I
> > scored it? Umm, actually I don't know. But I admit there is
> > probably
> > quite a bit of arbitrariness in my scoring.
> It would certainly offend my sense of fair play if you gave such a
> game a score of 1 for such a reason. On the other hand it would seem
> to me to be perfectly reasonably to score such a game a bit less
> than a more puzzle-intensive game because you enjoyed the latter
> more. Of course you're entitled to score games any way you like, but
> what you're entitled do is not automatically what's fair and just,
> and ISTM that the IF-Comp relies on the majority of judges at least
> trying to be reasonably fair and just in order to arrive at a
> meaningful result.
Well, then it seems that I might offend your sense of fairness and
justice.
I have an agenda here. I'd like to use my vote so as to encourage the
production of games that I like playing. If that means I score of 1 to
worthy works which fail to meet my criteria for fun, then so be it. If
that make me unjust, then so be it.
My wider point is that ones votes can be used to influence the
production of works. Perhaps not very much, but then, short of writing
sucessful works or key articles, how much influence were you expecting?
I would encourage people to think about what they want to do with this
influence.
I don't think that the competition does rely on even the majority of
judges being fair and just. I'm sure there are enough judges that each
person's individual whimsy gets averaged out and you end up with a sort
of justice of the average.
>
>
>
> > This is possibly related to Gene Wirchenko "judgement" post as
> > well. I
> > am a judge, but unlike real judges I haven't been selected by
> > society
> > because of my demonstrated ability to make fine judgements that
> > settle
> > disputes sensibly and fairly (unlike _real_ judges. Yeah), I've
> > been
> > selected by my ability to select options from 5 or more pull-down
> > menus
> > on a web page. Does that mean I don't apply my judgement when
> > scoring?
> > No, of course I apply my judgement. But I probably don't weigh
> > the
> > pros and cons of each game's score as much as if I was a specially
> > appointed judge; I'm almost certain I'm more whimsical.
> This seems to make a rather different claim from saying that you
> simply award scores just as you happen to feel like.
I don't think I claimed that. I said _if_ I did do that then some games
would get scores that they didn't deserve.
> I think what
> you're actually saying is that you don't start out with a set of
> formal criteria (so many points for good spelling, so many for
> bugless implementation, so many for convincing NPCs or a decent plot
> etc.) and then arrive at a final score by totting up the sub-points
> for each one. And that's fair enough (I don't work that way either).
Noted.
> But from what you're saying it sounds like you're operating with
> some idea of what constitutes a good game, as well as what you
> happen to enjoy, even if you haven't consciously formulated it, and
> that in practice both elements are factoring into the score you
> arrive at (but maybe I'm putting words into your mouth here).
>
> In practice it would be nigh on impossible to exclude a subjective
> element from scoring: most judges will probably want their scores to
> be some reflection of how much they enjoyed the various games, and
> different people will inevitably enjoy different things. If that's
> all you're saying, I'd have thought it was relatively uncontentious.
I think all I was saying was that I didn't think it was bad of me to
give score a game 1 even if someone else thought it deserved more. On
the flip side I'm likely to give at least one game 10, even if I think
that it wouldn't get a 10 if it were part of a larger competition
(heaven forbid!) that included past and possible future works. That
is, even if it didn't deserve a 10. I'm going to do that because
otherwise it seems that my "dynamic range" is wasted. If I didn't give
my favourite game a 10 then surely I could increase my favourite game's
chance of being #1 by giving it a 10? And if that is so, then surely I
should do it? Does that make my 10 unjust?
I think I'd like to score as if my notions of "good game" and "what I
enjoy" were really quite close. In other words, I'd like to increase
the subjective element.
Basically, I think I'm saying "I'm going to vote however I like", is
that such a contentious position?
Does it help to know that I don't try and write reviews the same way?
See, for example, my City of Secrets review for SPAG:
http://www.sparkynet.com/spag/c.html#cityof , where I try and give a
balanced and informed opinion on a game that I admit is not my cup of
tea.
drj
Michael Martin
> I have an agenda here. I'd like to use my vote so as to encourage the
> production of games that I like playing. If that means I score of 1 to
> worthy works which fail to meet my criteria for fun, then so be it. If
> that make me unjust, then so be it.
Well, given the way comps tend to go, I would suggest that "worthy
works which fail to meet a criteria for fun" would end up being, at
minimum, 2, because every year there's usually about half a dozen
*unworthy* works which *also* fail to meet a criterion for fun. (Or
which are totally broken or unplayable, or the game is unsolvable on
its own and the provided walkthrough doesn't actually work, or it's so
short and sparse that it's basically a SpeedIF, etc. etc. etc.) I tend
to reserve "1" scores for "What were you *thinking*, submitting this to
the Comp at all/in this state" games.
> I don't think that the competition does rely on even the majority of
> judges being fair and just. I'm sure there are enough judges that each
> person's individual whimsy gets averaged out and you end up with a sort
> of justice of the average.
That's about right, I'd say.
> I think all I was saying was that I didn't think it was bad of me to
> give score a game 1 even if someone else thought it deserved more.
[snip]
> Basically, I think I'm saying "I'm going to vote however I like", is
> that such a contentious position?
I think most of us are imagining someone trying to "wield influence" as
saying something like (to use a past comp as an example) "I hate
dragons! Any game with dragons in it gets a 1! The Colour Pink and
Ninja 2 are games of equal quality!", and yeah, I'd say the surrounding
community would be justified in giving someone who claimed that a Funny
Look, because it's pretty immediately obvious that these are games of a
different caliber -- one could see arguments for why they deserve the
same score, but I think most of us would expect that if both were given
1s, they were being given 1s for different reasons. (The Plague: Redux
got a bunch of 1s for not working on several versions of the ADRIFT
runner, for instance.)
> That is, even if [the best game] didn't deserve a 10. I'm going to do that because
> otherwise it seems that my "dynamic range" is wasted.
Lots of people give out Exactly One Ten each year, for the best game of
the comp, so that part is probably not going to be terribly
controversial. (Exactly One Ten also gives you a range of 9 for
everything else, which is handily divisible into good/mediocre/bad.
You could in theory instead do Exactly One One, but there's usually
far, far, far too much competition for the bottom spot.) I think it
was the "Only nine games don't get 1s" implication that's raising
hackles. I can imagine a comp where there are so few seriously
intended entries that this would be sensible, but that hasn't happened
yet in the entire Comp history. (I can also imagine a Comp where
everyone was sufficiently polished that I had to give everyone at least
a 4 by the scales of previous years; that would probably require a
rethinking of my scale. Just one prank game or broken game would
require the extra space though, so I'm not overly concerned.)
But, uh, yeah. "I vote how I want" is uncontroversial, but if you say
this publically, you have to expect that we're going to pressure you to
want the same things we do.
--Michael