[ANN] Results of the 2003 Beta-Test Comp! [long]
Jessica Knoch
A hush falls over the crowd. It's time to announce the awards for
the 2003 Beta-Test Competition! The announcer taps on the mike and
clears his throat.
"As you know," he begins, "the test subject this year was titled
Undersea Adventure. Eight entrants tested the game, wrote up bug
reports, and waited for the results. After hours of painstaking
review and evaluation, we finally have the results." He coughs, then
opens the envelope. He removes a small square of paper from it.
"And the winner for Best Organization of A Beta-Test Report goes
to... Xander Deubelbeiss!" The crowd cheers.
The announcer takes out another square. "The winner for Most
Thorough Listing of Missing Synonyms and Good Suggestions goes to...
Esa A. Peuha!" The crowd roars.
Another square. "And the coveted award for Most Attempts to Flatter
a Judge goes to... Jennifer Maddox!" The crowd laughs.
"The winner for Best Criticism of the Plot goes to... Mike Jones!"
The crowd applauds.
"Best Overall Criticism on All Aspects of the Game and Best Comments
on Grammar!" the announcer exclaims. "Goes! To! Adam Thornton!" The
crowd shouts its approval.
"The winner for Best Remarks on Puzzles and Most Bugs Caught..." The
crowd gasps. "Quintin Stone!" Cheers erupt from the audience.
"The award for Finding the Most Serious Bugs That No One, Yes, No
One Else Mentioned, goes to... Søren Løvborg!" The crowd oohs and
aahs.
"And finally..." A drumroll sounds from behind the curtains. "The
winner of the Best Beta-Test Report for the 2003 Beta-Test
Competition...
"is...
"Matthew A. Murray!"
The crowd goes wild!
---------------------
And now, all of my notes and other comments. If all you wanted to
know was who won, well, you've seen it. The rest of this is just
fluff. But it's good fluff, and it's my Comp, so I can write however
much I want.
First of all, thanks to everyone who entered. I can't tell you how
hard it was to select a winner, because every report had something
really important. The truth of the matter is, a game needs multiple
beta-testers! Everyone found different things, and of all the bugs
and suggestions that got reported, no one tester reported more than
about 50% of them, which was a pretty surprising number. Still,
there were a few things that everyone noticed and everyone
mentioned, like needing to call the fabric shapes, well, "fabric,"
the fact that a whole room description was missing, and the fact
that the spikes showed up in inventory as "an iron spikes."
I admit: I did include some bugs on purpose (like the missing room
description). However, the number of things that were NOT
intentional rather surprised me. For instance, did you know that you
can find the iron bar over and over again? Or that the bowl can be
emptied out over and over again, an endless source of weird mushy
sand? Anyhow, I included some bugs on purpose because I was writing
the mini-game specifically for this comp. I was also worried about
taking up too much of the entrants' time, so I didn't want to make
the game too long. Because of this, I cut out several ideas for
other puzzles and other objects. The consensus from the people who
entered seems to be divided: some said the game was a good length
for its purpose, while others said they would have preferred a
longer, slightly more polished game. I think that if I get to do
this again, probably next year same time, I would use a longer game
in a more advanced stage of testing. Fair warning!
Now, I'll get even more nitty-gritty from here on out, just in case
anyone is interested. First some comments about what I was looking
for, and then a few comments about each of the different reports.
I had mentioned four different criteria for judging in my
announcement: Organization, Clarity of points, Thoroughness, and
Ability to criticize. As I went over the reports, I realized that
the criticism came in several parts, each important in its own
right: comments on the plot and back story, comments on the puzzles
and their difficulty, and comments on overall game design.
Thoroughness is easy enough to judge, since I made a list of all the
bugs. Clarity was pretty easy, too, and not a very large range
between the highest and lowest entries in this category. Either I
had a few questions, or everything was clear (or, in one case,
everything was spelled out extra careful). Organization was a little
more tricky to judge. After all, even if the report was just laid
out in the order that the tester played the game, that's still
*some* organization. Better was to separate the bugs into different
categories, but best of all was to list them in order of priority,
from proofreading items that must be fixed, to programming errors
that need to be fixed, to "should be added," and then "could be
added." I noticed also I liked when a report had a few general
comments or an introduction to the report at the beginning.
So that, in a nutshell, is what I was looking for. Now on to
specific reports, in reverse alphabetical order.
Entrant: Adam Thornton
Organization: Pretty good, divided into excerpts from gameplay and
more general comments.
Clarity: Fine.
Thoroughness: Good - about 45% of overall bugs.
Ability to Criticize: Excellent. This is where he shines.
"Yourslef" caught: Yes
One of the few reports to comment on player motivation (or lack
thereof). In fact, Mr. Thornton does a great job of commenting on
the puzzles, the plot, the overall game design, you name it. At the
same time, his report is encouraging while staying very to-the-point
and direct. Also, Mr. Thornton doesn't hesitate to attack all parts
of spelling and grammar mistakes, and even things that aren't
mistakes but could be improved. I had to look up whether he was
right about "smokey" being incorrect, but he was right - it's
"smoky." So, excellent commentary on lots of important aspects. Mr.
Thornton would be a great addition to any beta-testing team.
Entrant: Quintin Stone
Organization: Impressions first, then in order of the transcript.
Clarity: Good.
Thoroughness: Very good - a little over 50% of all bugs.
Ability to Criticize: Good. Lots of puzzle comments.
"Yourslef" caught: Yes
Mr. Stone introduces his report quite well, with overall impressions
and general coments. He is also one of the most thorough when it
comes to finding bugs and problems in the game, from outright
problems to suggestions for improvements. The tone of his report is
friendly and very helpful, which makes it easier to read :). Not as
many comments about the plot as some, but plenty of good commentary
about the puzzle difficulty and things that might be changed. An
excellent tester to have!
Entrant: Esa A E Peuha
Organization: First bugs by location, then general suggestions for
improvements.
Clarity: Very clear.
Thoroughness: Very, very thorough, especially with missing synonyms
and verbs. About 50% of total bugs.
Ability to Criticize: Some.
"Yourslef" caught: Yes
This entry doesn't have as many comments on puzzles, but it is an
invaluable tool in trying to figure out which synonyms to add. And
I'm not talking about just "need more synonyms for this," either.
Here's a sample of words that should be referrable in one of the
rooms:
Largest Room [rock]
- should be referrable as "room" "doorway" "opening" "passage"
"tunnel"
"doorways" "openings" "passages" "tunnels" "cavern"
And in addition to all of these lists, the entry still manages to
comment on other bugs! Quite a feat. This sort of thing comes in
very handy when testing a game for real. And last but not least, a
lot of suggestions are included that some testers wouldn't mention,
such as including an online map, replacing the standard compass
objects with "directions," a call for a list of amusing things to
do, and a request for hints instead of just a walkthrough. These are
great suggestions.
Entrant: Matthew A. Murray
Organization: Good categories, like Typos vs. Story Issues vs. Bugs
vs. General Comments.
Clarity: Very, very good.
Thoroughness: Okay - about 36% of total bugs.
Ability to Criticize: Great, though more on plot than puzzles.
"Yourslef" caught: Yes
This is a very helpful report, from beginning to end. Mr. Murray
doesn't skimp on words and explanations of what he thinks, nor does
he mind sharing suggestions for how to change it.For instance, where
some might just say, "Add a response for xyzzy," Mr. Murray spends a
short paragraph explaining why it would be a good thing. But don't
think that this report bought its first-place prize with verbage.
There's lots to like :). The tone is friendly and chatty, which
makes it all go down easier, and there's quite a lot of grammar
help. While it doesn't have as many different bugs as some of the
other reports, the ones it does mention are basic, important ones to
bring up. A very solid beta-test report.
Entrant: Jennifer Maddox
Organization: In order of location.
Clarity: Um, not the clearest :).
Thoroughness: Okay - about 35% of total bugs.
Ability to Criticize: Somewhat.
"Yourslef" caught: Yes
Ms. Maddox doesn't have a lot in the way of introductory comments,
which is okay. By the way, I should mention that Ms. Maddox was the
only real beta-tester of my entry into the 2002 Annual Comp. I
strong-armed her into entering this competition because I wasn't
sure how many people I'd have, and I wanted at least two entries.
Anyway, while the report was organized fine, there was a bit of
confusion as to what some of the comments meant. In a real
beta-testing situation, the author would of course write back and
ask for clarification, but since this is a comp, all I had to go on
was the initial report. As an example, the comment from the Heat
room says "It's an oven! I've got a bowl!" That's it. From this I
assume that she wanted to be able to put the bowl in the oven, or
cook other items in the oven using the bowl, but nothing is spelled
out. Most of the report is much more clear than that. There are some
good comments on the different puzzles and puzzle difficulty, and a
few about the plot. My favorite aspect of this report was Ms.
Maddox's light-hearted tone, attempts to flatter the judge
notwithstanding.
Entrant: Søren Løvborg
Organization: Broken into categories.
Clarity: Good.
Thoroughness: Good - about 46% of total bugs.
Ability to Criticize:
"Yourslef" caught: Yes
This report starts with a nice intro, and then dives into the bugs
and problems in tidy categories. In fact, it does well in each
criteria of judging, making it a very solid bug report. And, in
addition to that, I was completely floored by two or three rather
bad bugs mentioned, bugs that no one else said a word about. For
instance, you can successfully "empty bowl onto cliff" when the bowl
is empty," or you can "empty bowl onto cliff" repeatedly. It doesn't
actually register in the code as emptied. The other biggie was that
you can repeatedly search the ledges, and repeatedly get the iron
bar. I'd like to say these were intentional oversights, but, um, I'd
be lying. Anyhow, the bottom line is that this would be a very
valuable tester to have on your team.
Entrant: Mike Jones
Organization: Good - overall comments, then broken up into
categories.
Clarity: Fine.
Thoroughness: Okay - about 32% of all the bugs.
Ability to Criticize: Great, especially comments about the plot.
"Yourslef" caught: Yes
Mr. Jones has a very straightforward way of laying out bugs and
problems, and is critical in all the right places. He explained what
he thought was wrong with the plot and the story, and listed out the
questions he would have liked answers to in the game (some of which
I'd considered and had answers to, but some I didn't!). Another
strong point of this report was the friendly way Mr. Jones had of
mentioning things that were right, as well as things that needed to
be fixed. Overall, great suggestions for changes.
Entrant: Xander Deubelbeiss
Organization: The best - prioritized.
Clarity: No trouble.
Thoroughness: Good - about 48% of total bugs.
Ability to Criticize: Pretty good - comments on all aspects.
"Yourslef" caught: Yes
An excellent report, very easy to read. Mr. Deubelbeiss highlights a
great many of the bugs and problems that others found, and explains
why it's bad or what he would change when appropriate. In addition,
he's not afraid to comment on grammar or weak writing, which is
highly valued in a beta-tester. He draws a line between things that
should be added, and things that could be, to help the author in
managing her time. In addition, comments and thoughts on the puzzles
and the character's view of things are quite good, and it's always
nice to hear what you've done well. I have no doubt Mr. Deubelbeiss
would be a great help for any game he beta-tested.
---------------------
This concludes the announcement of the results of the 2003 Beta-Test
Competition. Tune in next year for the next edition.
Jess Knoch, www.strangebreezes.com/if
Joe Mason
> sand? Anyhow, I included some bugs on purpose because I was writing
> the mini-game specifically for this comp. I was also worried about
... You mean this was serious?
We figured it was just a flowery call-for-betatesters on an actual game.
Joe
Jessica Knoch
> Jessica Knoch wrote:
>> I included some bugs on purpose because I was
>> writing
>> the mini-game specifically for this comp.
>
>
> We figured it was just a flowery call-for-betatesters on an
> actual game.
Um. Yeah, it was serious. I'd been planning it for months. I
was trying to wait for just the right time... I guess it was
too close to the Annual Comp.
Oh well. It was fun anyhow. :)
--
Jess K.
Quintin Stone
> Um. Yeah, it was serious. I'd been planning it for months. I was trying
> to wait for just the right time... I guess it was too close to the
> Annual Comp.
>
> Oh well. It was fun anyhow. :)
Not that I didn't take it seriously or expect a winner to be declared, but
I did go in under the impression that it was a clever and elaborate way to
garner beta-testers for a comp game. I have to say, it was a surprise
(and not an unpleasant one) to find out I was wrong. The nearness to the
IF Comp definitely fueled my misunderstanding. And so I focused heavily
on finding bugs, and not so much on a great presentation. :)
Thanks, Jessica, for the award. ;)
/====================================================================\
|| Quintin Stone O- > "You speak of necessary evil? One ||
|| Code Monkey < of those necessities is that if ||
|| Rebel Programmers Society > innocents must suffer, the guilty must ||
|| st...@rps.net < suffer more." -- Mackenzie Calhoun ||
|| http://www.rps.net/ > "Once Burned" by Peter David ||
\====================================================================/
OKB (not okblacke)
>> ... You mean this was serious?
>>
>> We figured it was just a flowery call-for-betatesters on an actual
>> game.
>
> Um. Yeah, it was serious. I'd been planning it for months. I
> was trying to wait for just the right time... I guess it was
> too close to the Annual Comp.
For what it's worth, and even though I didn't enter, I thought it
was a really cool idea, regardless of whether it was for a real game or
not!
--
--OKB (not okblacke)
"Do not follow where the path may lead. Go, instead, where there is
no path, and leave a trail."
--author unknown
Jennifer Maddox
> On Fri, 5 Sep 2003, Jessica Knoch wrote:
>
> > Um. Yeah, it was serious. I'd been planning it for months. I was trying
> > to wait for just the right time... I guess it was too close to the
> > Annual Comp.
> >
> > Oh well. It was fun anyhow. :)
>
> Not that I didn't take it seriously or expect a winner to be declared, but
> I did go in under the impression that it was a clever and elaborate way to
> garner beta-testers for a comp game. I have to say, it was a surprise
First let me say that as a Peter David reader I must tip my hat to the
man that quotes Makenzie Calhoun. THAT was a surprise. Secondly, as
someone who was in on this thing from the ground floor, I must say it
was an excellent oppurtunity for me to learn how, umm, how shall I put
this? How poorly I beta-test. All the success Jess received from her
Tookie's Song she earned entirely on her own, her main beta-tester
certainly didn't add much :) And thirdly, the two things I'm best at
are being unclear and flattering my twin sister. I do them all the
time! So you can have this award back, Jess, and give it to your
husband. He's in a much better position to kiss your ::censored::
than I am.
Here's hoping for a chance to improve in next year's beta-comp!
jennifer
Steve Evans
wrote:
>Jessica Knoch wrote:
>
>>> ... You mean this was serious?
>>>
>>> We figured it was just a flowery call-for-betatesters on an actual
>>> game.
>>
>> Um. Yeah, it was serious. I'd been planning it for months. I
>> was trying to wait for just the right time... I guess it was
>> too close to the Annual Comp.
>
> For what it's worth, and even though I didn't enter, I thought it
>was a really cool idea, regardless of whether it was for a real game or
>not!
Ditto. I think it was a great idea, and I wish I'd had some time to
spare to enter.
I received quite a few beta reports for my last year's comp game
(including a very thorough one from Jess). All of the reports were
valuable, all of them picked up a number of things that the others
hadn't, and each tester had an individual presentation style and a
slightly different emphasis. I still marvel at how much better the
game/story/thing ended up being than if I hadn't had all of those
perspectives to draw from, and shudder at the thought of some of those
bugs and poor design choices of mine not having been caught before the
Comp.
I was looking forward to seeing the results of this Beta-Test comp,
and sincerely hope you can spare the time, Jess to run it again next
year.
Cheers,
Steve
Joe Mason
Well, that's what *I* figured it was. (And I think someone else posted
the same thing.) I admired your guts, too, so I'm a little disappointed
to find out it was the real thing. Also fascinated.
But I think a lot of other people knew all along, so blame me for
jumping to conclusions.
Joe
Adrien Beau
>
> Well, that's what *I* figured it was. (And I think someone else
> posted the same thing.) I admired your guts, too, so I'm a
> little disappointed to find out it was the real thing. Also
> fascinated.
>
> But I think a lot of other people knew all along, so blame me
> for jumping to conclusions.
I hate to say just this, but I jumped too.
--
spam....@free.fr
You have my name and my hostname: you can mail me.
(Put a period between my first and last names).