Tips on conducting a beta-test cycle?
nathan...@msn.com
I am writing a computer game which will soon reach the beta stages of
development. The game contains elements of the Roguelike and
interactive-fiction genres, and so probably belongs to neither.
I intend to eschew what I view as the Roguelike release model in favor
of the interactive-fiction release model: release the game to a closed
group of beta-testers rather than make an unfinished product widely
available. The motivation here is this: the game has a detailed plot
and I'd rather it be bug free prior to public consumption. Or, in other
words, I want to patch the holes in my underwear before hanging them
out in public ;)
Which brings me to a few questions regarding beta-testing:
1.) For a game whose complexity is on the order of a roguelike or a
large IF adventure, how many beta testers seem to be appropriate? Is it
a job better for a small group, so communication does not get lost in
the noise? Or is it better to throw as many people at it as possible?
2.) What powers should beta-testers be given? Should they be given
access to wizard commands, game data, etc? Or should they be armed with
only the same repertoire that a real player would be offered?
3.) What is a good way to disseminate information among beta-testers,
to ensure bug reports, etc. are not duplicated? Would a bug tracking
tool be appropriate, and if so, which one? Are there better ways?
4.) Any opinions regarding how to measure when the game is, in fact,
ready for beta-testing?
5.) How long should a beta-testing period last? One month? Six? As long
as it takes? If so, is setting a schedule even worth the effort?
6.) How frequently should updates be released during beta-testing?
7.) How best to ensure thorough coverage of different areas of the
game? Should I prepare test scripts beforehand? Or perhaps a
walkthrough is more appropriate?
Your feedback is appreciated,
Nathan
Antoine
Possibly the most difficult step, which you don't mention above, is to
assemble a group of dedicated beta testers. RGRD may not be the best
place to find them.
A.
Mike Rozak
> I am writing a computer game which will soon reach the beta stages of
> development. The game contains elements of the Roguelike and
> interactive-fiction genres, and so probably belongs to neither.
My recommendation would be:
1) Distribute a copy to about 10 people that you're pretty sure will install
your game right away and give you feedback.
The initial feedback will probably be major stuff, like "it crashes all the
time" or "I cant figure out how to do X".
Once you have the major feedback fixed, and/or you stop getting significant
feedback, go to step 2.
2) Distribute a revised copy to the 10 original people PLUS 20 new ones.
Repeat. Roughly doubling every major release. Don't expect too much feedback
from the original people because they will be tired of testing.
3) When you're not getting much negative feedback in, and/or you're sick of
beta, call it done. (Or until the number of beta testers exceeds 10% of the
potential market.)
Also, step (0)... Get a few friends who own computers. Point them to your
download site and watch them actually download, install, and play the first
hour of the game. Sit right next to them and don't give them
hints/instructions unless they're really stuck. See how far they get; this
will give you an idea about how easy your product is to use. Make sure you
tell them that it isn't a test (of them), that you're trying to see what
mistakes you've made in the UI design, because you have become blind to the
flaws.
Specific comments:
> 2.) What powers should beta-testers be given? Should they be given
> access to wizard commands, game data, etc? Or should they be armed with
> only the same repertoire that a real player would be offered?
I woudln't give them anything special; if you do, then they'll give feeback
on how fun it is to play the game with cheats, not how fun the game is.
> 3.) What is a good way to disseminate information among beta-testers,
> to ensure bug reports, etc. are not duplicated? Would a bug tracking
> tool be appropriate, and if so, which one? Are there better ways?
You could go this far, but a simple word processor or post-its on your desk
are good for a single-person operation. When you provide an update, a list
of major bug fixes would be nice. Providing the entire bug database might be
overwhelming.
> 4.) Any opinions regarding how to measure when the game is, in fact,
> ready for beta-testing?
Usual definition of beta = feature complete... which means you think it's
done, but it's buggy and may need a tweak here or there. Realisticallty,
everyone defines beta differently. My opinion is that if there are major
flaws with the game that you can clearly identify, it's not really beta. You
want beta so real players can idenfity the flaws that you're blind to.
> 6.) How frequently should updates be released during beta-testing?
You'll get many answers for this. I suggest putting updates on your web site
as often as possible, and emailing beta testers at least one a week to
remind them of the updates. Whether or not they install the updates is
another question. Assign a unique version to each update though...
> 7.) How best to ensure thorough coverage of different areas of the
> game? Should I prepare test scripts beforehand? Or perhaps a
> walkthrough is more appropriate?
You should have test scripts with a full walkthrough for your own final
testing.
To help beta players test later parts, you could hand them high level
characters... but this causes its own problems. The players may discard
their current characters and skip to the end levels.
Timothy Pruett
<snip>
> Possibly the most difficult step, which you don't mention above, is to
> assemble a group of dedicated beta testers. RGRD may not be the best
> place to find them.
True. However, r.g.r.misc might be a good place to look.
ems...@mindspring.com
> 1.) For a game whose complexity is on the order of a roguelike or a
> large IF adventure, how many beta testers seem to be appropriate? Is it
> a job better for a small group, so communication does not get lost in
> the noise? Or is it better to throw as many people at it as possible?
I start out with a few testers and then expand the group on subsequent
iterations. That's partly so that there will be fresh eyes on the new
version, and partly because the first beta is likely to need enough
work that responses from many people would be overwhelming.
> 2.) What powers should beta-testers be given? Should they be given
> access to wizard commands, game data, etc? Or should they be armed with
> only the same repertoire that a real player would be offered?
I give them only what a real player would have. If they have trouble, I
provide hints, not cheats. Part of the point is to test not just
whether there are obvious bugs but how playable the game is overall,
which is information you can't get if you give them too many helps.
I don't know whether you're in a position to do this (since I'm not
sure where your game falls between rogue-like and IF), but I always ask
testers for complete transcripts of all their gameplay. Often reading
through transcripts turns up points where the game did something I
don't like, even if the tester didn't register it as an actual bug. It
also tends to turn up other infelicities of design -- things where the
player is having to do the same action over and over, etc. Reading
these is not quite as good as watching the player play through a
one-way mirror or something, but it can be much more useful than a bug
list alone.
> 3.) What is a good way to disseminate information among beta-testers,
> to ensure bug reports, etc. are not duplicated? Would a bug tracking
> tool be appropriate, and if so, which one? Are there better ways?
I don't have a formal system for this. I do tend to maintain fairly
close contact with my testers (chat sessions sometimes as opposed to
email) so that I can discuss with them any problems they're running
into -- and also let them talk to each other. This of course depends on
whether the tester is available for that kind of interaction, but I've
found it useful.
Since (as I said) I collect transcripts, I usually work my way through
them one by one, correcting all the problems I find as I go along; if
two people happen to have uncovered a bug I've already fixed, it's no
big deal.
> 4.) Any opinions regarding how to measure when the game is, in fact,
> ready for beta-testing?
This is kind of subjective, but I tend to put it out for beta-testing
when I'm no longer finding significant problems on my own playthroughs.
Beta-testing is terrific, and so are test scripts, and I would
encourage you to use both; but I also find that it is absolutely
critical to play the game myself, a lot. I try to forget that I know
anything about the underlying code and approach the game as though I
were playing it myself for fun. Any commands I get bored of typing,
puzzles that take too long to go through, missing scenery that I find,
etc., get dealt with at that stage. Once I can get a satisfying play
experience out of the game, it's ready for beta.
On the other hand, I do sometimes show a game to someone when it's not
finished that way -- making it clear that it's an alpha test. Usually
the point of that is to check whether the underlying game concepts are
interesting enough/well-designed enough, or whether I should make some
major changes before finishing it. I rarely consult more than one or
two people at a time in this way, and I have specific people I trust to
do it -- it's kind of above and beyond even the usual beta-testing call
of duty.
> 5.) How long should a beta-testing period last? One month? Six? As long
> as it takes?
This will depend on the size of the game, the complexity of the code
underlying it, the amount of content you've had to write, how shaky the
game was when you started beta, the degree of commitment shown by your
beta team, and (always a horrible X factor) whether you turn up any
critical usability issues which require serious feature re-design
instead of mere bugfixing.
> 6.) How frequently should updates be released during beta-testing?
Often enough to keep your beta-testers interested, and not so often
that you overwhelm them. Too many releases and they'll get weary of
redownloading new versions; too few and they'll think that you're not
really pushing the project too hard. This is largely going to pace
itself based on how rapidly the beta-testers are sending stuff back to
you.
It can help to post intermediate feedback or answer bug reports even
when you're not done with the new release, too -- just to let them know
that you are listening and working on the project, and that their
comments are having an effect.
> 7.) How best to ensure thorough coverage of different areas of the
> game? Should I prepare test scripts beforehand? Or perhaps a
> walkthrough is more appropriate?
You should have the means to test your game for basic winnability (a
walkthrough or test scripts as you like) and you should run the game
through that test every time you build a new release. But that's the
*minimum*, and it doesn't deal with the problem of getting enough
beta-tester coverage on the later parts of the game, which is
admittedly a challenge. Possibly encourage people to save games at
critical points and then replay from there?
Paul Drallos
>
> 3.) What is a good way to disseminate information among beta-testers,
> to ensure bug reports, etc. are not duplicated? Would a bug tracking
> tool be appropriate, and if so, which one? Are there better ways?
>
One of the hardest things for the programmer to catch are spelling and grammar mistakes since you often see what you know should be there but isn't. Game transcripts can help in the following way: Tell your testers to make comments during game play by preceeding them with a special symbol right after the prompt. For instance,
--
You are standing in a field. There is a white hose to the west.
>! you mispelled house. hose should be house.
--
This is useful, not just for grammar and spelling, but any comments the tester wants to convey to the writer. It is useful because the comments are in context with the actual gmae play. You can easily go through and find all of the comments with a find-string for '>!' in your test editor.
Dave Griffith
> One of the hardest things for the programmer to catch are spelling and
> grammar mistakes since you often see what you know should be there but
> isn't. Game transcripts can help in the following way: Tell your
> testers to make comments during game play by preceeding them with a
> special symbol right after the prompt. For instance,
> --
> You are standing in a field. There is a white hose to the west.
>>! you mispelled house. hose should be house.
> --
> This is useful, not just for grammar and spelling, but any comments
> the tester wants to convey to the writer. It is useful because the
> comments are in context with the actual gmae play. You can easily go
> through and find all of the comments with a find-string for '>!' in
> your test editor.
FWIW, here's how I implement that comment verb.
Verb meta '!'
* -> Comment
* topic -> Comment;
[ CommentSub; ];
--
David Griffith
dgr...@cs.csbuak.edu <-- Switch the 'b' and 'u'
Thomas
the subject.
What does everyone think about alpha testing. when i have a mostly
working version of my game i will probably go through three stages
>AlphaTest: give a game and the debug-mode key to a small group of 5-6 people i know personally or who are online but are very willing and are possibly developers themselves.
>BetaTest: give out the game to mostly anyone who wants one but not with online distrobution and not with the debug key. find friends/family/coworkers/peers/etc who are intersted and get their email and distribute biweeklyish updates and fixes over an email list.
>final BetaTest: as the very end of the betatest when i think it is absolutly perfect i will call it version 9.9.*Beta or whatever and give it out freely as a finished copy encouraging testing and feedback. After a month or so i will take all the small fixes and compile the first 1.0.0 non-beta copy for online free givawayness.
It maybe seems complicated by chazm is a huge project for me and the
largest coding endevor i have ever taken on and i want to do it right
this time ;)
My other question is what you ment by a test script. You mean an ai
that trys to play it through??? i dont think that would work out very
well. Do you mean just a walkthrough that tells a human player what to
do in cirtain cases so the the player can quickly win just to make sure
its possible?? If you could explain what you mean it would be great!
thanks.
-Thomas
RL: CHAZM
PS. My english sucks, I'm from the U.S... or maybe its because i
tired... or both.
nathan...@msn.com
> My other question is what you ment by a test script. You mean an ai
> that trys to play it through??? i dont think that would work out very
> well. Do you mean just a walkthrough that tells a human player what to
> do in cirtain cases so the the player can quickly win just to make sure
> its possible?? If you could explain what you mean it would be great!
> thanks.
>
Well, 'test script' is a very general term. In the context of IF, a
test script might be a series of predetermined commands that, when fed
into the parser by a test program, produces a certain output. You could
then diff your results with what you expected to ensure you haven't
broken anything.
However, I am not as familiar with IF development so I could be off
here.
For a roguelike, a 'test script' could be the playback of a series of
keystrokes recorded from a previous session. You could then register a
KeystrokePlaybackListener which is invoked when the recorded keystrokes
are exhausted. This listener could then assert many conditions about
the current game state. (Be sure to pin your RnG seed to get the same
results every time!) The keystroke recording and the set of assertions
together could constitute a test script.
Nathan
Gobleteer
of
> beta, call it done. (Or until the number of beta testers exceeds 10% of
the
> potential market.)