IF Tools for Unit/Regression Testing
David A. Cornelson
currently used processes are fine and I tend to agree with that sentiment. I
think we just need to make some of the processes more public. I might write
something up on this subject, or if anyone else feels an urge and has a good
beta-testing process down, then go for it. A Guide To Beta-Testing
Interactive Fiction sounds like a good essay to write and probably will be
receive a great deal of attention, especially from newer authors or authors
with less software development experience.
Now, what also has been discussed both on usenet and on the ifMUD is the
types of tools we currently have to do unit testing and then regression
testing. First, let's get some definitions out of the way.
Unit Testing:
I would describe this as such. The author builds an object or objects and
runs through all of the commands they can think of to test the new code.
They basically play that part of the game file that contains references to
the new object(s).
Regression Testing:
Generally script oriented, regression testing is like taking a bunch of
scripted unit tests and running them after a change has been made to code.
So you might have something like this after words:
House Scene:
Living Room: Tested - OKAY
Dining Room: Tested - OKAY
Hallway: Tested - OKAY
Backyard Scene:
Pond: Tested - ERRORS (see script log)
* * * * *
The idea is that as we write code, we also build test scripts and these test
scripts continually test our game to make sure we haven't broken anything.
We also must add to these test scripts or rebuild them when major changes
occur. For instance, a single change may effect dozens of locations and more
than a single scene.
Now this is assuming that we're only testing functionality. I would also
like to figure out a way to do this sort of testing and at the same time do
spelling and grammar checking. Before everyone gets in an uproar about how
easy it is to cut and paste into Word or manage your text in Word to begin
with, I was thinking that such a tool is not only possible to build, but
relatively simple. In fact, many useful testing tools can be built by
extending the functionality of any of the available interpreters.
There are shareware dictionaries and thesauruses available to implement
within code. I know we like things that are free, but the last time I
searched for such items, there were no freebies unless you went to a 1917
dictionary or something. In any case, this is only one of the items people
have suggested and I myself would be interested in.
Another extension would be the ability to organize and execute various
scripts better than the current methods. Currently, you have to pick one
script and back it runs. I'd like to see a window that opens to a list of
pre-defined scripts, saved game files, with descriptions, and output
locations selectable. I'd also like to see this extension allow me to select
more than one script and run through them automatically. For example:
"Living room scene", "c:\inform\house\lhouse.z5",
"c:\inform\house\test\lrtest.scr", "c:\inform\house\test\logs"
"Kitchen scene", "c:\inform\house\house.z5",
"c:\inform\house\test\ktest.scr", "c:\inform\house\test\logs"
The extension would use the above lines to determine that I currently have
two unit tests that can be executed, which game file to use, which script to
run, and where to put the resulting log file. The next logical step _might_
be to include a previous log file and do a file comparison and reviewing
only the differences.
There are more extensions, but this is my first crack at these thoughts.
Jarb
L. Ross Raszewski
You might want to look into program slicing. It's a decomposition
technique that extracts all the code in a program relevant to a
specified computation. One could, say, slice on the White House, and
get back a subset of Zork which contains only that code relevant to
afforementioned house. There's also been work done on using slicing
for regression testing: you can have a program isolate the differences
between two versions of a program, and slice on them, yielding a
program which captures how the behavior differs (The claim being "if
version A works and version B doesn't, the bug is going to be in
delta(A,B)")
Stuart "Sslaxx" Moore
> You might want to look into program slicing. It's a decomposition
> technique that extracts all the code in a program relevant to a
> specified computation. One could, say, slice on the White House, and
> get back a subset of Zork which contains only that code relevant to
> afforementioned house. There's also been work done on using slicing
> for regression testing: you can have a program isolate the differences
> between two versions of a program, and slice on them, yielding a
> program which captures how the behavior differs (The claim being "if
> version A works and version B doesn't, the bug is going to be in
> delta(A,B)")
Also, could CVS work for some of what you want to achieve?
Bye,
--
Stuart "Sslaxx" Moore.
http://www.sslaxx.demon.co.uk
AIM: MrSslaxx
LiveJournal: http://www.livejournal.com/~sslaxx
Lewis Raszewski
Well, CVS can produce the textual differences between two versions of a
file, but you need to perform slicing if you want to extract a program
which captures the semantic differences.
For instance, if you've made a change to object A which exposes a bug in
object B, a simple text differencing won't show object B to you. A
slice on A will.
* Some work has been done in this area; you use a tool like CVS to
isolate the textual differences between two versions of a file, then
slice on those differences. You can compile the program which the slicer
outputs and run your test suite on it. Since you;ve got a smaller
program, testing is faster.
--
L. Ross Raszewski
The Johns Hopkins University
Wyman Park 407
"'No reason to get excited,' the thief, he kindly spoke. 'There are many
here
among us who feel that life is but a joke.'" -- Bob Dylan, All Along the
Watchtower
Plugh!
> Another extension would be the ability to organize and execute various
> scripts better than the current methods. Currently, you have to pick one
> script and back it runs. I'd like to see a window that opens to a list of
> pre-defined scripts, saved game files, with descriptions, and output
> locations selectable. I'd also like to see this extension allow me to select
> more than one script and run through them automatically. For example:
>
> "Living room scene", "c:\inform\house\lhouse.z5",
> "c:\inform\house\test\lrtest.scr", "c:\inform\house\test\logs"
> "Kitchen scene", "c:\inform\house\house.z5",
> "c:\inform\house\test\ktest.scr", "c:\inform\house\test\logs"
>
> The extension would use the above lines to determine that I currently have
> two unit tests that can be executed, which game file to use, which script to
> run, and where to put the resulting log file. The next logical step _might_
> be to include a previous log file and do a file comparison and reviewing
> only the differences.
>
> There are more extensions, but this is my first crack at these thoughts.
Sounds simple enough (and the spell checker is probably do-able too).
Keep specifying & it may just get implemented (would Java be
acceptable?)
Kirk Macolini
If you are interested in program slicing, GrammaTech produces a
commercial program slicer for C called CodeSurfer. You can read more
about it at:
http://www.grammatech.com/products/codesurfer/overview.html
Or download an evaluation copy at:
http://www.grammatech.com/products/codesurfer/download.html
Kirk J. Macolini
CodeSurfer Product Manager
GrammaTech, Inc.
317 North Aurora Street
Ithaca, NY 14850
e-mail: ki...@grammatech.com
www: http://www.grammatech.com
L. Ross Raszewski
Wow. What're the odds? CodeSurfer's one of the (extremely small)
number of program slicers I've used -- and certainly the most usable.
IF is certainyl one of those areas where computing a system dependence
graph could come in handy (I never knew that the plover sword could
have an impact on objects in the Frobozz room)
Of course, the real trouble would be finding an existing slicer that
could be made to cope with the IF-languages; Codesurfer, iirc, speaks
only C.
Now that I think about it, though, the various program decompostition
techniques could be applied pretty usefully to IF in general; a tool
that could cleanly extract a single area of a large game into
separate, compilable source for testing or for separate distribution
as a demo or to break the really huge games everyone's into these days
into chunks manageable on handhelds.
Timothy Partridge
trust to look at your code and make constructive comments I would strongly
recommend it. We do it at work for software and it is helpful. Even the best
programmer can miss something now and then.
A reviewer can spot mistakes and bugs, and make (helpful) comments about
possible alternative approaches. The author isn't expected to indulge in
major rewrites at the whim of the reviewer!
I would say it is much easier for a reviewer to spot some mistakes like
movable doors than a tester to type in the commands to try and move them.
A reviewer can also proof read the text, including that for situations that
are time consuming to set up in testing. (You might need two reviewers if
the combination of being good at checking computer and natural languages is
hard to find in one person.)
A review should happen before beta testing. The code should be in reasonable
shape before it is offered for review, unless you have agreed otherwise with
a very patient reviewer.
I noticed several games in the recent competition had spelling mistakes. I'm
surprised authors didn't go to the trouble of running a spell checker
against their text. (I don't have the same standards for news posts :-)
Tim
--
Tim Partridge. Any opinions expressed are mine only and not those of my employer
Kirk Macolini
The internals of CodeSurfer are designed to be language-independent
and CodeSurfer is set up so that users can write their own front-ends
for other languages. So theoretically one could use CodeSurfer for IF
languages.