Auto Beta tester

2 views
Skip to first unread message

David Fisher

unread,
May 18, 2005, 8:16:55 PM5/18/05
to
From the "What do non programmer types want in an IF language ?" thread:

Maybe there is a need for an "auto beta tester" program that tries
everything & dumps the output to a file so you can examine it and see if
anything unexpected happened. It would need to be a "white box" not a "black
box" test, though (I feel like throwing technical terms around today :-) ) -
ie. with knowledge of the code it is testing. Otherwise it might try saying
"pick up frog" in every room in the game, even if there is no frog ...

Any thoughts ?

David Fisher


David Alex Lamb

unread,
May 19, 2005, 7:30:49 AM5/19/05
to
In article <dZQie.14128$Le2....@nasal.pacific.net.au>,

How do you define "tries everything?" What came to mind was something that
would examine the grammar and figure out what sentences to generate. But
(specilating from complete ignorance) I doubt the grammar is accessible in the
form it would be needed, being organized for parsing instead.
--
"Yo' ideas need to be thinked befo' they are say'd" - Ian Lamb, age 3.5
http://www.cs.queensu.ca/~dalamb/ qucis->cs to reply (it's a long story...)

Graham Holden

unread,
May 19, 2005, 8:30:32 AM5/19/05
to
On 19 May 2005 11:30:49 GMT, dal...@qucis.queensu.ca (David Alex Lamb)
wrote:

>In article <dZQie.14128$Le2....@nasal.pacific.net.au>,
>David Fisher <da...@hsa.com.au> wrote:
>>From the "What do non programmer types want in an IF language ?" thread:
>>
>>Maybe there is a need for an "auto beta tester" program that tries
>>everything & dumps the output to a file so you can examine it and see if
>>anything unexpected happened. It would need to be a "white box" not a "black
>>box" test, though (I feel like throwing technical terms around today :-) ) -
>>ie. with knowledge of the code it is testing. Otherwise it might try saying
>>"pick up frog" in every room in the game, even if there is no frog ...
>>
>>Any thoughts ?
>
>How do you define "tries everything?" What came to mind was something that
>would examine the grammar and figure out what sentences to generate. But
>(specilating from complete ignorance) I doubt the grammar is accessible in the
>form it would be needed, being organized for parsing instead.

Merging this idea and one from another topic (about unexpected responses,
in particular, IIRC, EAT STONE --> "You are not a cannibal") to could
write a fairly mechanical generator that knows all the "standard" verbs and
is given a list of (a) extra verbs and (b) your objects/locations and
generates a command-script like:

zapto Kitchen
x sink
feel sink
eat sink
...
crumple sink
x toaster
...
zapto Bedroom
x banknote
...
crumple banknote
...

and so on. The ZAPTO would be a 'teleport' command so you don't have to
bother with movement.

This wouldn't in any way exhaustively test things -- some/many objects
won't be in any location to start with; certain commands on certain objects
will alter their state etc.

Personally, I'm not convinced that the amount of output you'd have to wade
through would be worth it, but it might be a useful tool to catch obviously
wrong responses.


Regards,
Graham Holden (g-holden AT dircon DOT co DOT uk)
--
There are 10 types of people in the world;
those that understand binary and those that don't.

BrettW

unread,
May 19, 2005, 9:17:47 AM5/19/05
to
I've thought about an bot-beta tester, but I'm not convinced by your
approach. You'll get an avalanche of things to check, most of them just
returning standard responses. And it'd only be really useful for
spell-checking/grammar checking and testing if the responses are okay. A
lot of these things are probably best checked in-code rather than via
transcript. If you have adaptive text (text that modifies depending on
experience, objects carried, time of day...) then you'll need to account
for those as well (which is maybe best checked in-game, but how would you
check all possibilities autonomously?)

But if you wanted to do things in this way, maybe it's best to check for
scope and applicability first. If you're really sophisticated you could
check which verbs are different from the default and just check those.

I'm more interested in a bot that checks stylistic things (like going
north then south brings you back to where you were), candidates for simple
oversights (like not nailing certain things down, or doors not being
openable multiple times), and maybe tabulating statistics. Of course these
have limits in its applicability and usefulness, but it might be
interesting.

Though this is more a thing for an IF library rather than an IF language,
I would have thought.

BrettW

JohnnyMrNinja

unread,
May 19, 2005, 1:46:20 PM5/19/05
to

There's a bot, Billy, that adapts and learns based on how you talk to
it, but is also programmable. It communicates with other programs
through a simple text file: one program erases the text of this file
and adds it's line, then the other program does the same.
http://leedberg.com/glsoft/billy/index.shtml Perhaps with a Frotz hack
and an IF "mind" for Billy, this could serve as an auto tester? If you
set it to auto-restart (or reload), then you could leave Billy going
for days and come back and skim the log.

David Fisher

unread,
May 20, 2005, 5:04:16 AM5/20/05
to
"David Alex Lamb" <dal...@qucis.queensu.ca> wrote in message
news:d6htd9$nnp$1...@knot.queensu.ca...

> In article <dZQie.14128$Le2....@nasal.pacific.net.au>,
> David Fisher <da...@hsa.com.au> wrote:
>>From the "What do non programmer types want in an IF language ?" thread:
>>
>>Maybe there is a need for an "auto beta tester" program that tries
>>everything & dumps the output to a file so you can examine it and see if
>>anything unexpected happened. It would need to be a "white box" not a
>>"black
>>box" test, though (I feel like throwing technical terms around today
>>:-) ) -
>>ie. with knowledge of the code it is testing. Otherwise it might try
>>saying
>>"pick up frog" in every room in the game, even if there is no frog ...
>>
>>Any thoughts ?
>
> How do you define "tries everything?"

Ideally, it would explore the "space" of all possible sequences of actions
in the story - not exploring any further when a branch runs into a dead end
(ie. no internal state change) or reaches a previously-explored state. You
could filter out repetitive output of a particular type if you wanted to.

Just thinking about that, it would be useful to have a list of "all possible
states reached by the game". This sounds hard because if you have only two
numbers in the game, that makes 2^32*2^32 = 2^64 possible states already
(assuming 32 bit integers) ... it really needs to store possible ranges of
numbers (eg. 0 <= X <= 100), or something like that ... but this could get
more complicated if the numbers aren't independent (Y depends on the value
of X) ... ramble, ramble, ramble ...

Another use for this kind of thing is that it will tell you if there are any
ways to solve a puzzle in your game that you hadn't thought of ... (it
wouldn't be as good as a real Beta tester though, because they can tell you
things that you have left out as well). And it would just be kind of fun to
see all of the states your game could get into.

Anyway, just some thoughts (as you can see :-) )

IF is probably one of the best domains to try doing something like this
anyway, since most variables in an IF game have a small possible set of
values (and there are usually no floating point values in the program).

Anyone have any experience in data flow analysis ?

David Fisher


David Fisher

unread,
May 21, 2005, 1:07:45 AM5/21/05
to
"David Fisher" <da...@hsa.com.au> wrote in message
news:xNhje.14582$Le2....@nasal.pacific.net.au...

Another quick thought: if there are N objects in the game and you can carry
all of them, then the inventory alone has 2 to power of N possible states
... might have to rethink this a little bit. Surely there is a way of doing
something like this ?

David Fisher


Reply all
Reply to author
Forward
0 new messages