query regarding limitations of I7's automated testing features
MW
"blessing" features of the I7 editor (Windows version, 5Z71), and I've
run into some painful limitations of these that I hope are illusory.
Could y'all verify my perceptions?
With the skein features, it's easy to make a single long test run
through my game, and declare the text appearing in that run to be
correct, and then later rerun the test and automatically determine
whether the new output is correct. And it's easy to make many such
tests, as long as they only differ with respect to textual input at
the game's command prompt. This much is awesome, but it's only barely
sufficient for my purposes.
First problem: When a test fails, it's hard to find the failure. My
skein is big, and it'll be much bigger by the time I'm done testing.
Do I really have to manually scroll over it to find the red marks?
That's crazy!
Second problem: When different tests concern themselves with not only
different in-game commands, but also different file-system
environments, the skein system would seem to break down. My game
stores some important information in a file. I would like to be able
to specify the contents of the file as part of the artificial input a
test provides, and possibly also to check the contents of the file
after a test run. It seems that I can't do that in the I7 editor.
That's not so crazy, I guess....
Still, my problem with the file would be easier to deal with if I
could at least script my test runs via a Windows batch file. Does the
I7 editor take command line arguments? Can't it print out test
results at the command line? It looks like it can't. :(
Thanks for any help.
MW
blessed some nodes that had been marked in red (thus clearing the
marks), closed the editor, and opened it again. The nodes were again
marked in red, and my changes appeared to be gone.
When is the skein saved? I don't understand how that works. It
usually seems to quietly record itself to the .skein file at correct
(albeit mysterious) times, but once in a while, I lose something.
Tentative conclusion: I am the first person ever to make heavy use of
the skein.
I really need an automatic scrolling-to-the-next-error function. Or
maybe a way to step, in the Transcript view, from one leaf in the
skein to the next--that wouldn't eliminate all the tedious scrolling I
have to do to find errors in the Skein view, since I'd just end up
scrolling in the transcripts instead, but it would still help a lot,
because scrolling in the transcripts is at least guaranteed to show
all of my test runs in their entirety, whereas scrolling in the skein
is unreliable due to the enormous amount of empty space between
branches.
The ability to zoom in and out in the Skein view might also suffice.
How in the world do y'all normally do your testing? Just manually? I
guess that makes sense--time on the mainframes is pretty expensive.
But microcomputers are only getting faster--I claim that it would be
good for the medium to be ready a little early for the next generation
of hardware!
Ron Newcomb
The following may or may not be true only for the Mac IDE, but, after
running "test all", that runs all your blessed tests in the Skein, the
IDE will either 1) pull up the transcript for the first error, or 2)
screen-blink and beep without changing to the Transcript signifying
that all blessed tests passed.
There's also buttons to jump to the next error as well. I've used
them. ... in build 5T18 on Mac. I presume they're there in
Windows. Maybe someone more familiar with any IDE differences could
chime in.
(There's differences in the Find dialog box, too. The Mac's was
awesome until 5T18 I think, in which it got replaced with a new dialog
box that frequently crashes the IDE. GRRRR)
MW
David Kinder
> When is the skein saved? I don't understand how that works. It
> usually seems to quietly record itself to the .skein file at correct
> (albeit mysterious) times, but once in a while, I lose something.
On Windows, the skein is saved whenever the project is saved: either by the
user explicitly saving it, or doing something that makes Inform 7 save it
(usually, compiling it).
However, thinking about it, I note that if you change the skein and then
exit, you don't get any warning. Possibly this is why your changes disappeared.
> Tentative conclusion: I am the first person ever to make heavy use of
> the skein.
On Windows, quite possibly.
> I really need an automatic scrolling-to-the-next-error function. Or
> maybe a way to step, in the Transcript view, from one leaf in the
> skein to the next--that wouldn't eliminate all the tedious scrolling I
> have to do to find errors in the Skein view, since I'd just end up
> scrolling in the transcripts instead, but it would still help a lot,
> because scrolling in the transcripts is at least guaranteed to show
> all of my test runs in their entirety, whereas scrolling in the skein
> is unreliable due to the enormous amount of empty space between
> branches.
The Mac version does have a menu of items related to this (such as "Find
Next Difference"). This is on the list of things to add to the Windows
version. Possibly implementing this will be sufficient to help you.
David
Adam Thornton
MW <giocondo....@gmail.com> wrote:
>How in the world do y'all normally do your testing? Just manually? I
>guess that makes sense--time on the mainframes is pretty expensive.
>But microcomputers are only getting faster--I claim that it would be
>good for the medium to be ready a little early for the next generation
>of hardware!
I have a bunch of chained test commands (so, effectively, a chained set
of unit test suites). But since I rely heavily on randomized text, the
Skein's "bless" feature isn't actually very useful to me (and seeding
the random number generator would mean that I didn't get good coverage).
Adam
MW
> user explicitly saving it, or doing something that makes Inform 7 save it
> (usually, compiling it).
>
> However, thinking about it, I note that if you change the skein and then
> exit, you don't get any warning. Possibly this is why your changes disappeared.
Probably, yes. And to be clear: That is a bad bug. Should I
consider it already reported? Or should I report it?
> The Mac version does have a menu of items related to this (such as "Find
> Next Difference"). This is on the list of things to add to the Windows
> version. Possibly implementing this will be sufficient to help you.
Yes, it sounds like it would. I guess I'll be done by the time it
comes out; but for my present project I'm eventually gonna have to
write my own test driver in Java anyway (assuming that's possible), to
flush out the deepest potential bugs, so it doesn't matter all that
much.
David Kinder
> Probably, yes. And to be clear: That is a bad bug. Should I
> consider it already reported? Or should I report it?
It's on my list, so you don't need to report it.
David