New harness for shell JSTests
David Mandelin
shell JSTests that we've been working on for a while. My primary goal
was to make the tests run faster on multiple cores and to be easier to
run day-to-day for devs, but it also supports Tinderbox-style reporting
and a bunch of other new features.
The basic usage is:
# in directory js/src/tests
python jstests.py PATH_TO_JS
This will run with a progress bar and report results in a compact way.
The 3 numbers on the left are mochitest-style "passed/failed/known
fails". There is no need to do a baseline run; bc's new manifest files
indicate which tests are known to fail, should be skipped, etc.
If you have more than 2 cores, then you probably want '-j N' to run
faster. I think it can run the whole suite in about a minute on an
8-core machine.
Here are a few more sample command lines:
# print options help text
python jstests.py -h
# tinderbox-style output
python jstests.py PATH_TO_JS --tinderbox
# run only tests that contain the string '263935'
python jstests.py PATH_TO_JS 263935
# run only those tests and print command lines and output
python jstests.py -so PATH_TO_JS 263935
# run only those tests in valgrind
python jstests.py --valgrind PATH_TO_JS 263935
# run that test in gdb
python jstests.py -g PATH_TO_JS 263935
Note that the options are processed with Python OptionParse, so they can
come in any order. E.g., this is fine:
# run only those tests and print command lines and output
python jstests.py PATH_TO_JS 263935 -so
Finally, thanks to Bob Clary and Jason Orendorff for patiently reviewing
multiple versions of this patch and guiding me on features,
understanding the test, and Python styling. And thanks to everyone who
encouraged me by trying out the prototype versions and liking it. :-)
--
Dave
Nicholas Nethercote
> I just landed to the tracemonkey repo a new harness for running the shell
> JSTests that we've been working on for a while. My primary goal was to make
> the tests run faster on multiple cores and to be easier to run day-to-day
> for devs, but it also supports Tinderbox-style reporting and a bunch of
> other new features.
This is awesome, thanks for doing it.
trace-test.py has the wonderful -f option which prints the command
lines of all failed tests. This is much nicer than jstest.py's -s
option because it only prints command lines for failed tests, and it
prints the command line exactly, not as an array of strings, so you
can cut-and-paste them easily. Could this be added to jstests.py? I
see -f has a different meaning for jstests.py, perhaps a different
option could be used, although it would be nice if jstests.py and
trace-test.py were as similar as possible.
Also, I just tried a i386 debug build, got this:
ecma/Date/15.9.3.8-1.js
ecma/Date/15.9.3.8-2.js
ecma/Date/15.9.5.14.js
ecma/Date/15.9.5.22-1.js
ecma/Date/15.9.5.22-3.js
ecma/Date/15.9.5.23-1.js
ecma/Date/15.9.5.23-15.js
ecma/Date/15.9.5.23-17.js
ecma/Date/15.9.5.24-1.js
ecma/Date/15.9.5.24-2.js
ecma/Date/15.9.5.24-3.js
ecma/Date/15.9.5.24-5.js
ecma/Date/15.9.5.24-4.js
ecma/Date/15.9.5.24-6.js
ecma/Date/15.9.5.24-7.js
ecma/Date/15.9.5.24-8.js
ecma/Date/15.9.5.26-1.js
ecma/Date/15.9.5.25-1.js
ecma/Date/15.9.5.27-1.js
ecma/Date/15.9.5.28-1.js
ecma/Date/15.9.5.29-1.js
ecma/Date/15.9.5.30-1.js
ecma/Date/15.9.5.31-1.js
ecma/Date/15.9.5.32-1.js
ecma/Date/15.9.5.33-1.js
ecma/Date/15.9.5.34-1.js
ecma/Date/15.9.5.35-1.js
ecma/Date/15.9.5.36-1.js
ecma/Date/15.9.5.36-2.js
ecma/Date/15.9.5.36-3.js
ecma/Date/15.9.5.36-4.js
ecma/Date/15.9.5.36-5.js
ecma/Date/15.9.5.36-6.js
ecma/Date/15.9.5.36-7.js
ecma/Date/15.9.5.37-1.js
ecma/Date/15.9.5.8.js
ecma_3/Date/15.9.5.6.js
ecma_3/Date/15.9.5.5.js
ecma_5/Date/15.9.4.2.js
js1_5/extensions/toLocaleFormat-02.js
js1_5/extensions/toLocaleFormat-01.js
js1_8_1/decompilation/regress-530537.js
From the looks of it, most of them are caused by all-too-common
"waddya mean there are timezones outside of North America?" problem
(eg. see bugs 515254, 526665). Are these failures known? Should I
file a bug?
Nick
itroot
oh, that's nice, thank you!
But while i run this test suite on my machine, i get
itroot@itroot-desktop:~/prefix/js/tests$ python jstests.py ../shell/js
Traceback (most recent call last):
File "jstests.py", line 262, in <module>
test_list = manifest.parse(OPTIONS.manifest, xul_tester)
File "/home/itroot/prefix/js/tests/manifest.py", line 102, in parse
ans += parse(os.path.join(dir, parts[1]), xul_tester)
File "/home/itroot/prefix/js/tests/manifest.py", line 102, in parse
ans += parse(os.path.join(dir, parts[1]), xul_tester)
File "/home/itroot/prefix/js/tests/manifest.py", line 130, in parse
if xul_tester.test(cond):
File "/home/itroot/prefix/js/tests/manifest.py", line 78, in test
raise Exception("Failed to test XUL condition '%s'"%cond)
itroot@itroot-desktop:~/prefix/js/tests$ python --version
Python 2.6.2
itroot@itroot-desktop:~/prefix/js/tests$ uname -a
Linux itroot-desktop 2.6.28-16-generic #55-Ubuntu SMP Tue Oct 20
19:48:32 UTC 2009 x86_64 GNU/Linux
I built tracemonkey as standalone library.
I can fix this problem just setting ans to False instead of raising
exception, but it is not appropriate for everyone.
In "out" variable i get "x86_64" instead of true false or null.
Maybe this is wrong place to post it, if it so, where i can post bug?
David Mandelin
> On Tue, Nov 24, 2009 at 8:13 AM, David Mandelin<dman...@mozilla.com> wrote:
>> I just landed to the tracemonkey repo a new harness for running the shell
>> JSTests that we've been working on for a while. My primary goal was to make
>> the tests run faster on multiple cores and to be easier to run day-to-day
>> for devs, but it also supports Tinderbox-style reporting and a bunch of
>> other new features.
>
> This is awesome, thanks for doing it.
>
> trace-test.py has the wonderful -f option which prints the command
> lines of all failed tests. This is much nicer than jstest.py's -s
> option because it only prints command lines for failed tests, and it
> prints the command line exactly, not as an array of strings, so you
> can cut-and-paste them easily. Could this be added to jstests.py? I
> see -f has a different meaning for jstests.py, perhaps a different
> option could be used, although it would be nice if jstests.py and
> trace-test.py were as similar as possible.
Very good idea. Could you please file a bug on this? On the option
names, I've been picking things fairly arbitrarily. I agree they should
be the same for both harnesses. Tell me what you think and I'll do it.
> Also, I just tried a i386 debug build, got this:
>
> ecma/Date/15.9.3.8-1.js
...
> ecma_5/Date/15.9.4.2.js
> js1_5/extensions/toLocaleFormat-02.js
> js1_5/extensions/toLocaleFormat-01.js
> js1_8_1/decompilation/regress-530537.js
>
> From the looks of it, most of them are caused by all-too-common
> "waddya mean there are timezones outside of North America?" problem
> (eg. see bugs 515254, 526665). Are these failures known? Should I
> file a bug?
IMO the baseline is that everything should just work, so yes, please
file a bug on this too.
--
Dave
David Mandelin
That's definitely a bug. I think you can file it as a SpiderMonkey bug.
I'll find it there.
--
Dave
Nicholas Nethercote
>
> But while i run this test suite on my machine, i get
>
> itroot@itroot-desktop:~/prefix/js/tests$ python jstests.py ../shell/js
> Traceback (most recent call last):
> File "jstests.py", line 262, in <module>
> test_list = manifest.parse(OPTIONS.manifest, xul_tester)
> File "/home/itroot/prefix/js/tests/manifest.py", line 102, in parse
> ans += parse(os.path.join(dir, parts[1]), xul_tester)
> File "/home/itroot/prefix/js/tests/manifest.py", line 102, in parse
> ans += parse(os.path.join(dir, parts[1]), xul_tester)
> File "/home/itroot/prefix/js/tests/manifest.py", line 130, in parse
> if xul_tester.test(cond):
> File "/home/itroot/prefix/js/tests/manifest.py", line 78, in test
> raise Exception("Failed to test XUL condition '%s'"%cond)
>
I just filed https://bugzilla.mozilla.org/show_bug.cgi?id=530969.
Nick
Igor Bukanov
> IMO the baseline is that everything should just work, so yes, please file a
> bug on this too.
There are some tests that fails if the amount of memory they can
allocate is limited to, say, 512 MB. In addition using -t switch is
useful to limit the execution time on a slow CPU. So it would be nice
to have an explicit switch that output the list of failures into a
file that can be passed using -x switch.