Rhino unit and performance tests?
Marc Guillemot
can someone answer following questions concerning Rhino sources. This
would help me in providing patches:
- has Rhino a valid test suite?
Running the tests as documented at
http://developer.mozilla.org/en/docs/Running_the_Rhino_tests leads to
some errors and can't therefore be used to know if changes introduce errors.
- is there a continuous integration for Rhino somewhere?
- are there some benchmarks validating the choice of some "optimizations"?
I'd like for instance to try to use standard java collections rather
than self made hash table to store Slots.
Thanks,
Marc.
--
Blog: http://mguillem.wordpress.com
Attila Szegedi
On 2008.01.16., at 9:54, Marc Guillemot wrote:
> Hi,
>
> can someone answer following questions concerning Rhino sources. This
> would help me in providing patches:
>
> - has Rhino a valid test suite?
> Running the tests as documented at
> http://developer.mozilla.org/en/docs/Running_the_Rhino_tests leads to
> some errors and can't therefore be used to know if changes introduce
> errors.
Which ones are failing for you? I'm running my tests on Mac OS X, with
a non-english locale, and I too have some tests failing. Namely:
ecma_3/Date/15.9.5.5.js: says it'll fail for a non-english locale, and
my machine indeed runs with Hungarian, so that's probably okay
js1_5/Regress/regress-58116.js: this might be genuine, I'm not sure
js1_5/Regress/regress-98901.js: throws StackOverflowError. From
reading the associated bug, this is probably (unfortunately) expected.
They fixed it for SpiderMonkey, but it is caused by a hideously ugly
piece of script... I guess we might just want to add it to our exclude
list and forget about it :-)
lc2/JavaToJS/char-002.js: it is simply not platform independent. I
don't even think this test is correct. It presumes that the Java path
separator character is ; (semicolon), but on UNIX-like OSes it is
actually : (colon). Therefore, this test fails on all non-Windows OSes.
I discussed this with Norris back in December, they work for him.
> - is there a continuous integration for Rhino somewhere?
I started setting one up at <http://cn-rhino01.nl.mozilla.org:8085>.
Problem is, there are still some quirks with it. One of them is we're
using the Mozilla anonymous CVS (as I don't want to store anyone's
developer credentials in the system), and it can't detect commits
automatically for some reason. Also, it must check out the mozilla/js
tree, not the mozilla/js/rhino tree, as it needs mozilla/tests in
order to run.
The biggest problem is that the JUnit test driver produces even more
failures than the command-line driver, so we're yet to produce a
successful build (it reports 507 failed tests out of 4647) :-(
> - are there some benchmarks validating the choice of some
> "optimizations"?
> I'd like for instance to try to use standard java collections rather
> than self made hash table to store Slots.
That's admittedly one of my pet peeves too. Rhino codebase goes back
to Java 1.1 days when the only collection was java.util.Hashtable.
It'd be due to be modernized, certainly.
Attila.
Marc Guillemot
grep Failed on the output gives me following failures (meaning that
stackoverflow problems are not included here). It differs a bit from yours:
-------------------------------
Failed: ../tests/ecma_3/Date/15.9.5.5.js:
Math.abs(Date.parse(now.toLocaleString()) - now.valueOf()) < 1000 =
false FAILED! expected: true
Failed: ../tests/js1_5/Regress/regress-58116.js: FAILED! expected:
Expected value '-120', Actual value '-60'
Failed: ../tests/js1_5/Regress/regress-89443.js: JavaScript errors:
Failed: ../tests/js1_5/Regress/regress-98901.js: JavaScript errors:
Failed: ../tests/js1_5/extensions/regress-226507.js: JavaScript errors:
-------------------------------
In fact I don't really care if these tests are excluded or not but it
would help if the tests that should pass could be up to date
(additionally a mechanism to detect tests that already pass even if they
are marked as excluded would be useful).
>> - is there a continuous integration for Rhino somewhere?
>
> I started setting one up at <http://cn-rhino01.nl.mozilla.org:8085>.
> Problem is, there are still some quirks with it. One of them is we're
> using the Mozilla anonymous CVS (as I don't want to store anyone's
> developer credentials in the system), and it can't detect commits
> automatically for some reason. Also, it must check out the mozilla/js
> tree, not the mozilla/js/rhino tree, as it needs mozilla/tests in order
> to run.
>
> The biggest problem is that the JUnit test driver produces even more
> failures than the command-line driver, so we're yet to produce a
> successful build (it reports 507 failed tests out of 4647) :-(
do you know where the difference comes from?
Working on java.util.regex based regex support for Rhino I've seen that
some problems occur due when tests are run in a separated thread which
has a smallest stack size. Perhaps is it the same problem here?
>> - are there some benchmarks validating the choice of some
>> "optimizations"?
>> I'd like for instance to try to use standard java collections rather
>> than self made hash table to store Slots.
>
> That's admittedly one of my pet peeves too. Rhino codebase goes back to
> Java 1.1 days when the only collection was java.util.Hashtable. It'd be
> due to be modernized, certainly.
does it mean that there is no benchmark and that passing the tests would
be considered as ok?
Cheers,
David P. Caldwell
> Attila Szegedi wrote:
> Failed: ../tests/ecma_3/Date/15.9.5.5.js:
> Math.abs(Date.parse(now.toLocaleString()) - now.valueOf()) < 1000 =
> false FAILED! expected: true
> Failed: ../tests/js1_5/Regress/regress-58116.js: FAILED! expected:
> Expected value '-120', Actual value '-60'
> Failed: ../tests/js1_5/Regress/regress-89443.js: JavaScript errors:
> Failed: ../tests/js1_5/Regress/regress-98901.js: JavaScript errors:
> Failed: ../tests/js1_5/extensions/regress-226507.js: JavaScript errors:
> -------------------------------
>
> In fact I don't really care if these tests are excluded or not but it
> would help if the tests that should pass could be up to date
> (additionally a mechanism to detect tests that already pass even if they
> are marked as excluded would be useful).
The test driver can help here (without resorting to grep) -- it emits
the results in HTML format (easier to browse) and also in XML format
(easier to postprocess). Running the suite with -f myfile.html will
create two files -- myfile.html and myfile.html.xml containing data
about what happened.
You can run the excluded tests (alone) just by loading the list of
tests to run using the -l switch (that's a lowercase "L"), as in -l
@<skip-list-file>.
-- David P. Caldwell
http://www.inonit.com/
Marc Guillemot
not really a solution, too much manual work ;-(
Norris Boyd
I've added the tests Attila mentions above to the skip list
(incidentally, lc2/JavaToJS/char-002.js works for me on Linux). That
leaves js1_5/Regress/regress-89443.js and js1_5/extensions/
regress-226507.js as tests that work for both me and Attila but fail
for Marc. I don't know why those tests are problematic. I reran the
tests and grepped for Failure (I usually use Calwell's tool) and
didn't get any failures.
And I would certainly support moving to one of the Java collection
classes in ScriptableObject as long as there wasn't a negative
performance impact.
I have run performance tests using the benchmarks at
http://ejohn.org/projects/javascript-engine-speeds/. I don't have
anything automated, however. That would be great to have.
--N
Attila Szegedi
On 2008.01.16., at 20:53, Norris Boyd wrote:
> I have run performance tests using the benchmarks at
> http://ejohn.org/projects/javascript-engine-speeds/. I don't have
> anything automated, however. That would be great to have.
There's also SunSpider test suite at WebKit project: <http://webkit.org/blog/152/announcing-sunspider-09/
>. I haven't figured out yet how to get Rhino run it, but it is very
attractive as it can apparently measure the statistical confidence
level of its own performance measurements!
Attila.
--
home: http://www.szegedi.org
weblog: http://constc.blogspot.com
Attila Szegedi
On 2008.01.16., at 20:53, Norris Boyd wrote:
> And I would certainly support moving to one of the Java collection
> classes in ScriptableObject as long as there wasn't a negative
> performance impact.
I can't imagine there'd be. Especially if we allowed people to somehow
plug in a Map factory (we discussed this somewhere already about a
year ago), so people could plug in a Map implementation of their
choosing -- i.e. the Trove project provides open addressing
implementation similar to what Rhino uses now. People could also
choose between a HashMap and ConcurrentHashMap based on the
concurrency expectations of their data model. And so on.
Anyway -- I guess since you've been around the project since its
initiation, you'd probably be the most qualified to touch that code,
as it is a rather central piece of the library.
Attila.
Attila Szegedi
> I have run performance tests using the benchmarks at
> http://ejohn.org/projects/javascript-engine-speeds/. I don't have
> anything automated, however. That would be great to have.
Well, Bamboo could run those as well. Provided I could get it to
finally a) run tests once cleanly and b) update from CVS in a timely
manner...
Attila.
> --N
Marc Guillemot
> ...
> I've added the tests Attila mentions above to the skip list
> (incidentally, lc2/JavaToJS/char-002.js works for me on Linux). That
> leaves js1_5/Regress/regress-89443.js and js1_5/extensions/
> regress-226507.js as tests that work for both me and Attila but fail
> for Marc. I don't know why those tests are problematic. I reran the
> tests and grepped for Failure (I usually use Calwell's tool) and
> didn't get any failures.
On Linux using Java 1.6.0_03 (with empty JAVA_OPTS) I have
- Testcase js1_5/Regress/regress-89443.js failed
Failure reason:
JavaScript errors:
null:0: exception from uncaught JavaScript throw:
java.lang.StackOverflowError
- Testcase js1_5/extensions/regress-226507.js failed
Failure reason:
JavaScript errors:
../tests/js1_5/extensions/regress-226507.js:305: Too deep recursion
while parsing
this second failure denotes a StackOverflowError too.
Do you run the tests with extra parameters?
Attila Szegedi
tested a stack overflow in SpiderMonkey parser for a JS page that had
a single monstruous statement few hundreds of KBytes long that
appeared in some, I think Sony, website back in nineties. It was of
course not manually coded, but autogenerated from something. You can
look it up by the numeric ID of the test in Bugzilla for gory details
if you wish. If we never fixed that, I won't necessarily feel bad
about that :-)
Attila.
Norris Boyd
Aha! I had been running with Java 1.5. When I try with Java 1.6 I get
the failures. Looking at the tests, I see these tests are stress
tests. I'll add them to the skip list.
--N
Marc Guillemot
I've created a patch to use a Map in ScriptableObject
https://bugzilla.mozilla.org/show_bug.cgi?id=412928
It adds a predictable iteration order which is not required by ECMA but
exists in the js implementations of different browsers.
"normal" tests pass except js1_5/extensions/regress-367501-04.js which
now causes a StackOverflow. I don't see it as a problem as it probably
comes from the way the map grows.
Execution speed seems comparable for the test suite as well as for
HtmlUnit test suite which is surely more representative of real use.