How to run the test262 test suite?

[P]

Hi,

Am trying our a few things in the codebase and would like to run the test262 tests, but cannot find how.

Did find some pointers in some merged pull requests like https://github.com/mozilla/rhino/pull/205, but no steps how to actually run the test suite.

This seems out of date: https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Rhino/Documentation/Running_the_Rhino_tests

From the readme in https://github.com/mozilla/rhino I figured out I can run ./gradlew test and I've run `./gradlew tasks` to see all tasks, but that list doesn't include anything that looks like just the test262 suite nor does that list include `testBenchmark`, which, according to the readme is one of the tasks

TIA
Paul

[Gregory Brail]

Sorry about the out of date Mozilla pages. The README should be helpful, and if not, let's fix it!

The test262 tests live in a submodule. The readme says:

git submodule init
git submodule update
./gradlew test

Did that work for you? If you get on the order of 90,000 tests then the test262 tests ran...

--
You received this message because you are subscribed to the Google Groups "mozilla-rhino" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mozilla-rhin...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mozilla-rhino/d2444872-d80b-4d65-bb0f-553d8130cb75n%40googlegroups.com.

[P]

Hi Greg,

Yeah, running all tests worked, was under the impression one could run just the test262 suite somehow, as running all tests take +/- 30 minutes on my machine

But that's  not an option, you have to run the entire testsuite?

I was planning in modifying the test262.properties to only run a few sets of tests in the area I'm working on, as to reduce the feedback loop from +/- 30 minutes to run all tests to hopefully max a few minutes to run only a specific set of tests from the test262 suite.

If this is currently not possible, does it sound feasible to make this possible? Never done any gradle stuff before, but if it seems feasible I probably will dig into it

Paul

[Gregory Brail]

If you can find "Test262SuiteTest" in the source tree, then it's got all the logic to parse the properties file, figure out which tests to run, and run them.

One thing you can do is set the environment variable (not Java system property) TEST_262_OPTLEVEL to -1. 

By default, the Test262 suite runs each test three times at three different optimization levels. By setting that level (to anything, really) you only run each test once, which is fine when you're building stuff. Furthermore, compiling actually takes some time so setting it to -1 (interpreter mode) actually makes the test suite run fastest in my experience because the tests are all very short.

With that set, I've had success just running "Test262SuiteTest" in Intellij and it's not terrible.

I could see how some improvements to that test runner might produce another environment variable that makes it possible to cut down the list of tests even farther.

But yeah, for real new feature development I find it easiest to first create a test case that runs as a standalone script, like the ones in the testsrc/jstests directory, and then follow up by enabling Test262 when closer to check the smaller details of spec compliance.

To view this discussion on the web visit https://groups.google.com/d/msgid/mozilla-rhino/9d7b66a4-5839-41da-a444-6f7139811b4fn%40googlegroups.com.

[P]

Tnx, I'll go and fiddle :-)

[P]

Quick update: figured out I can just run the  Test262SuiteTest  suite from the commandline with teh following command:

 `./gradlew test --tests org.mozilla.javascript.tests.Test262SuiteTest`

And if you then modify testsrc/test262.properties to only contain the set of tests you're interested in, running the tests is taking a lot less time: I just run the Proxy and Reflect tests and they're done in +/- 20 seconds

[Gregory Brail]

Thank you! If you'd like to submit a PR with additional helpful docs in the README.md I bet a lot of people would appreciate it.

To view this discussion on the web visit https://groups.google.com/d/msgid/mozilla-rhino/ebb6138f-cdcc-4435-bf66-5316bf99db98n%40googlegroups.com.

[P]

yeah, already cooking up something. Also added the ability to set the optimizationLevel and which .properties to use from the command line

Expect a PR this week or next

[P]

Took a while, but: https://github.com/mozilla/rhino/pull/897