Integrating AMDJS Tests w/ Frameworks

[Jakob Heuser]

Hey everyone,

Post Inject 0.4.0, I've been working with the team here at LinkedIn to improve our Travis-CI integration. One of the major discussion points revolved around what we're calling "spec tests". AMD has an online test suite (https://github.com/amdjs/amdjs-tests) and CommonJS has a set of agnostic tests (https://github.com/ashb/interoperablejs or https://github.com/commonjs/commonjs/). With checkins to the code base, we'd like to have Travis auto-check against our supported specifications to ensure compliance doesn't go away. Here's our working ideas so far:

- Create a standalone public site (removes the cookie / xss fear) which lets you load your own custom config + impl and run the test suite

The idea behind this is similar to how AMD's loader already works with a config= and impl= in the URL. One bootstrap to write out the source files, and away you go. Currently the AMD JS tests prevent non-relative paths, but for the online-only loader this may not be a problem. Security concerns go away as the site is on a domain nothing else is using. An overly simplified prototype is at http://amdjs-online.github.com/amdjs-tests-online/start.html to prove the concept out.

- Import our spec tests as git submodules

This was our plan B. Potentially, all AMD libraries could pull this in as a submodule. Tests could be ran for AMD since ../../../../...etc would still be a valid path. CommonJS lacks any of this functionality, but that's outside the scope of this list. I have a branch of Inject that is trying this, but I still haven't solved how to make PhantomJS and Dojo Objective Harness play nice.

- Reimplement the tests in our QUnit framework (we're currently doing this... and it makes us sad) :)

We have a suite of QUnit tests based on the specification. However, there's no assertion of perfection, and we must still run the spec tests (and pass) before we move from RC to final. So all this is providing us with is a bit of sanity as we go. It can't serve as a signoff though.

I'd love to hear this group's ideas on how to approach the spec testing problem.

--Jakob

[Domenic Denicola]

This question of spec-testing is of interest to the nascent Promises/A+ standard as well. Currently we are planning on an approach similar to that of the existing Promises/A test suite that I wrote:

https://github.com/domenic/promise-tests

You can see how when.js integrates it into their npm test script here:

https://github.com/cujojs/when/blob/master/package.json#L38

and RSVP.js integrates it into their Rakefile here:

https://github.com/tildeio/rsvp.js/blob/master/Rakefile

The test-in-browser question is a bit stickier, and seems to be a lot of what you're addressing. RSVP.js has somewhat of a solution, and for the Promises/A+ suite we'll probably have something similar (but without the Ruby dependency). I'm not sure if that kind of thing is robust enough for your needs though.

Hope this is helpful and not too off-topic!

[James Burke]

I'm open to figuring out how best to do this.

Pusztai Tibor recently got almond working with travis, and almond uses
DOH. Some files that may be interesting:

https://github.com/jrburke/almond/blob/master/.travis.yml

which calls phantomjs with this file that hooks into the DOH reporting:

https://github.com/jrburke/almond/blob/master/tests/runner.js

That said, I'm open to switching to qunit, particularly if others felt
it was easier to add tests. The one thing I like about the DOH tests
is that they can run each test in an iframe and aggregate the results
in the top frame. In the past I have not seen a way to do that with
qunit, but qunit has changed over time, so I could be behind the
times.

As far as running the tests:

I like the idea of having impls just committed to the test repo and
then we can use a travis pass, similar to the one used in almond, to
get any pushes validated.

I can see some advantage of having the tests on a stand-alone domain
-- this would allow potential AMD implementers an easy way to try out
testing without having to clone a repo, and get wider browser test
coverage. Although, at the same time, encouraging a checkout of the
test repo so they can run locally may nudge them to doing the pull
request to get into the repo.

So, if we want a way to start, maybe just wire up the amdjs-tests repo
to use the same travis approach as almond as a start.

Although, I'm not sure if that meets with what you want, in
particular, if you want to do more "as you develop" testing for an
individual loader, that may not be ready for inclusion as a commit to
the test repo. Maybe that just means using a branch of the repo? Just
thinking out loud.

James

[Jakob Heuser]

QUnit Composite supports individual iframe testing. What I've managed to do is create a shim for DOH that reaches in to start the tests automatically and capture the output.

https://github.com/jakobo/inject/tree/test_restructure

Travis-CI: https://travis-ci.org/#!/Jakobo/inject/builds/3084570

The above is a branch that shows how we're doing our integration. Some specific files to look at:

https://github.com/Jakobo/inject/blob/test_restructure/build_tasks/server.js - this file creates some special redirects to support the DOH tests at a non-root level. Specifically runner.js and _browserRunner.js need to be in the "root". We remap these to their real location via redirect. We also redirect the impl/inject/inject.js and impl/inject/config.js files to versions living in our source code.

https://github.com/Jakobo/inject/blob/test_restructure/tests/spec/amd/all.html - this is our QUnit driver file. We reach into the Dojo Harness, and create wrapper functions that enable us to listen to interesting events.

For our purposes, this method has allowed us to support both AMD and CJS in our tests. If any other libraries are interested in this kind of integration, let me know and I'll be more than happy to help.

--Jakob

[James Burke]

A bit late in responding, but I fully support if you would like to
rewrite the tests with qunit and qunit composite instead of doh. My
main thing was wanting to run each test in its own fresh iframe and
getting aggregate results at the top. doh included a basic Deferred
too that allows for longer running async tests, but I am sure there is
a way to get that feel in QUnit.

If part of that also includes a travis yml to run the tests via
travis, that works for me too.

James

[Jakob Heuser]

Hi everyone,

I've taken the feedback, and decided portability may be better long term rather than something like a QUnit adaptation. Taking the CommonJS/InteroperableJS suite ideas, https://github.com/Jakobo/amdjs-tests/tree/universal contains a universal set of tests.

• The individual tests can be ran in your own framework, with the only requirement being the implementation of amdJS.print(message, passFail) to bridge into your library's testing harness
• The suite comes with a minimalist test harness that uses the same iframe sandboxing that made DOH very effective for AMD runs
• The suite also comes with a built in web server via node, making test acceptance a near-zero operation

I'd love to get some feedback on the current iteration. This could be very helpful for library maintainers who want to automate regression tests against the official suite.

--Jakob

[James Burke]

I'm not opposed to the change, but hard for me to see the advantage.
This looks like swapping out DOH for an amdJS test object. I'm not wed
to DOH, but also need to understand more what the change gains.

Perhaps I'm not understanding the first point you mention:

> The individual tests can be ran in your own framework, with the only requirement being the implementation of amdJS.print(message, passFail) to bridge into your library's testing harness

Is the idea that I could reuse these same tests, but not have then run
via this amdJS test runner? But it seems like the amdJS test runner
needs to run anyway to get the final test reports? If something like
travis is available, is just a travis.yml needed for the existing form
of the tests?

bug report: selecting requirejs causes the server to fail:

Users/jr/git/temp/amdjs-test-jakobo/server/server.js:55
res.end(data.toString());
^
TypeError: Cannot call method 'toString' of undefined

James

> --
> You received this message because you are subscribed to the Google Groups
> "amd-implement" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to amd-implemen...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Jakob Heuser]

Yeah, that's the general purpose. If you implement the amdJS.print/group/done(), you can tie those tests to your own testing framework and not use the AMD test runner. For example, the CommonJS tests at https://github.com/ashb/interoperablejs are pulled into Inject's QUnit suite at https://github.com/linkedin/inject/tree/master/tests/spec/cjs Then, Inject can then run the CJS specification tests alongside its normal test suite without changing frameworks. This allows the spec to seamlessly integrate with the CI setup.

The original concept was super-rough, as I figured early feedback was better than a polished version. I'll add some safety checking to the node server.

[James Burke]

Sounds fine with me. I do not have any objections to switching if the
rough edges are cleaned up.

It will require people to use node to run the tests. Ideally it would
not be needed, but I can see where it may give a nicer feel for the
tests and switching between implementations. So I'm OK with it.

Others should speak up if they have any questions or concerns.

James

[Jakob Heuser]

Based on your feedback, some offline feedback, and some discussions w/ people who are using the interoperablejs framework, I made a bunch of changes to the test execution structure at https://github.com/jakobo/amdjs-tests/tree/universal

First, and most importantly, the reporter structure was reduced down to be self contained. Implementing window.amdJSPrint(message, type) makes it very easy to link to an existing framework. No additional methods to override required. As an added bonus, if you'd like to run your own reporter, your framework's config can map "_reporter" to the AMD module of your choosing, it needs to export assert() and print().

The second change is for the built-in test runner. Stripping away the reporter code made the final implementation much simpler. The iframes can still be used, and while I haven't added any "rerun" functionality, individual tests can be clicked into and ran multiple times. console.log() is still used as the main messaging system.

Finally, the whole thing is hooked up to Travis-CI with its first run @ https://travis-ci.org/Jakobo/amdjs-tests/builds/4974490 

--Jakob

[Jakob Heuser]

Just circling back with everyone on the revisions for the universal tree. Specifically, I'd like James to weigh in as the one who had the largest amount of feedback and suggestions around the changes.

For those new to the discussion, we are considering removing DOH as a dependency for AMD. I have a proposal that behaves more like InteroperableJS, the CommonJS specification tests. The branch can be found at https://github.com/jakobo/amdjs-tests/tree/universal and Travis-CI reports only one issue with lsjs which was fixed in their github.

[James Burke]

Just tried it out. Looks good to me. As far as I'm concerned, feel
free to merge it in directly, just send out email when done. At that
point I'll also do an update for require.js since I have had a couple
of point releases since then.

James

[Jakob Heuser]

To relaunch, I don't mind pulling in the latest versions of everyone's code as well... I think a refresh could be a good thing.

[James Burke]

I wouldn't block on it, but go for it if you feel up to it.

For requirejs, this link will always have the latest release:
https://raw.github.com/jrburke/requirejs/latest/require.js

James