Testing Vimium

[Jez]

Unittests run fine under plain node, but some tests, like those for the link hints, require the DOM environment. Ideally, we should be running all these tests as pre-commit hooks to ensure that we catch bugs right when they are created. What do you guys think of using PhantomJS to provide the DOM for those tests that require it? It seems less hacky than using a Javascript-simulated DOM library (like envjs). We'd still have to mock out the chrome.* APIs, of course.

Jez

[Jez]

Unfortunately it seems like PhantomJS isn't sufficiently compatible with Chrome. Our tests pass on Chrome but not in PhantomJS. I'm going to have a look at ZombieJS to see if it is any better.

Jez

[Jez]

ZombieJS' parser chokes on perfectly good JS too. Oh well.

Jez

[Jez]

Turns out that PhantomJS was a viable solution after all. I've things up here.

Jez

[Jez]

Now that we have testing set up, it would be nice to have coverage reports too, so we know what else needs tests written. Do you guys know of any good coverage tools? Ideally, the tool should be able to run tests in a browser's DOM environment, but also have scriptable access the filesystem (so it can look for all *_test.js files automatically). Here's what I've found so far:

• Mocha provides coverage reports and seems fairly popular, but it seems like a monolithic framework, so we can't just plug into it.
• I've seen a number of links to JSCoverage, but it's bundled as a browser-only setup. It also hasn't had an update since 2010, but from a preliminary test, it seems to work with our codebase just fine. In fact, Mocha relies on a forked copy of JSCoverage to instrument the code.
• There are a bunch of coverage modules on npm like node-cover and coveraje, but I'm not sure how good those are. And it's not clear whether they support running tests in a browser environment. Also, they seem moderately popular, but not on the level of Mocha.

Since JSCoverage seems fairly popular, I'm thinking of writing a PhantomJS driver for it -- possibly as a small standalone project since it won't involve Vimium-specific code. It would look for the *_test.js files in the filesystem and pipe them through to JSCoverage, then save the results to disk. Of course, I'd be more than happy to use an existing solution if it suffices.

Jez