Test framework rework

[Stéphane Poirier]

Hi,

I investigated failing tests (#1944) under PhantomJS >= 2.0. There are now 20 failing tests. Fixing those is possible, but I believe it would be a big win to use Selenium for testing.

Main advantage: Test Vimium on Google Chrome, as an actual extension. PhantomJS is Webkit-based, and out-of-sync with Google Chrome releases. I'm convinced that it's the source of multiple false-positive and false-negative failing tests. This results in a false sense of confidence in tests, wasted time fixing non-broken code, and fixes risking of breaking code on Google Chrome.

Other Selenium niceties:

• Receive a "head's up" by also testing on Google Chrome pre-releases, to prevent bad surprises.
  
• Selenium has a PhantomJS driver. That opens the door to a particularly smooth test framework transition.

Some concerns (and resolution):

• Time needed: Depending on the scope, I estimate it would take me between 1-6 weeks. I'm committed to take care of the whole transition.
• Continuous integration: Selenium testing requires a GUI. SauceLabs, TestingBot and possibly Travis-CI itself could do the job.
• Using locally: Setup would remain simple. About 2 additional dependencies are required (Selenium Standalone Server, Webdriver).
  Local Selenium tests can be slightly intrusive while they run (i.e. it opens a browser window to performs its operations). This is minor IMO, but I'll investigate further if needed.
• Run time: Selenium + Google Chrome is slower than PhantomJS, but nothing worth mentioning.
• Learning curve for writing tests: Little to none. The current API can be used.
• Code boundaries: As PhantomJS, Selenium is not intrusive and easily replaceable in the future, if need be.

This is an important task and I'd like to get your opinion and insights.

Best regards

[Stéphane Poirier]

Another major advantage: test the extension on external/live webpages. That can't be done with the current infrastructure.

[Stéphane Poirier]

Update: With the new headless Google Chrome in version 59, Selenium/GUI might not even be required.

[mrmr1993]

It might still be worth considering Selenium over headless Chrome: it could also be used for tests to the (functioning) Firefox and (possible) Edge builds without duplicating much/any code.

[Stéphane Poirier]

If Vimium is to be gaining grounds on other browsers, I fully agree. I'll fork and experiment.