Buster for continuous integration

[Christian Johansen]

There's been some discussion around how to use Buster for CI lately. I think we need a new binary, `buster-ci`.

Some relevant input:
- We're missing one feature that makes JsTestDriver very useful for CI: the ability to spawn the server, browsers and run tests all in one go
- Tiago recently did a presentation on Buster CI with Phantom: http://trodrigues.net/presentations/buster-ci/
- Based on Tiago's work I recently set up CI for a client. One of the things I did was to use this script: https://gist.github.com/2655403

My suggestions for the buster-ci binary:
- It should accept (almost) all the options of both buster-server and buster-test
- It should have the ability to spawn browsers. Ideally we'd figure out some way of configuring how to start certain browsers (like `firefox -P special-profile`)
- It should be able to spin up Phantom too
- It should spawn the buster server, unless it's already running. This needs to either be an option (e.g. `buster-ci --no-server`) or we use some heuristics. I guess the option will be faster.
- It should run tests, passing in relevant options.
- It should stop the server, IF it also started it, and we didn't ask it to keep running (e.g. buster-ci --persistent-server)
- Not sure, but I suspect supporting a separate CI configuration file would be useful. What browsers to run and so on.

Please use this thread for brainstorming so we're sure we're covering all the bases.

Christian

[Christian Johansen]

I should also mention that I've recently rearranged our main CLI helper stuff, buster-cli (https://github.com/busterjs/buster-cli/tree/stateless). If anyone decides to start hacking on this, I'd love to help build up the skeleton at least.

Christian

[sas...@dawandamail.com]

i just made buster work on travis, based on headless firefox: https://github.com/dawanda/jquery-rss/blob/master/.travis.yml

[Tiago Rodrigues]

This sounds like an amazing idea!

JSTestDriver does have an ability to capture browsers when you start the server (can't be sure about spawning them).

I guess having something like a list of commands to spawn browsers would be a good idea. You can also spawn a headless firefox version, or you might have a head and would like to spawn actual browsers.

Of course that if you're running browsers on virtual machines (say for instance windows VM's) you can't probably spawn them so you can just capture them.

And I do like the idea of having a config file for this. Different projects have different needs and passing everything in CLI parameters becomes confusing (hence all the scripts to fire test runs).

[Rod Vagg]

How does the browser end in that scenario? Is it forced-quit when Travis gets a return from `npm test`?

[Tiago Rodrigues]

I don't know how Travis does it, but on Jenkins, if your job spawns
processes and they are not terminated by your job it will complain
about it and terminate those processes by itself.

--
Tiago Rodrigues
http://www.trodrigues.net
E-Mail / MSN Messenger / Jabber / GTalk:
  tmcrodrigues [at] gmail [dot] com
Skype: trodrigues.net

[Tauren Mills]

I've been very interested in buster since discovering it a few months ago, but haven't had time to really play with it yet. I'm definitely planning to use it on my own projects where I have full control over environments.

I've suggested we consider using it at Nike (my day job), but have met with some resistance primarily because it was unclear how well it would work with TeamCity and java/maven build processes. It may be a battle to get node available on our CI servers.

Is there any information, advice, or projects that might help my cause. Others without node.js experience are leaning toward jsTestDriver currently with its better java integration.

Thanks,
Tauren

[August Lilleaas]

CC-ing Malcolm, he said he's interested in writing the "start server and spawn browsers" part to scratch his own itch. We could then improve on that code and add all the other features.

[Malcolm Locke]

On Fri, May 11, 2012 at 06:26:44PM +1000, Rod Vagg wrote:
> How does the browser end in that scenario? Is it forced-quit when Travis
> gets a return from `npm test`?

Not 100% sure, but I expect Travis just shuts down the whole VM the test
runs on once the test command completes. I believe they use a VM
snapshot as the starting point for all test runs.

Malc

[Malcolm Locke]

Thanks August, I'm already subscribed to buster-dev though so no need :)

I'd personally like there to be a way spawn buster server and a
configurable list of browsers and for the these to persist until the
spawning process gets SIGTERM.

Agree with having a config file, ideally with something like this as a
bare minimum:

browsers: ['chrome', 'chromium', 'firefox']

and with additional flexibility to configure paths, options, etc. for
each browser for when the defaults don't work. Mostly a tokenized
string for the command to run would probably be enough I expect, with
token values for hostname / port (anything else?) swapped out.

Not sure of a sane way of stopping all the potential races from
occurring, particularly in CI mode, i.e.:

- No browsers should try and capture until buster server has fully
started.
- In CI mode, buster test shouldn't try and run until all the required
browsers are captured.

Malc

[Christian Johansen]

Agree with having a config file, ideally with something like this as a
bare minimum:

 browsers: ['chrome', 'chromium', 'firefox']

and with additional flexibility to configure paths, options, etc. for
each browser for when the defaults don't work.  Mostly a tokenized
string for the command to run would probably be enough I expect, with
token values for hostname / port (anything else?) swapped out.

Good idea. This way you could even allow configuring custom browsers too, like:

browsers: ['firefox4', 'firefox5'],

somethingSomething: {
    firefox4: {
        command: "/home/christian/bin/firefox4 -P ff4"
        ...
    }
}

Or something like that.
 

Not sure of a sane way of stopping all the potential races from
occurring, particularly in CI mode, i.e.:

- No browsers should try and capture until buster server has fully
 started.

Easy. I'm reworking the buster-server binary as a separate module now, and focusing on API. The new module will allow you to do:

busterServerCli.create({ ... }).run(args, function (err, server) {
    // Server is ready unless there was errors
});
 

- In CI mode, buster test shouldn't try and run until all the required
 browsers are captured.

Given the above API, you could ask the server object to check that the number of connected clients is consistent with the number of spawned browsers :)
 

--
MVH
Christian

[August Lilleaas]

Just some input on the API. I would prefer this:

    browsers: [buster.ci.firefox, buster.ci.chrome, buster.ci.chromium]

These would be objects that specify how the browser should be spawned, something like {name: "Firefox", command: "/path/to/firefox {{url}} -no-remote"}.

I'm generally not a fan of just passing magic strings, it means we create a black box you need to figure out, and we also need code to test if it's a string, look it up, etc. If it's always just an object, it's easier for everyone imo.

[mru...@gmail.com]

Hi !

Thought I'd chip in on this thread as I'm guessing 1.0 is closing in which again might mean that a buster-ci module might not be so far off :-)

I've seen a couple of questions on this forum regarding ci support and ci servers.
Thought I'd give a heads up on one possibly helpful alternative  in the jvm space.
I've created a gradle-buster-plugin. I mostly use gradle for my builds and gradle happens to be pretty well supported by a range of ci-servers.
Meaning you don't have to adopt gradle for your whole build to use the plugin, one can easily set up a separate build step on most ci servers and let that use the plugin. (Personally tested and using it on jenkins and teamcity. Also know that is in active use on bamboo).

I've added some rudimentary browser capture support (using selenium) to the plugin, but it would be very useful to move this functionality to a buster-ci module.  I copped out on trying to do this as buster module as my node skills are uhm zilch and my javascript skillz probably not quite up to buster std as of now :-)

Wishlist/ideas for buster-ci:
- Easily capture browsers (maybe selenium, or maybe a different route altogether ?)
- It would be neat if it was easy to set up an environment which support running browser on a remote machine
-- Something like a selenium grid, but with close to no pain to set up grid nodes. Even on my local setup witth a mac host, windows and linux guest os this would be useful, but even more so in bigger ci settings.
- Solid process handling, no dangling processes (but if you want to; an option to keep server and captured browsers running)
- If indeed you leave server and captured browsers running; api to query about state and possiblity to stop server (and release/stop capture browsers)
- Friendly API, to make it easy to integrate with (from jvm build tools and ci servers)

From the jvm side of thing, it might be that a thin java wrapper for the api might be useful for a range of build tools like Ant, Maven, Leiningen, SBT, buildr in addition to gradle. That's certainly something I could help out with once the buster-ci module starts to shape up.
 
cheers
Magnus

[August Lilleaas]

Thanks for taking the initiative on ironing out whatever needs to be done to work in a CI environment!

It makes sense to look to existing solutions like selenium for capturing browsers. Sounds like a lot of trouble to re-implement all of that.

We already support running browsers on a remote machine. All you need to know is the host and port of the buster  server instance, and open http://host:port/capture. The "/capture" path is a public API that will stay true to semver. Is there anything else we can do to make it easier to capture browsers on remote machines?

For selenium grid, I'm not sure why it would make sense for buster to re-implement that. Is it easy and/or just as good to integrate with selenium grid? Not sure what exactly buster needs to do in this regard.

We should indeed have solid process handling. Initially the idea was to have a single buster-server instance that stayed alive across test runs, and that had browsers captured on it that you opened manually via VNC or whatever and stayed open forever. This is however only sensible in local environments, where you want fast test runs and have your scripts cached on the server etc. For CI, it definitely makes sense to start a new server for each CI run, so it's easy to achieve parallelism. As mentioned on the issue tracker, we should make it easy to start a server on an ephemeral port.

In light of this, I'm not sure if there should be a public API to query the state of a server from the outside. Servers should die a lot in CI, and CI is where querying would be useful. This is of course very open to discussion.

Various wrappers for various platforms to start the node process for buster-server would be very nice, and we'd love to host it on github.com/busterjs.

--
You received this message because you are subscribed to the Google Groups "Buster.JS development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to busterjs-dev...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[mru...@gmail.com]

A few comments below:

On Tuesday, 8 October 2013 21:52:11 UTC+2, August Lilleaas wrote:

Thanks for taking the initiative on ironing out whatever needs to be done to work in a CI environment!

It makes sense to look to existing solutions like selenium for capturing browsers. Sounds like a lot of trouble to re-implement all of that.

Looks like Karma went their own route, might be worth having a peak to consider the alternative at least.
 
 

We already support running browsers on a remote machine. All you need to know is the host and port of the buster  server instance, and open http://host:port/capture. The "/capture" path is a public API that will stay true to semver. Is there anything else we can do to make it easier to capture browsers on remote machines?

If you do decide to go down the selenium grid route, the actual capture part of remotes could be made transparent to end-user, he/she would only need to define which browsers to test with, buster-ci could then just communicate with the selenium hub to find available webdrivers and use those to perform capture.

 

For selenium grid, I'm not sure why it would make sense for buster to re-implement that. Is it easy and/or just as good to integrate with selenium grid? Not sure what exactly buster needs to do in this regard.

Maybe nothing, or maybe it would make sense for buster-ci to start a selenium hub. Starting nodes would have to be the responsibility of the user. However setting up a selenium node typically involves several steps. Would be neat to provide some assistance in automating those steps. If I would implement it I would probably have created a gradle script that: downloads appropriate selenium jars, downloads any webdriver executables needed and start the node with appropriate config set.

 

[Garrick Cheung]

Hey guys,

A config file would be very useful, especially for listing desired browsers and grid/hub config info.

I think the config should contain objects specifying how browsers should be spawned. It could be something like:

browsers: [{

browserName: 'firefox',

platform: 'mac',

version: 28

}, {

browserName: 'chrome',

platform: 'windows',

version: 24

}]

Or something like the config I saw on theintern (https://github.com/theintern/intern/wiki/Configuring-Intern):

browsers: [{

browserName: 'chrome',

version: ['23', '24'],

platform: ['Linux', 'Mac OS 10.8']

}]

It should probably also contain grid/hub/server configs. Not to start up, but to connect to

server: {

hostname: 'localhost',

port: ''

}

WD node module has a good example of the server config. It also can be a string like "http://localhost:123456".

There's a selenium-grid-status node module that provides a method for returning a list of available nodes. I think it's only a matter of filtering out the desired browsers in the config with the ones available in order to retrieve sessions from the selenium-grid. Then we can have those nodes captured by buster-server.

I've been playing with selenium-webdriver and WD node modules for a while. Although their main purpose is webdriver control, selenium-webdriver provides a solution for starting up a server (https://code.google.com/p/selenium/wiki/WebDriverJs#Using_the_Stand-alone_Selenium_Server). It may be possible to start/control a grid and nodes as well. Unfortunately some recent code refactoring broke the solution in selenium-webdriver (tagged selenium-2.35.0). I've filed a ticket about what broke and am awaiting for the fix for it.

As of Phantomjs 1.8, GhostDriver is built in to Phantomjs so it's easy to start Phantomjs as a webdriver: http://phantomjs.org/release-1.8.html