Upcoming feature: Running tests automatically from the command line using plovr (ideal for continuous integration)

[Michael Bolin]

Hi, if you use plovr to run tests as described at:

http://plovr.com/testing.html

I just landed a change that will let you run

java -jar plovr.jar test <config_id>

to run the tests headless using WebDriver/HtmlUnit. The change is really sloppy and I'm going to be doing a bunch of follow-up work, which you can get the gist of by reading the TODOs in the change:

http://code.google.com/p/plovr/source/detail?r=47f6f2a3a783952f88d71d407fd391ac721a8478

For example, the above command for running the tests is a bit of a lie because I do not want to bundle all of HtmlUnit/WebDriver with plovr because the plovr jar is already pretty big. That means that you are responsible for including the subset of WebDriver that you need to run your tests using the various WebDrivers:

http://code.google.com/p/selenium/downloads/list

plovr will only include the selenium-java-2.21.0.jar that is needed to compile against the WebDriver APIs, not all of the drivers themselves (IE, Chrome, Firefox, etc.).

Unfortunately, building up a long classpath for Java on the command line really sucks, and the easiest way that I do it is by using Ant. For example, to test this feature, I have been using this Ant script:

<project>

  <target name="test">

    <java classname="org.plovr.cli.Main" fork="true">

      <classpath>

        <fileset dir="../selenium-2.21.0/selenium-2.21.0" includes="*.jar"/>

        <fileset dir="../selenium-2.21.0/selenium-2.21.0/libs" includes="*.jar"/>

        <fileset dir="tools" includes="plovr.jar" />

      </classpath>

      <arg value="test" />

      <arg value="pconfigs/desktop.js" />

    </java>

  </target>

</project>

Ultimately, when things are working properly, I will have better information documented on http://plovr.com/testing.html. But if you are interested in this feature, it would be great to provide feedback now on how you think it should work before I invest more time in it.

Cheers,

Michael

[Tom]

This sounds fantastic. I'd say my main requirement would be easy integration with jenkins (i.e., showing individual test results, etc.) Is that provided "for free" by selenium? I found http://wiki.hudson-ci.org/display/HUDSON/seleniumhtmlreport+Plugin and http://wiki.hudson-ci.org/display/HUDSON/Seleniumhq+Plugin.

Tom

[Michael Bolin]

I just checked in a change that adds the config option support. Basically, you pass in a list of Java class names that implement the WebDriver interface. The following would run your tests using two such test drivers:

  "test-drivers": [

    {

      "class": "org.openqa.selenium.firefox.FirefoxDriver"

    },

    {

      "class": "org.openqa.selenium.htmlunit.HtmlUnitDriver"

    }

  ],

The stuff it prints to the console right now is a bit messy, but the exit code should be correct (0 if the tests pass; 1 if they fali), which should be sufficient to use it on Jenkins or any other automated test runner.

I think the most important things for me to do next are to clean up the output and update the docs.

[Janez]

Hi Michael, 

I just hope that the new features will provide all tools to fully automated testing inside CI (hudson or http://jenkins-ci.org/)

The shortest way to CI  seems to be with jstestdriver (http://cjohansen.no/en/javascript/javascript_continuous_integration_with_hudson_and_jstestdriver)

but it seems that plovr lacks manifest output?? http://stackoverflow.com/questions/10856785/using-closure-library-with-jstestdriver to properly configure dependencies.

Cheers, Janez

[Tom Hughes]

I just wanted to follow up and say that I tried this out and have it working in a continuous integration mode using Jenkins. Thanks for working on this!

Some feedback:

* It would be nice to have better formatted output to more easily distinguish which tests failed. It looks like there is a JSUnit plugin for jenkins that looks for XML reports, so if those were generated after testing with plovr I'd probably have all I need: https://wiki.jenkins-ci.org/display/JENKINS/JSUnit+plugin

* The whole selenium classpath ant-file hack is a bit painful. Personally, I'd be ok with bundling all the dependencies into plovr. Maybe, you could have two builds of plovr: one with the selenium bundle and one without.

Tom

[Nate]

AWESOME Feature!

I managed to get this up with: java -cp "selenium/*:selenium/libs/*:plovr.jar" org.plovr.cli.Main test plovr.js where the selenium dir is the java client libraries.

A detailed junit xml style test output option and code coverage would greatly enhance this. As Tom said, including selenium would be nice, maybe as a separate download if size is an issue.

Keep up the great work!

[Nate]

Couple days of usage.. Noted a  few things, in order:

• A way to use selenium from javascript would complete the circle - maybe, http://code.google.com/p/selenium/wiki/WebDriverJs mixed with the test template
• Best possible output, stack traces, console.log, etc - https://github.com/mguillem/JSErrorCollector
• A way to run specific tests (by file or namespace).
• An option to point to a selenium grid.

On Wednesday, May 2, 2012 9:51:45 PM UTC+2, Michael Bolin wrote:

[mvn...@gmail.com]

This seems like an awesome feature. Is this still supported. When I try to run it I get this error:

java -cp "selenium/*:selenium/libs/*:plovr.jar" -server -Xmx4096m org.plovr.cli.Main test config.js

Listening on /0:0:0:0:0:0:0:0:9810

Exception in thread "main" java.lang.IllegalArgumentException

        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)

        at org.plovr.webdriver.TestRunner.<init>(TestRunner.java:32)

        at org.plovr.cli.TestCommand.runCommandWithOptions(TestCommand.java:56)

I did download selenium-server-2.21.0.zip. Is that not the right thing to download?

[mvn...@gmail.com]

OK Got it working. I guess the "test-drivers" field in the config is required for this to work.