Automated browser-based testing with Clojure (blog post)

[Sean Corfield]

A bit of promotion for clj-webdriver:

http://corfield.org/blog/post.cfm/automated-browser-based-testing-with-clojure

Also interested in feedback from clj-webdriver users around techniques
shown in that post and the issue (in particular) of running a whole
series of tests in a single browser session. Also, how do folks
develop and test their webdriver scripts?
--
Sean A Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/
World Singles, LLC. -- http://worldsingles.com/

"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)

[Mayank Jain]

Hi,
I already added your blog post to clj-webdriver External Links Section. :)

I found your way of doing the test very interesting. I'll have to take a proper look at it before I can comment on it. Will share my feedback soon.

The most interesting problem that I am facing is the state that we have to maintain before some tests. Some tests are just independent of states while others need some state to be ready.

Right now I am trying to figure out what is the right way of running same tests across multiple browser/platform/versions in parallel using Grid Feature.

Will share my thoughts on it once I am done experimenting. :)

Thanks for sharing.
Cheers!

[semperos]

Sean,

Thanks for the promotion. I'm glad to see you've gotten some good use out of clj-webdriver. Your post presents an interesting mechanism for getting tests across namespaces to run in a single browser session. Sounds like the beginnings of a nice lein-webdriver project ;)

If the need for a single browser session across namespaces has to do with the state of your application, then the tactic you implement in that post seems like a fine approach. However, if the issue is simply performance, a couple of points come to mind:

 * After the initial opening of a given browser on a given box, subsequent openings/closings are generally quite speedy

 * Selenium's Grid system provides a way to run tests across several machines with various configuration scenarios in parallel

As for authoring tests with clj-webdriver, I don't have any profound observations. As a common Clojure mantra, I'd remind folks to leverage Clojure's strengths and avoid one-to-one translations of their existing Selenium-WebDriver test suites.

As a personal preference, I do tend to author my tests in two sweeps. After first outlining the steps (including CSS/XPath selectors for all target elements) necessary for a test run, I take a dedicated pass through my tests to determine where Selenium-WebDriver needs help waiting for the page to be "ready" (usually only determined by running the code and seeing where Selenium-WebDriver gets ahead of itself, often associated with JavaScript execution and AJAX callbacks). My testing functions often get broken down into units of waiting, one wait per function. This tends to keep my testing functions at a proper level of abstraction.

As a final note, I've begun a separate project called webdriver-logic that leverages clj-webdriver using core.logic to author declarative tests. The idea is to use plain clj-webdriver to bring your application to a certain state, but then to use webdriver-logic to verify several disparate facts about the state of your application at that point. It's still in pre-pre-alpha, but I think it opens up some interesting directions for authoring tests.

-Daniel

[Sean Corfield]

On Sun, Sep 9, 2012 at 8:46 PM, semperos <daniel.l...@gmail.com> wrote:
> * After the initial opening of a given browser on a given box, subsequent
> openings/closings are generally quite speedy

Yes, definitely faster than the first open but with hundreds of tests
it's still going to be a noticeable difference. Plus it's also really
annoying to have an app constantly opening and closing while you're
trying to work - it constantly steals focus and opens in front of
whatever your working on, effectively preventing you from working
while the test suite is running.

> * Selenium's Grid system provides a way to run tests across several
> machines with various configuration scenarios in parallel

Right now we're looking at how to use this in our Ant script which
every developer can run locally on their own machine. We're a ways off
trying to run our test suite on multiple browsers on multiple
machines. Hopefully we'll get to that point...

> As for authoring tests with clj-webdriver, I don't have any profound
> observations. As a common Clojure mantra, I'd remind folks to leverage
> Clojure's strengths and avoid one-to-one translations of their existing
> Selenium-WebDriver test suites.

Having tried to do a couple of 1:1 translations already, I'll heartily
agree with that :)

> My testing functions often get broken down
> into units of waiting, one wait per function. This tends to keep my testing
> functions at a proper level of abstraction.

Interesting.

Do you develop tests via the REPL at all? Or do you code them straight
up in a file? Do you run them thru a test harness or thru some manual
process (or simple driver)? How do you decide at what points to test
for certain on-screen values (and how much such testing do you do)?

[semperos]

Yes, definitely faster than the first open but with hundreds of tests
it's still going to be a noticeable difference. Plus it's also really
annoying to have an app constantly opening and closing while you're
trying to work - it constantly steals focus and opens in front of
whatever your working on, effectively preventing you from working
while the test suite is running.

Understood. I agree, it is rather obnoxious to have browsers pop up and steal focus. As an aside, Chrome is generally better about this than Firefox.

 

Right now we're looking at how to use this in our Ant script which
every developer can run locally on their own machine. We're a ways off
trying to run our test suite on multiple browsers on multiple
machines. Hopefully we'll get to that point...

As I mentioned above, I think you've come up with a clever way to make your setup performant (i.e., keep one browser for the whole suite). If you plan on moving to a Grid-based setup at some point, you can start with a Grid on a single machine (start one hub and one node locally; clj-webdriver has some bash scripts to help folks get started). With the new DesiredCapabilities support being added to clj-webdriver in the next release, you end up writing your test suites without any concern for specific browsers, then use your Grid configuration and DesiredCapabilities to decide which browsers, OS'es, etc. to test with during a given run.

 

Do you develop tests via the REPL at all? Or do you code them straight
up in a file? Do you run them thru a test harness or thru some manual
process (or simple driver)? How do you decide at what points to test
for certain on-screen values (and how much such testing do you do)?

I personally start in the REPL, copy out successful REPL forms into a file, then as I mentioned, generally have to run the file and take a second pass to figure out where Selenium-WebDriver needs me to tell it to wait (at the REPL, there's so much time in between my own REPL entries that waiting isn't an issue up front). I've integrated my Selenium-WebDriver tests as part of larger Maven builds, intended as system-level tests at the very end of a CI pipeline. As for values to test, I'm sure there are several schools of thought. I'd test for as many on-screen values as impact the behavior of the app and determine correctness; clj-webdriver's Taxi API, combined with Clojure's own expressivity, make for some fairly terse yet readable tests, so there's no need to skimp.

At a higher level of test authoring, the Selenium-WebDriver folks push the "page object" design pattern, which for Clojure would involve putting together a collection of low-level functions using clj-webdriver (e.g., fill-in-user-name, click-login-submit, profile-link-visible?, etc.) that become the building blocks for concept-level functions (e.g., login, logged-in?, etc.), which is what you author your actual tests with. Again, this is just about levels of abstraction, but dealing with the nitty-gritty of DOM elements and timing should be somewhat hidden at the highest level of test authorship so testers can focus on app behavior and exploration.

-Daniel

[Daniel Renfer]

Since we're all sharing how we use clj-webdriver, I'd like to point
you to my code.

I do things a bit differently as I'm using Cucumber for all of my
webdriver testing. This part of my testing cycle gets the least amount
of attention and these tests are broken more often than not, but maybe
it can help to see another way that people are using this library.

I too have had issues with controlling when browsers open and close
and have taken to just using the :htmlunit driver most of the time. I
haven't been maintaining these tests since I did a major js overhaul,
so I'm sure that won't be sufficient soon.

Browser startup is a real issue for me as I usually develop on my
laptop with a SSH forwarded shell and emacs. When firefox gets
launched, it's over the wifi to my mac. (real speed drain)

Most of my webdriver usage is at
https://github.com/duck1123/jiksnu/blob/master/test/jiksnu/features_helper.clj
but if you want to see the rest of the test code, it's in /features

If you can't make heads or tails of that, I don't blame you. This file
needs more love soon.

[Sean Corfield]

Thank you for your long and thoughtful answer! Responses inline below...

On Tue, Sep 11, 2012 at 8:52 PM, semperos <daniel.l...@gmail.com> wrote:
> on moving to a Grid-based setup at some point, you can start with a Grid on
> a single machine (start one hub and one node locally; clj-webdriver has some
> bash scripts to help folks get started). With the new DesiredCapabilities

Sounds good. When I resurrect our CI environment, I'd love to go down
this path (long story).

> I personally start in the REPL, copy out successful REPL forms into a file,

OK, that's been how I've been working. It seems to make a lot of sense
with Clojure.

> larger Maven builds, intended as system-level tests at the very end of a CI
> pipeline. As for values to test, I'm sure there are several schools of

Likewise. We run unit tests, integration tests (pure server-side at
the controller level), browser-based Selenium tests (legacy) and now
Clojure-based webdriver tests at the end.

> thought. I'd test for as many on-screen values as impact the behavior of the
> app and determine correctness; clj-webdriver's Taxi API, combined with

Good insight, thank you.

> together a collection of low-level functions using clj-webdriver (e.g.,
> fill-in-user-name, click-login-submit, profile-link-visible?, etc.) that
> become the building blocks for concept-level functions (e.g., login,
> logged-in?, etc.), which is what you author your actual tests with.

That seems to be the natural direction we're heading. We're building a
small library that's close to a DSL for interacting with our
application, an outgrowth of just making readable tests with reusable
functions.

Thank you for confirming that we're probably going in the right
direction and making sane choices!

[Nelson Morris]

> Browser startup is a real issue for me as I usually develop on my
> laptop with a SSH forwarded shell and emacs. When firefox gets
> launched, it's over the wifi to my mac. (real speed drain)

>> Understood. I agree, it is rather obnoxious to have browsers pop up and
>> steal focus. As an aside, Chrome is generally better about this than
>> Firefox.

If you are running the tests on a linux system, I highly recommend
`xvfb-run` to create a headless X session and run the browser tests in
it. I don't know of an equivalent for mac/windows unfortunately.

[Mayank Jain]

On Wed, Sep 12, 2012 at 6:47 PM, Nelson Morris <nmo...@nelsonmorris.net> wrote:

If you are running the tests on a linux system, I highly recommend
`xvfb-run` to create a headless X session and run the browser tests in
it.  I don't know of an equivalent for mac/windows unfortunately.

Do you know how to run firefox in headless mode via clojure? Or do you set this up prior to running your clojure test scripts?

[Nelson Morris]

I would expect `xvfb-run lein test` to work, as it opens an headless X
session, then runs the command with it, and closes the X session.
That is similar to how I would run selenium tests with
ruby/capybara/webdriver. I've not tried with clj-webdriver as my
needs for web testing in clojure so far have been handled by
peridot/kerodon.

[Daniel Gregoire]

Using xvfb is a system-level thing; if you have it running and all the right settings set, it should "just work." I've used the xvfb integration with Jenkins to run standard Java Selenium-WebDriver tests on Firefox without any issue. Even services like Travis-CI support using xvfb.

-Daniel

[Mayank Jain]

The reason I asked is I saw this thread where they switched to headless mode in python and back to run the selenium code.
So I was wondering if I could do that in clojure as well.

[Nelson Morris]

Looking at https://github.com/ponty/PyVirtualDisplay/blob/master/pyvirtualdisplay/abstractdisplay.py
it appears to be doing the same thing, just with a python wrapper and
processes instead of a shell script. I'm not aware of any libraries
in java or clojure that do it.