hacking on automated testing a bit

[Lloyd Hilaiel]

yo testing crew (and everyone else if interested),

I did a bit more hacking on our vows harness and web driver stuff to try to make it so that in automated tests:
1. we can safely use the .chain() functionality of web driver
2. if any function in a chain() fails, we halt processing of remaining functions in the chain and subsequent tests
3. vows gives us great error messages right where the failure occurs
4. we shutdown the remote browser immediately after failure so that the protocol trace is available 90s sooner

Along the way I fixed the returning-user test which was trying to click a "this is not my computer" button that was not being displayed (recall that the display of this button requires 60s to pass from first usage in the current browser profile).

This one test demonstrates most of the new features:
https://github.com/mozilla/browserid/blob/kapow/automation-tests/tests/returning-user.js#L96-113

* pass an object with .onError into .chain() and that function will be invoked when a chained function invokes its (automatically supplied) callback parameter with a non-falsey first parameter
* if you supply a callback parameter, your .onError handler will not be automatically invoked
* If in your supplied callback you wish to halt processing of remaining items in the chain, you can manually invoke .haltChain() on the (web driver) browser object.

Finally, notice that there's now a third parameter to runner:
https://github.com/mozilla/browserid/blob/kapow/automation-tests/tests/returning-user.js#L130-133

a cleanup will be invoked when all the tests are complete, whether or they failed.

So my goals were because we're really writing tests where each is dependent on the success of the former. We want to fail at the first detected error, we want to cleanup fast, and we want good error messages that give us an idea of which test block failed and why…

curious for review and feedback!
lloyd

[Jared Hirsch]

Arrrrgh. Used wrong email. See below.

On Nov 4, 2012 5:11 PM, "Jared Hirsch" <ja...@jaredhirsch.com> wrote:
>
> Comments inline

>
> On Nov 4, 2012 1:18 PM, "Lloyd Hilaiel" <ll...@mozilla.com> wrote:
> >
> > yo testing crew (and everyone else if interested),
> >
> > I did a bit more hacking on our vows harness and web driver stuff to
try to make it so that in automated tests:
> > 1. we can safely use the .chain() functionality of web driver
> > 2. if any function in a chain() fails, we halt processing of remaining
functions in the chain and subsequent tests
> > 3. vows gives us great error messages right where the failure occurs
> > 4. we shutdown the remote browser immediately after failure so that the
protocol trace is available 90s sooner
>

> This is great; thanks so much for hacking this stuff together. (I've been
working up from the bottom of the email & am running out of superlatives.)

>
> >
> > Along the way I fixed the returning-user test which was trying to click
a "this is not my computer" button that was not being displayed (recall
that the display of this button requires 60s to pass from first usage in
the current browser profile).
>

> Awesome, thanks for the fix. Just for reference, here is how we hack the
localstorage timer in another test:
https://github.com/mozilla/browserid/blob/kapow/automation-tests/tests/public-terminals.js#L64

>
> >
> > This one test demonstrates most of the new features:
> >
https://github.com/mozilla/browserid/blob/kapow/automation-tests/tests/returning-user.js#L96-113
> >
> > * pass an object with .onError into .chain() and that function will be
invoked when a chained function invokes its (automatically supplied)
callback parameter with a non-falsey first parameter
>

> This is fantastic.

>
> > * if you supply a callback parameter, your .onError handler will not be
automatically invoked
> > * If in your supplied callback you wish to halt processing of remaining
items in the chain, you can manually invoke .haltChain() on the (web
driver) browser object.
>

> Even more fantastic. I think this should get upstreamed for sure, this is
a significant gotcha with the chain() function. I will open an issue
upstream.

>
> >
> > Finally, notice that there's now a third parameter to runner:
> >
https://github.com/mozilla/browserid/blob/kapow/automation-tests/tests/returning-user.js#L130-133
> >
> > a cleanup will be invoked when all the tests are complete, whether or
they failed.
>

> Not in front of a laptop atm, but I think we can do one better:
testSetup.setup() is what creates browsers, so we should be able to have it
do "for browser in this.fixtures.browsers, browser.quit()". And we might
even be able to build that into the runner somehow (wishing for explicit
dependency injection at this point, or just refactoring all these little
includes into one setup object).
>
> Jared

>
> >
> > So my goals were because we're really writing tests where each is
dependent on the success of the former. We want to fail at the first
detected error, we want to cleanup fast, and we want good error messages
that give us an idea of which test block failed and why…
> >
> > curious for review and feedback!
> > lloyd

> > _______________________________________________
> > dev-identity mailing list
> > dev-id...@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-identity

[Lloyd Hilaiel]

On Nov 4, 2012, at 6:11 PM, Jared Hirsch <ja...@jaredhirsch.com> wrote:
> > * if you supply a callback parameter, your .onError handler will not be automatically invoked
> > * If in your supplied callback you wish to halt processing of remaining items in the chain, you can manually invoke .haltChain() on the (web driver) browser object.
>
> Even more fantastic. I think this should get upstreamed for sure, this is a significant gotcha with the chain() function. I will open an issue upstream
>

I expect that there is no way to "abort" the chain cause this feature is built on caolan/async's queue - who itself does not support removing work (and I think that might be because the queue is designed to support parallelism).

More generally, the hacks we've made to chaining might be distasteful upstream because we formalize the every chainable method must be async (last arg callback) and follows typical node error conventions (non-falsey first arg indicates error). But then again, maybe not.

Then I wonder if there's a well written promise/seq/chain/future implementation that we could consider which handles stuff like this and we can adopt to reduce pyramids of doom all over the place, and hopefully reduce the amount of manual error checking code we have while improving readability (yeah, I realize I might be a little late to this party - you've been hinting in this direction for a while now, Jared. I'm slow).

I don't think this is the kinda thing we want to launch a major effort for, or pause progress on tests and features, but seems like someone doing a little research and writing a blog post with a suggestion could be good.

> >
> > Finally, notice that there's now a third parameter to runner:
> > https://github.com/mozilla/browserid/blob/kapow/automation-tests/tests/returning-user.js#L130-133
> >
> > a cleanup will be invoked when all the tests are complete, whether or they failed.
>
> Not in front of a laptop atm, but I think we can do one better: testSetup.setup() is what creates browsers, so we should be able to have it do "for browser in this.fixtures.browsers, browser.quit()". And we might even be able to build that into the runner somehow (wishing for explicit dependency injection at this point, or just refactoring all these little includes into one setup object)
>

This sounds great! the only thing I'm wondering is if we want to keep a subset of the selenium test framework general so we can gradually apply it to our backend tests. I think we've spend a lot more time up front on ergonomics and readability for the selenium stuff, while backend API tests were written under time pressure while learning. Maybe vows_harness.js is the potentially re-usable bit?

lloyd

[Lloyd Hilaiel]

On Nov 4, 2012, at 8:39 PM, Lloyd Hilaiel <ll...@mozilla.com> wrote:

> Then I wonder if there's a well written promise/seq/chain/future implementation that we could consider which handles stuff like this

Or we could just ask "What would substack do?" https://github.com/substack/node-seq

that guy…
lloyd

[Jedediah Parsons]

he truly is the enfant terrible of Node

[Jared Hirsch]

On Sun, Nov 4, 2012 at 7:39 PM, Lloyd Hilaiel <ll...@mozilla.com> wrote:

> On Nov 4, 2012, at 6:11 PM, Jared Hirsch <ja...@jaredhirsch.com> wrote:
>
> > * if you supply a callback parameter, your .onError handler will not be
> automatically invoked
> > * If in your supplied callback you wish to halt processing of remaining
> items in the chain, you can manually invoke .haltChain() on the (web
> driver) browser object.
>
> Even more fantastic. I think this should get upstreamed for sure, this is
> a significant gotcha with the chain() function. I will open an issue
> upstream
>
> I expect that there is no way to "abort" the chain cause this feature is
> built on caolan/async's queue - who itself does not support removing work
> (and I think that might be because the queue is designed to support
> parallelism).
>
> More generally, the hacks we've made to chaining might be distasteful
> upstream because we formalize the every chainable method must be async
> (last arg callback) and follows typical node error conventions (non-falsey
> first arg indicates error). But then again, maybe not.
>

> Then I wonder if there's a well written promise/seq/chain/future

> implementation that we could consider which handles stuff like this and we
> can adopt to reduce pyramids of doom all over the place, and hopefully
> reduce the amount of manual error checking code we have while improving
> readability (yeah, I realize I might be a little late to this party -
> you've been hinting in this direction for a while now, Jared. I'm slow).
>
> I don't think this is the kinda thing we want to launch a major effort
> for, or pause progress on tests and features, but seems like someone doing
> a little research and writing a blog post with a suggestion could be good.
>

You know, a robust Promises implementation (of node libraries, I'm most
familiar with kriskowal's Q) can handle the stuff we're talking about
without needing to put the flow control bits inside the upstream project. Q
has lots of nice features, like built-in helpers to interact with
node-style functions. The thing about it is, it's one of those big
conceptual leaps that is a process of head-wrapping-around. Takes a while.

If you look at the testSetup.setup function, I use Q to create separate
personatestusers via parallel network calls, handle any errors via one
fail() call, and only return after all the promises have resolved. I use
the ncall function to convert a node-style API to a chainable promise-style
API. See the 15 or so lines below here:
https://github.com/mozilla/browserid/blob/kapow/automation-tests/lib/test-setup.js#L131

Here's a thread on flow control from the nodejs mailing list with decent
signal/noise ratio and some excellent links:
https://groups.google.com/forum/?fromgroups=#!topic/nodejs/mWtqFE0spWk

Brian seems to have had his mind warped into Promisey shape at some point
in the past, so I'm sure he would be happy to sing their praises, too :-)

Oh also, speaking of hacking on upstream, there's an unmaintained fork of
wd that uses Q for chaining, so you can check that out:
https://github.com/Stuk/q-wd

>
> > Finally, notice that there's now a third parameter to runner:
> >
> https://github.com/mozilla/browserid/blob/kapow/automation-tests/tests/returning-user.js#L130-133
> >
> > a cleanup will be invoked when all the tests are complete, whether or
> they failed.
>
> Not in front of a laptop atm, but I think we can do one better:
> testSetup.setup() is what creates browsers, so we should be able to have it
> do "for browser in this.fixtures.browsers, browser.quit()". And we might
> even be able to build that into the runner somehow (wishing for explicit
> dependency injection at this point, or just refactoring all these little
> includes into one setup object)
>
> This sounds great! the only thing I'm wondering is if we want to keep a
> subset of the selenium test framework general so we can gradually apply it
> to our backend tests. I think we've spend a lot more time up front on
> ergonomics and readability for the selenium stuff, while backend API tests
> were written under time pressure while learning. Maybe vows_harness.js is
> the potentially re-usable bit?
>

Hmm, interesting question. I'd personally like to see all the test runner
and test setup stuff tightened up and merged into the harness code, I could
definitely see that being reusable for unit tests: setup, runner,
reporting, fixture management. I should get out a copy of Meszaros' xUnit
book and look for patterns we can apply.


>
> lloyd
>
>