test automation crew update

[Jared Hirsch]

Hey all,

Couple of quick updates on the goals/timeline/strategy for Identity's test
automation crew. All this info is available online, so I'm not going to
include all the details, but do want to call out the big points.

What's a crew? Identity has gotten big. To keep things lightweight, we're
self-organizing into small crews of roughly 2-4 engineers, with one lead, a
crisp definition of success, and a fairly short lifespan (by default, six
weeks). See the list of crews here: https://id.etherpad.mozilla.org/crews.
In keeping with that definition, the test automation crew exists to
help Identity by automating the tedious manual browser testing jrgm has to
do before a release. We're building selenium tests and related
infrastructure. We're calling it quits at the end of October, aiming to hit
the 80/20 sweet spot by then, in terms of test coverage and infrastructure.
See our more detailed quantitative goals, a specific list of tests we aim
to complete, and related infrastructure todos here:
https://id.etherpad.mozilla.org/test-automation-spec.

After initially working with webqa's python infrastructure, we've opted to
move to using javascript webdriver bindings instead. This was a big move,
and a lot of factors were considered in making the decision. To summarize
our perspective semi-briefly: using JS tests on a JS project makes it
easier for devs on the team to write and maintain tests (no context
switch); webqa taking over maintenance of bidpom meant marketplace et al.
wouldn't be marooned if we quit using python; and we are taking a much less
abstract approach to writing tests, opting for ease of moving quickly, with
some duplication, over the additional structure (and slower development
speed) of writing page objects (POMs). See a longer list of factors in the
minutes from our crew standup today:
https://id.etherpad.mozilla.org/automated-test-crew-standup-20121010.

The new JS-based tests haven't landed in our main dev branch yet; they are
in the 'kapow' branch of mozilla/browserid. We aim to land them this week,
but it's likely this will slip to early next week.

Going back to the question of strategy, Francois put together a great
high-level summary of jrgm's approach to QA. It's worth a look if you're
wondering about how automation fits into the big big picture:
https://id.etherpad.mozilla.org/quality-strategy.

If you'd like to get involved: ping me, jrgm, or stomlinson; have a look at
the "slides" from a test talk I gave last week (
https://gist.github.com/0874785e0479451fa801); grab the kapow branch of
mozilla/browserid and peruse the automation-tests directory.

Please ping this thread (as opposed to commenting in any of the linked
etherpads) if you have questions, concerns, comments, or hilarious ascii
art to share.

Cheers,

Jared

[Sean McArthur]

When it merges into master, will it be possible to add the full suite to
the "npm test" script, perhaps with another WHAT_TESTS option, to spare
Travis.

This way we're all running the tests automatically when they merge in.

[Jared Hirsch]

Thanks for the suggestion!

I've added this to our infrastructure todos on the
test-automation-spec<https://id.etherpad.mozilla.org/test-automation-spec>etherpad.