Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
test automation crew update: almost there!
11 views
Skip to first unread message
Jared Hirsch
Nov 7, 2012, 10:05:13 PM11/7/12
to dev-id...@lists.mozilla.org
As mentioned on this list last week[1], the test automation crew has
gone into extra innings to finish up a few crucial tasks. Here's an
update:
TL;DR: super important stuff is nearly done; important stuff is still
to-do. Probably one more week.
Travis integration: done
- Pull requests from core contributors will automatically be
selenium tested on submission. Submissions from community contributors
won't be automatically selenium tested, simply to conserve saucelabs
resources.
Jenkins integration: ongoing
- Jenkins jobs for dev testing have been updated to use the node
infrastructure; stage and prod will be updated when node tests reach
those environments.
- If you're interested in digging further, our Jenkins instance is
public[2]; the dev tests are labeled identity.browserid.dev.*.
Stabilization: looks great so far, except for newly added platforms
- Our metric for stability is 99% for health check tests, which are
run frequently as a monitoring tool, and 90% for all other tests.
- I ran all platforms repeatedly this afternoon, with the following results:
- 90%+ success rate: win XP IE8, vista chrome, vista firefox 16,
vista IE9, mac firefox 14, linux firefox 16, linux opera 12
- Those platforms with <90% success rate were added to our list very
recently & haven't gotten debugged yet: win 8 IE10, mac safari 5.
- Mobile platforms are also new and need work.
- Still waiting on overnight cron which will gather more data on the
health check tests.
Merge into mainline dev: poised for completion
- Conversation on this topic is in browserid pull request 2670 [3]
Stuff not yet addressed includes documentation (code, wiki, bloggy)
and code cleanup. It's looking like we'll be wrapped after another
week, I'm guessing.
See the test automation crew's etherpad[4] for up-to-date status on
individual tasks.
Comments/questions welcome.
Jared
[1] https://groups.google.com/forum/?fromgroups=#!topic/mozilla.dev.identity/u-nmeS-2eoA
[2] https://ci.mozilla.org/view/Persona/
[3] https://github.com/mozilla/browserid/pull/2670
[4] https://id.etherpad.mozilla.org/test-automation-spec
gone into extra innings to finish up a few crucial tasks. Here's an
update:
TL;DR: super important stuff is nearly done; important stuff is still
to-do. Probably one more week.
Travis integration: done
- Pull requests from core contributors will automatically be
selenium tested on submission. Submissions from community contributors
won't be automatically selenium tested, simply to conserve saucelabs
resources.
Jenkins integration: ongoing
- Jenkins jobs for dev testing have been updated to use the node
infrastructure; stage and prod will be updated when node tests reach
those environments.
- If you're interested in digging further, our Jenkins instance is
public[2]; the dev tests are labeled identity.browserid.dev.*.
Stabilization: looks great so far, except for newly added platforms
- Our metric for stability is 99% for health check tests, which are
run frequently as a monitoring tool, and 90% for all other tests.
- I ran all platforms repeatedly this afternoon, with the following results:
- 90%+ success rate: win XP IE8, vista chrome, vista firefox 16,
vista IE9, mac firefox 14, linux firefox 16, linux opera 12
- Those platforms with <90% success rate were added to our list very
recently & haven't gotten debugged yet: win 8 IE10, mac safari 5.
- Mobile platforms are also new and need work.
- Still waiting on overnight cron which will gather more data on the
health check tests.
Merge into mainline dev: poised for completion
- Conversation on this topic is in browserid pull request 2670 [3]
Stuff not yet addressed includes documentation (code, wiki, bloggy)
and code cleanup. It's looking like we'll be wrapped after another
week, I'm guessing.
See the test automation crew's etherpad[4] for up-to-date status on
individual tasks.
Comments/questions welcome.
Jared
[1] https://groups.google.com/forum/?fromgroups=#!topic/mozilla.dev.identity/u-nmeS-2eoA
[2] https://ci.mozilla.org/view/Persona/
[3] https://github.com/mozilla/browserid/pull/2670
[4] https://id.etherpad.mozilla.org/test-automation-spec
0 new messages