mozmill tests appearing on tinderbox
Benjamin Smedberg
https://developer.mozilla.org/en/Mozilla_automated_testing mentions
instructions for running these tests at
https://bugzilla.mozilla.org/show_bug.cgi?id=610941#c9
What are the tree rules related to these tests? As noted previously in the
discussion of jetpack SDK testing, I don't think it is good for tree health
to have a required testsuite which is not part of mozilla-central. It makes
it much more difficult to do regression testing, and changes to the
testsuite which may affect test results are not visible on Tinderbox or TBPL.
If we intend for mozmill tests to become part of the normal testsuite, the
test harness and code should be imported into mozilla-central. Otherwise, we
should move reporting of these tests somewhere off the main mozilla-central
tree.
Additionally, wherever they are reported, we should list the revision of
mozmill that is being tested, in addition to the revision of
mozilla-central. This will at least make it possible to figure out whether
test failures are caused by mozmill or mozilla-central changesets.
--BDS
Clint Talbert
> Sometime recently, mozmill test results have started appearing on
> tinderbox. https://developer.mozilla.org/en/Mozilla_automated_testing
> mentions instructions for running these tests at
> https://bugzilla.mozilla.org/show_bug.cgi?id=610941#c9
>
> What are the tree rules related to these tests?
Tree rules are something we need to figure out. The Mozmill tests are
generally of two different types. One set are UI behavior tests, these
are likely to break each time we rev our UI. I would argue those should
probably not be run via buildbot automation as when those break, it is
not a developer that needs to engage the breakage but a QA test
developer to update the test.
The second type of mozmill test are more behavioral/functional based --
these are the tests that are largely automations of the existing manual
Litmus test cases. The tests that "graduate" from the QA repo of
hg.m.o/QA/mozmill-tests into m-c should only be those tests that have
proven themselves stable and therefore when those break, they should be
treated as any other failing mochitest or reftest.
Currently, in order to simply "get something working" in buildbot
automation we went live with a very tiny subset of mozmill tests, all of
which are the "behavioral/functional" variety.
As noted previously in
> the discussion of jetpack SDK testing, I don't think it is good for tree
> health to have a required testsuite which is not part of
> mozilla-central. It makes it much more difficult to do regression
> testing, and changes to the testsuite which may affect test results are
> not visible on Tinderbox or TBPL.
The Mozmill tests that are being run do live in mozilla central.
There is a much larger set of mozmill tests that are not being run by
the buildbot automation which lives outside of mozilla-central, and that
is likely what makes this confusing. The mozmill harness and tests that
are running in mozilla-central can be found here:
http://mxr.mozilla.org/mozilla-central/source/testing/mozmill/
>
> If we intend for mozmill tests to become part of the normal testsuite,
> the test harness and code should be imported into mozilla-central.
> Otherwise, we should move reporting of these tests somewhere off the
> main mozilla-central tree.
This was done. It was actually fairly impossible to have these be
integrated into buildbot without them being in mozilla-central.
> Additionally, wherever they are reported, we should list the revision of
> mozmill that is being tested, in addition to the revision of
> mozilla-central. This will at least make it possible to figure out
> whether test failures are caused by mozmill or mozilla-central changesets.
>
That's true. I'll file a bug to make this change in the mozmill
reporting. The version on tinderbox is always the "latest release" of
the mozmill harness. So the current version is 1.5.1.
We plan to only land full versions of the harness into mozilla-central
in order to make it easy to manage. In the case that we have to do a
bugfix to the version in m-c, that would necessitate an immediate
release to pypi. For example, if we had to do a bugfix right now on the
m-c code, I'd do that and then release a mozmill-1.5.1.1 immediately to
pypi. This way the versions of the software stay in sync.
Hope that clears it up. Let me know if you have other questions. I
should have posted about this sooner, apologies for that.
Clint
gavin
> The Mozmill tests that are being run do live in mozilla central.
> There is a much larger set of mozmill tests that are not being run by
> the buildbot automation which lives outside of mozilla-central, and that
> is likely what makes this confusing. The mozmill harness and tests that
> are running in mozilla-central can be found here:http://mxr.mozilla.org/mozilla-central/source/testing/mozmill/
How do mozilla-central developers run those tests? Mozmill test
failures are currently turning mozilla-central orange, and no one
knows how to run the tests to reproduce the failure.
http://mxr.mozilla.org/mozilla-central/source/testing/mozmill/README.txt
points to https://developer.mozilla.org/en/Mozmill , which in turns
talks about needing an addon or some separate "command line client".
I think being able to run the tests from a mozilla-central checkout
using something like |make mozmill| is a baseline requirement for
mozmill reporting to mozilla-central. Until this gets cleared up I
think we need to hide them on the Firefox tree.
Gavin
Ehsan Akhgari
> On Nov 18, 12:01 pm, Clint Talbert <ctalb...@mozilla.com> wrote:
>
> > The Mozmill tests that are being run do live in mozilla central.
> > There is a much larger set of mozmill tests that are not being run by
> > the buildbot automation which lives outside of mozilla-central, and that
> > is likely what makes this confusing. The mozmill harness and tests that
> > are running in mozilla-central can be found here:http://mxr.mozilla.org/mozilla-central/source/testing/mozmill/
>
> How do mozilla-central developers run those tests? Mozmill test
> failures are currently turning mozilla-central orange, and no one
> knows how to run the tests to reproduce the failure.http://mxr.mozilla.org/mozilla-central/source/testing/mozmill/README.txt
> talks about needing an addon or some separate "command line client".
>
> I think being able to run the tests from a mozilla-central checkout
> using something like |make mozmill| is a baseline requirement for
> mozmill reporting to mozilla-central. Until this gets cleared up I
> think we need to hide them on the Firefox tree.
I've filed bug 613804 for the |make mozmill| syntax.
Also, because of bug 613802, I had to hide the Linux mozmill boxes on
mozilla-central. I've posted a thread [1] to dev.tree-management in
order for us to decide on how we want to handle the failures in the
mozmill suite. Please reply in that thread if you have insights on
this.
Thanks!
Ehsan
[1] <http://groups.google.com/group/mozilla.dev.tree-management/
browse_thread/thread/f5aeec1990ac36b0#>