Mozmill suite breakage on mozilla-central
Ehsan Akhgari
orange as a result. We didn't have a sheriff on a Saturday, so I took
it upon myself to handle this by filing a bug on the breakage (bug
613802), filing a bug to get the Mozmill suite running on the try server
(bug 613799), hiding the Linux mozmill boxes on mozilla-central, and
letting the patch remain on trunk.
Benjamin has brought this point up before [1], but I don't believe that
it's addressed. We need to have an official policy on how to handle
test breakage for the suites which are run on m-c check-ins. Normally,
we require either a safe and quick fix, or an immediate backout.
However, no policy has been decided on for Mozmill, which makes us
address Mozmill failures on an ad-hoc basis.
I'm posting this in the hopes of starting a discussion to reach such a
policy.
[1]
<http://groups.google.com/group/mozilla.dev.planning/browse_thread/thread/f879f63a4dbfa7b1#>,
among other threads.
Cheers,
Ehsan
Robert Strong
mochitest though I haven't verified this. Along with the mozilla-central
policy regarding mozmill tests we should also have a guideline similar
to what we have for xpchsell / mochitest regarding whether something
should be tested using xpcshell / mochitest / mozmill. I agree with the
current guideline that if something can be tested using xpcshell then
xpcshell should be the preferred method. If it can't, then mochitest
(-chrome, -browser-chrome, etc.) should be used. If neither of those
types of tests can be used then mozmill should be used. With this in
mind, if something is tested with mozmill and can be or already is
tested with xpcshell or mochitest then the test should be converted to
xpcshell or mochitest.
Robert
> _______________________________________________
> dev-tree-management mailing list
> dev-tree-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tree-management
Phil Ringnalda
> Benjamin has brought this point up before [1], but I don't believe that
> it's addressed. We need to have an official policy on how to handle
> test breakage for the suites which are run on m-c check-ins. Normally,
> we require either a safe and quick fix, or an immediate backout.
> However, no policy has been decided on for Mozmill, which makes us
> address Mozmill failures on an ad-hoc basis.
Well, we have an official policy for test breakage, exactly as you say:
fix it or get out of the tree (or say "this test is bogus" and disable
it and file on someone rewriting it).
What we need is an official policy for acceptance of a test suite onto
the tree. Even though it seems like that's too infrequent a thing to
need a policy, it is not: we currently have Mozmill and Jetpack, and
recently had the unacceptable n900 tests on Try, and even though I don't
really know of anyone other than me ever having noticed them, the addons
Talos tests on 1.9.2 that mean that anyone who lands there on Saturday
lands into an unstarred orange tree, every time.
There are a few things I'm sure should be in such a policy:
* If it isn't running on Try, you are *not* going to make it visible on
any other tree with rules.
* If it's still mostly or permanently orange or red, you aren't ready to
make it visible anywhere yet.
* If running it involves something other than checking out
mozilla-central or whatever tree it runs on, building with tests
enabled, and running something like make yourtestsuite, then it better
be something we want as badly as we wanted Talos with the private
pagesuite, and you better get tree owner approval before you even start.
There are some other things I think we might want in such a policy:
* Requiring that result reporting either works with existing Tinderbox
error parsers, or that the suite add an error parser that works with its
reporting. If you look at the current Jetpack failures with &noignore=1,
they are apparently... it's failing in... the problem is.... Mozmill is
barely any better, since both the existing intermittent and the new
permafail are just reported as "testDefaultBookmarks\test1.js |
testVerifyDefaultBookmarks" on some platforms, and the intermittent
(assuming it really is just one bug) is reported differently on Linux
and Mac and Windows.
* Requiring that a suite not hit the network. We screwed up by allowing
in mochitests that do, and we see that it was a screwup every time the
build network hiccups, and several times we've seen that it explained
why a test was intermittently failing even when there wasn't a
widespread build network outage. I don't know whether any of the in-tree
Mozmill tests hit the network, but it made me nervous seeing a Mozmill
bug the other day about switching all tests that hit the network to only
hit mozilla.org sites (which haven't had a particularly good track
record with being things that must be up for tests to pass).
* Requiring that it be possible to disable individual tests. It looks to
me like Mozmill just runs every test in /testing/mozmill/tests/, so
disabling a test (which isn't all that unusual a thing to need to do)
can only be done by deleting it, which seems rather extreme.
* Deciding who actually owns various trees - right now, it's pretty much
anyone with the sheriffpass and a willingness to make their decision
stick, which is rather vague if someone wants to know whether their
suite is acceptable in some way other than "get releng to start running
it, make it visible, and then see whether anyone gets mad and hides it."
Even without having a formal policy (or a tree owner) yet, I think we
should still decide that Mozmill should be hidden on all platforms on
mozilla-central until it's runnable on Try and has documentation for how
to run it locally (and, arguably, until it has a manifest or a list of
files in the makefile so individual tests can be disabled).
Ehsan Akhgari
> What we need is an official policy for acceptance of a test suite onto
> the tree.
I agree. I think the criteria you listed are a good set of requirements
for new test suites; I can't think of anything to add to them right now.
> * Deciding who actually owns various trees - right now, it's pretty much
> anyone with the sheriffpass and a willingness to make their decision
> stick, which is rather vague if someone wants to know whether their
> suite is acceptable in some way other than "get releng to start running
> it, make it visible, and then see whether anyone gets mad and hides it."
Why can't we use our existing module owners in order to make the
decision here? For example, if a test suite wants to test the UI of the
browser (as is the case with the Mozmill suite), they should get the
approval from the Firefox module owner.
> Even without having a formal policy (or a tree owner) yet, I think we
> should still decide that Mozmill should be hidden on all platforms on
> mozilla-central until it's runnable on Try and has documentation for how
> to run it locally (and, arguably, until it has a manifest or a list of
> files in the makefile so individual tests can be disabled).
I agree.
Ehsan
Justin Wood (Callek)
> On 10-11-21 5:49 AM, Phil Ringnalda wrote:
>> What we need is an official policy for acceptance of a test suite onto
>> the tree.
>
> I agree. I think the criteria you listed are a good set of requirements
> for new test suites; I can't think of anything to add to them right now.
>
The only possible addition I could have is an outline (on MDC?) about
"When should you use my shiny new test suite for a new test vs one of
the other [existing] suites."
We have clear lines for xpcshell, reftests, and mochikit tests. I for
one am not sure on when/where other tests are too useful [note: this is
NOT a call for said explanation here]
--
~Justin Wood (Callek)
Justin Dolske
>[...]
Yes, yes, a thousand times yes!
Justin
Mike Beltzner
cheers,
mike
hskupin
> should be tested using xpcshell / mochitest / mozmill. I agree with the
> current guideline that if something can be tested using xpcshell then
> xpcshell should be the preferred method. If it can't, then mochitest
> (-chrome, -browser-chrome, etc.) should be used. If neither of those
This test will be kept in our repository even if we would have a
mochitest for it. But we would probably not distribute it to mozilla-
central. The reason is that this test also checks localized builds and
will get run with our Mozmill Crowd extension soon.
> types of tests can be used then mozmill should be used. With this in
> mind, if something is tested with mozmill and can be or already is
> tested with xpcshell or mochitest then the test should be converted to
> xpcshell or mochitest.
Who would take care of those conversions? I assume the developers, who
are working in this component? Also would such a mochitest completely
work on ui elements? If that wouldn't be the case, the Mozmill test
would still be valid. There are a dozen of good examples i.e. in the
TabbedBrowser component.
hskupin
> widespread build network outage. I don't know whether any of the in-tree
> Mozmill tests hit the network, but it made me nervous seeing a Mozmill
> bug the other day about switching all tests that hit the network to only
> hit mozilla.org sites (which haven't had a particularly good track
> record with being things that must be up for tests to pass).
This bug was not about tests which exists in mozilla-central, but for
tests in our own repository (http://hg.mozilla.org/qa/mozmill-tests/).
We only accept tests for mozilla-central which make use of the local
httpd server.
But it looks like we made a mistake in selecting this particular test,
because it accesses the getting started page. We will discuss this
tomorrow in the Mozmill meeting.
> * Requiring that it be possible to disable individual tests. It looks to
> me like Mozmill just runs every test in /testing/mozmill/tests/, so
> disabling a test (which isn't all that unusual a thing to need to do)
> can only be done by deleting it, which seems rather extreme.
We don't have a manifest at the moment, which makes it indeed harder
to see how tests can be disabled. Right now it's possible by using
"setupModule.__force_skip__" in the test itself, see
http://hg.mozilla.org/qa/mozmill-tests/file/6585aa4bca81/firefox/restartTests/testExtensionInstallGetAddons/test2.js#l69.
I will also put it on the action list for our tomorrows meeting.
> Even without having a formal policy (or a tree owner) yet, I think we
> should still decide that Mozmill should be hidden on all platforms on
> mozilla-central until it's runnable on Try and has documentation for how
> to run it locally (and, arguably, until it has a manifest or a list of
> files in the makefile so individual tests can be disabled).
Seems to be reasonable. But I would like to get an answer from Clint
today, how much work is necessary to get at least the documentation
side done.
Shawn Wilsher
> This test will be kept in our repository even if we would have a
> mochitest for it. But we would probably not distribute it to mozilla-
> central. The reason is that this test also checks localized builds and
> will get run with our Mozmill Crowd extension soon.
builds instead of duplicating testing efforts. We have a limited number
of resources, so we should be shooting to use as little as possible.
> Who would take care of those conversions? I assume the developers, who
> are working in this component? Also would such a mochitest completely
> work on ui elements? If that wouldn't be the case, the Mozmill test
> would still be valid. There are a dozen of good examples i.e. in the
> TabbedBrowser component.
Conceivably, the developers should be writing these tests when their
code lands. If they aren't, people need to start speaking up. Per
mozilla-central's tree rules, tests should be landing with the change in
question. As for exists tests, yes, developers should probably start
working on converting them (maybe this could be a summer of code project
too).
Can you elaborate on situations where mozmill needs to be used instead
of a mochitest? I don't see how anything in tabbed browser code would
not be able to be tested in mochitest.
Cheers,
Shawn
Shawn Wilsher
> mochitest for it. But we would probably not distribute it to mozilla-
> central. The reason is that this test also checks localized builds and
> will get run with our Mozmill Crowd extension soon.
builds instead of duplicating testing efforts. We have a limited number
of resources, so we should be shooting to use as little as possible.
> Who would take care of those conversions? I assume the developers, who
> are working in this component? Also would such a mochitest completely
> work on ui elements? If that wouldn't be the case, the Mozmill test
> would still be valid. There are a dozen of good examples i.e. in the
> TabbedBrowser component.
Conceivably, the developers should be writing these tests when their
code lands. If they aren't, people need to start speaking up. Per
mozilla-central's tree rules, tests should be landing with the change in
question. As for exists tests, yes, developers should probably start
working on converting them (maybe this could be a summer of code project
too).
Can you elaborate on situations where mozmill needs to be used instead
of a mochitest? I don't see how anything in tabbed browser code would
not be able to be tested in mochitest.
Cheers,
Shawn
Phil Ringnalda
> Why can't we use our existing module owners in order to make the
> decision here? For example, if a test suite wants to test the UI of the
> browser (as is the case with the Mozmill suite), they should get the
> approval from the Firefox module owner.
Two problems with that:
* It puts the fox in charge of guarding the henhouse. Mobile made the
right call about not having their permaorange tests visible on try, but
that doesn't mean that when the Firefox For Implanted Computers team
writes a flaky suite in anger because people keep breaking them, they'll
make the right decision about whether their flaky suite should block
mozilla-central pushes.
* It assumes that suites map directly to modules. Who then owns
crashtest (or even reftest, layout or CSS?), or mochitest, or talos?
jsreftest, mochitest-a11y and mochitest-ipcplugins are about the only
existing suites that I could pick a module owner for (maybe
browser-chrome, but it's not really exclusively about Fx's chrome).
Ehsan Akhgari
On example that I know of are tests which involve restarting the browser.
Cheers,
Ehsan
jhammel
As largely a person to blame for landing Mozmill as a buildbot test
suite (bug 516984) and as a developer of the harness, let me try to
address (some of) these issues[*]:
* If it isn't running on Try, you are *not* going to make it visible
on any other tree with rules.
+1; though I would rephrase like "Enabling a test harness [on
e.g. mozilla-central] should require enabling the test harness on
try". While to echo, adding a harness is a rare enough thing, I'm in
favor of a policy that for issues such that enabling a harness
requires a number of steps to happen, turning on the harness on try
being one of them.
* If it's still mostly or permanently orange or red, you aren't ready
to make it visible anywhere yet.
This was not true when the tests were landed. I'm sorry that it is
true now.
* There is no `make mozmill` to run tests within the tree.
See bug 568642 ; I'll work on this this week.
* It is unclear what is expected to do when a mozmill test breaks.
This is largely my fault for not communicating adequately. While I've
been here for most of a year, I am still confused by where the
information should live in our wikis and/or who to email regarding
what developers should know about a new harness and its tests.
If a test breaks, is it the fault of the code or the test? Probably
difficult to tell. I'm not sure how this works currently, but there
are some pointers on this thread
There is a bit of a problem for Mozmill in that the canonical home of
the harness and tests is not mozilla-central. I am not aware of a
strategy for syncing non m-c mozilla code with m-c. Is there one?
For mozmill tests in particular, it is fine if developers fix the test
on m-c but such changes should be communicated to Henrik or one of the
other mozmill developers or test maintainers so that the fix can be
upstreamed to hg.m.o/qa/mozmill-tests
I'd really like a better strategy for syncing non-m-c code to m-c
across the board, but that's a bigger discussion than I plan on taking
on here. The fact is, as much as we'd like to think m-c is a giant
monolith that everything can go in, the fact of the matter is there is
several pieces of code that live elsewhere and are currently synced by
hand when people remember to do it.
* Requiring that a suite not hit the network [...]
AFAIK, none of the mozmill tests in m-c currently hit the network. If
this is not the case, it is a bug.
* Requiring that it be possible to disable individual tests. It looks
to me like Mozmill just runs every test in /testing/mozmill/tests/,
so
disabling a test (which isn't all that unusual a thing to need to
do)
can only be done by deleting it, which seems rather extreme.
Outside of disabling in the individual test case as Henrik pointed out
("setupModule.__force_skip__"), I would advocate getting a
manifest-driven system going forward quickly. There is already a bug
for this: https://bugzilla.mozilla.org/show_bug.cgi?id=585106 . I
will
try to implement this sooner rather than later.
* There is ambiguity wrt what mozmill tests test:
http://groups.google.com/group/mozilla.dev.planning/msg/9338e23f43ea3d01
As pointed out, there are behaviour driven tests, which shouldn't
have to change much, and there are UI tests which may need to change
often. The latter should probably be kept out of m-c. Henrik?
----
[*] Apologies for repitition of what others have already said. It was
done for clarity, emphasis, and to express agreement. Apologies also
for what may be overly verbose quotes which I have been criticized for
before. I try to include enough for context, until someone writes
software to make this transparent, and if I present too much context
it is a failure of my common sense.
jhammel
> error parsers, or that the suite add an error parser that works with its
> reporting. If you look at the current Jetpack failures with &noignore=1,
> they are apparently... it's failing in... the problem is.... Mozmill is
> barely any better, since both the existing intermittent and the new
> permafail are just reported as "testDefaultBookmarks\test1.js |
> testVerifyDefaultBookmarks" on some platforms, and the intermittent
> (assuming it really is just one bug) is reported differently on Linux
> and Mac and Windows.
Can you elaborate here (and/or ticket)? I'm probably just being dense
and/or a n00b but I only have a vague idea what you mean
jhammel
probably go on a wiki page who-knows-where linked to from who-knows-
where:
Things that need to be done for a new harness:
- enable on TBPL
- enable on try
- enable `make testsuite`
- communicate on what to do when a test breaks
- how to fix?
- how to tell if the test is broken or the code is broken?
- how to disable a failing test?
- the tests should NOT touch network (outside of possibly mozilla
URLs); in general, we should have a good way of proxying/serving
pages in a general/easy-to-use way for harnesses/tests that require
TTW testing with local pages (that does not require code copy+paste)
- Update some wiki pages:
https://developer.mozilla.org/en/Mozilla_automated_testing
Robert Kaiser
> There is a bit of a problem for Mozmill in that the canonical home of
> the harness and tests is not mozilla-central. I am not aware of a
> strategy for syncing non m-c mozilla code with m-c. Is there one?
AFAIK, the policy is that m-c cannot be dependent on any code outside of
m-c, and changes in m-c can't require changes anywhere else in terms of
keeping Firefox "green".
> * Requiring that a suite not hit the network [...]
Note that this means it can't even require access to Mozilla pages
(mainly because we don't want our builds to break when we change
something on web servers/pages). If you need any content served, do it
locally via httpd.js.
Robert Kaiser
--
Note that any statements of mine - no matter how passionate - are never
meant to be offensive but very often as food for thought or possible
arguments that we as a community needs answers to. And most of the time,
I even appreciate irony and fun! :)
Phil Ringnalda
I just hid it for all platforms on mozilla-central, since as you
discovered, not having it running on try makes it _really_ difficult to
deal with :)
smaug
> It seems that this failure is for a test that could be tested via
> mochitest though I haven't verified this. Along with the mozilla-central
> policy regarding mozmill tests we should also have a guideline similar
> to what we have for xpchsell / mochitest regarding whether something
> should be tested using xpcshell / mochitest / mozmill. I agree with the
> current guideline that if something can be tested using xpcshell then
> xpcshell should be the preferred method.
If the test can be made cross-browser, it should be mochitest or
something similar, not xpcshell.
Robert Strong
> On 11/21/2010 08:44 AM, Robert Strong wrote:
>> It seems that this failure is for a test that could be tested via
>> mochitest though I haven't verified this. Along with the mozilla-central
>> policy regarding mozmill tests we should also have a guideline similar
>> to what we have for xpchsell / mochitest regarding whether something
>> should be tested using xpcshell / mochitest / mozmill. I agree with the
>> current guideline that if something can be tested using xpcshell then
>> xpcshell should be the preferred method.
> Since when is there this kind of guideline?
> If the test can be made cross-browser, it should be mochitest or
> something similar, not xpcshell.
xpcshell the mochitest page states it should be written in xpcshell.
https://developer.mozilla.org/en/Mochitest
Specifically from the page referenced above:
"Try to avoid Mochitest
Yes, really. For many things Mochitest is overkill. In general you
should always try to use one of the lighter-weight testing frameworks.
For example, if you only want to test a single XPCOM component then you
should use xpcshell. On the other hand there are some things that
Mochitest cannot do, or isn't designed to do. For example, for visual
output tests you should try to use the reftest framework. For more
information on the different types of automated testing frameworks see
Mozilla automated testing."
Also, if something can be tested by xpcshell the vast majority of the
time all apps can run that test whether it is a browser or not. Also
note that I stated that if it can't be tested with xpcshell then
mochitest (-chrome, -browser-chrome, etc.) should be used.
Robert
L. David Baron
> On 11/25/2010 12:05 PM, smaug wrote:
> >>It seems that this failure is for a test that could be tested via
> >>mochitest though I haven't verified this. Along with the mozilla-central
> >>policy regarding mozmill tests we should also have a guideline similar
> >>to what we have for xpchsell / mochitest regarding whether something
> >>should be tested using xpcshell / mochitest / mozmill. I agree with the
> >>current guideline that if something can be tested using xpcshell then
> >>xpcshell should be the preferred method.
> >Since when is there this kind of guideline?
> >If the test can be made cross-browser, it should be mochitest or
> >something similar, not xpcshell.
> xpcshell the mochitest page states it should be written in xpcshell.
> https://developer.mozilla.org/en/Mochitest
>
> Specifically from the page referenced above:
> "Try to avoid Mochitest
>
> Yes, really. For many things Mochitest is overkill. In general you
> should always try to use one of the lighter-weight testing
> frameworks. For example, if you only want to test a single XPCOM
> component then you should use xpcshell. On the other hand there are
> some things that Mochitest cannot do, or isn't designed to do. For
> example, for visual output tests you should try to use the reftest
> framework. For more information on the different types of automated
> testing frameworks see Mozilla automated testing."
>
> Also, if something can be tested by xpcshell the vast majority of
> the time all apps can run that test whether it is a browser or not.
> Also note that I stated that if it can't be tested with xpcshell
This has never been policy for our code that implements features for
Web content, nor should it be. I guess I should just edit that wiki
page at some point.
It's important that our tests test the behavior that we need to
preserve; this gives us the maximum flexibility to refactor quickly
without breaking tests, and it means our tests are the most usable
for contributions to standards bodies, etc.
Our experience with unit tests of internal Gecko interfaces in the
past has been that they bitrot and just get disabled. Having good
tests is most important when doing redesign of existing code (e.g.,
for performance, memory use, etc.); unit tests of interfaces are
*useless* when doing that since the interfaces are being replaced.
-David
--
L. David Baron http://dbaron.org/
Mozilla Corporation http://www.mozilla.com/
Robert Strong
> On Thursday 2010-11-25 15:01 -0800, Robert Strong wrote:
>> On 11/25/2010 12:05 PM, smaug wrote:
>>>> It seems that this failure is for a test that could be tested via
>>>> mochitest though I haven't verified this. Along with the mozilla-central
>>>> policy regarding mozmill tests we should also have a guideline similar
>>>> to what we have for xpchsell / mochitest regarding whether something
>>>> should be tested using xpcshell / mochitest / mozmill. I agree with the
>>>> current guideline that if something can be tested using xpcshell then
>>>> xpcshell should be the preferred method.
>>> Since when is there this kind of guideline?
>>> If the test can be made cross-browser, it should be mochitest or
>>> something similar, not xpcshell.
>> xpcshell the mochitest page states it should be written in xpcshell.
>> https://developer.mozilla.org/en/Mochitest
>>
>> Specifically from the page referenced above:
>> "Try to avoid Mochitest
>>
>> Yes, really. For many things Mochitest is overkill. In general you
>> should always try to use one of the lighter-weight testing
>> frameworks. For example, if you only want to test a single XPCOM
>> component then you should use xpcshell. On the other hand there are
>> some things that Mochitest cannot do, or isn't designed to do. For
>> example, for visual output tests you should try to use the reftest
>> framework. For more information on the different types of automated
>> testing frameworks see Mozilla automated testing."
>>
>> Also, if something can be tested by xpcshell the vast majority of
>> the time all apps can run that test whether it is a browser or not.
>> Also note that I stated that if it can't be tested with xpcshell
> Web content, nor should it be. I guess I should just edit that wiki
> page at some point.
think that xpcshell tests are less fragile and cause less oranges
because of this. Making it clear that any test harness that doesn't test
the functionality that needs to be tested might be a good thing but I
would hope that is obvious.
> It's important that our tests test the behavior that we need to
> preserve; this gives us the maximum flexibility to refactor quickly
> without breaking tests, and it means our tests are the most usable
> for contributions to standards bodies, etc.
Agreed wholeheartedly which is why I stated that if the functionality
can be tested in xpcshell then it should per the guideline and if not
then a different harness should be used.
> Our experience with unit tests of internal Gecko interfaces in the
> past has been that they bitrot and just get disabled. Having good
> tests is most important when doing redesign of existing code (e.g.,
> for performance, memory use, etc.); unit tests of interfaces are
> *useless* when doing that since the interfaces are being replaced.
That is my experience of any test that isn't maintained and it is also
my experience that "good tests" have more to do with how the test is
written and having the knowledge to choose the best harness for what is
being tested out of the harnesses available. We have lots of xpcshell
tests that were simple to create / maintain and test very specific
functionality that have caught regressions. If a test isn't of value by
all means disable / remove it but if it does provide value why not
maintain the simpler test?
One example of this is I have several app update xpcshell tests that use
XHR, there were changes to XHR that busted XHR errors for offline mode,
tests went orange, and the regression was noticed / fixed immediately.
Since there was no equivalent test for XHR living next to XHR I created
/ landed an xpcshell test specifically for XHR. This is not useless as I
see it and is a good test.
Robert
Robert Kaiser
> On 11/21/2010 08:44 AM, Robert Strong wrote:
>> It seems that this failure is for a test that could be tested via
>> mochitest though I haven't verified this. Along with the mozilla-central
>> policy regarding mozmill tests we should also have a guideline similar
>> to what we have for xpchsell / mochitest regarding whether something
>> should be tested using xpcshell / mochitest / mozmill. I agree with the
>> current guideline that if something can be tested using xpcshell then
>> xpcshell should be the preferred method.
> Since when is there this kind of guideline?
> If the test can be made cross-browser, it should be mochitest or
> something similar, not xpcshell.
I disagree, even if only for the fact that xpcshell-tests can be run on
Thunderbird, while mochitests (of any kind) can't.
Gavin Sharp
>> Since when is there this kind of guideline?
>> If the test can be made cross-browser, it should be mochitest or
>> something similar, not xpcshell.
>
> I disagree, even if only for the fact that xpcshell-tests can be run on
> Thunderbird, while mochitests (of any kind) can't.
"xpcshell" and "on Thunderbird" aren't really compatible concepts...
"xpcshell in a thunderbird build's objdir", sure, but you're not
getting much Thunderbird-specific code coverage there, if any.
Gavin
Robert Kaiser
Thanks for the correction on the nitty-gritty details, I of course meant
"on a Thunderbird build" or "a testing setup made from a Thunderbird
tree's dist/bin" (I'm explicitely not saying "objdir" to not close out
packaged tests as done on buildbot setups).
It's often quite helpful to run platform/backend tests against your
favorite non-Firefox product to find out if the platform actually works
as it should, even if the tests are not specific to your product.
Testing the should-be-generic Mozilla platform included in your product
helps preventing unforeseen problems.
(That said, Thunderbird also extensively tests it's mail/news backend
via xpcshell-tests, but that runs out of the scope of what this thread
was meant to be about. They do all their automated UI testing via
mozmill-tests, by the way, as they can't run any mochitests - they just
have yet another harness than Firefox for mozmill.)
Clint Talbert
ought to have before landing on m-c. I think though that for the "must
be runnable by developers" argument, I think we should start moving away
from make target per testsuite. Even though they exist, even though
they are logical, people still don't remember them. If we have only one
entry point for tests then perhaps that would be a better API.
Something like:
$> make tests [testtype] [optional list of test arguments in key-value
pairs, defaults to standard run-on-buildbot arguments]
and:
$> make tests
Errors and echoes the list of testtype's that are valid.
Unifying all this ought to help make the code underlying all this much
more maintainable.
Anyhow, that's a very good set of rules to make such a policy and I
think we should keep driving forward with that effort (perhaps in a new
thread), but before I derail the original thread too much, let's cover
where we currently are with the Mozmill breakage.
Upon getting back after my vacation, I found a bunch of efforts already
underway, and many of those are coming to completion soon.
* The test breakage - I landed Henrik's patch on bug 613802 this morning
to fix the test itself, and the tests have gone green on tinderbox
(still hidden on tbpl)
* Try server - Mozmill was available on try server, but the schedulers
weren't running it (John Ford, correct me if I'm mis-representing).
This is now fixed.
* Make target for Mozmill - work is underway in bug 568642 to create
this make target. We should have it by early next week.
* Being able to disable tests easily in Mozmill - leveraging some
experimental manifest work we are doing for xpcshell tests, we have
accelerated our existing plans to use manifests for Mozmill tests. We
already have mozmill running with manifests which will make it trivially
easy to disable tests going forward. We are now investigating
accelerating our 1.5.2 release vehicle to see if we can land this on m-c
before year's end.
= Mozmill Test Responsibilities =
This gets much easier now that Mozmill is on try. We discussed the
following tree rules with the QA Automated Test team, and they are in
agreement with them.
There are two ways you can break these tests:
* Deliberately -- you're checking in a feature that removes, renames, or
changes a specific element of UI that the test is depending on. In this
case, we'd expect the developer to correct the test like they would for
any other test. If there are any issues, CC Henrik or myself on the bug
and we can help with it.
* Non-deliberately -- you checked something in, the test went haywire
and you don't know why. If that happens, disable the test, CC Henrik to
the bug, and the QA Automation team will repair/diagnose the breakage.
Of course, our manifests are going to be essential for this
functionality so that you can more easily disable the tests.
Now that Mozmill on try is working properly, I expect far fewer
surprises. If you hit a snag with a Mozmill test on try, let Henrik
know and his team (and mine if necessary) are more than happy to help
you figure out what's going on.
As you get more accustomed to the new framework, I think you'll find
it's just as easy to work with as any of our other frameworks. We want
to make this transition as smooth as possible, let us know if you hit
any other unexpected issues.
Thanks,
Clint
Benjamin Smedberg
> runnable by developers" argument, I think we should start moving away from
> make target per testsuite. Even though they exist, even though they are
> logical, people still don't remember them. If we have only one entry point
> for tests then perhaps that would be a better API. Something like:
> $> make tests [testtype] [optional list of test arguments in key-value
> pairs, defaults to standard run-on-buildbot arguments]
We can rephrase this as "we should also have a single entry point which runs
all test suites". Since having separate entry points is obviously helpful
for developers who have particular things failing and need to run just those
tests.
--BDS
Henrik Skupin
> On 11/22/2010 6:37 AM, hskupin wrote:
>
>> This test will be kept in our repository even if we would have a
>> mochitest for it. But we would probably not distribute it to mozilla-
>> central. The reason is that this test also checks localized builds and
>> will get run with our Mozmill Crowd extension soon.
>>
> It seems like we should fix mochitests that don't work in localized builds
> instead of duplicating testing efforts. We have a limited number of
> resources, so we should be shooting to use as little as possible
Do we have a number of how many Mochitests actually could be executed in
localized builds? Tests I have checked so far always using hard-coded
keyboard shortcuts.
> Who would take care of those conversions? I assume the developers, who
>> are working in this component? Also would such a mochitest completely
>> work on ui elements? If that wouldn't be the case, the Mozmill test
>> would still be valid. There are a dozen of good examples i.e. in the
>> TabbedBrowser component.
>>
> Conceivably, the developers should be writing these tests when their code
> lands. If they aren't, people need to start speaking up. Per
> mozilla-central's tree rules, tests should be landing with the change in
> question. As for exists tests, yes, developers should probably start
> working on converting them (maybe this could be a summer of code project
> too).
>
Speaking as a non-developer, it looks like that this restriction isn't such
a hard requirement anymore. There are a lot of patches I could reference,
which do not include tests. And I only watch some components. A complete
different scenario is the add-ons manager component, where we always have an
automated or a manual testcase.
> Can you elaborate on situations where mozmill needs to be used instead of a
> mochitest? I don't see how anything in tabbed browser code would not be
> able to be tested in mochitest.
>
As far as I can see most of those tests are more unit tests as actually ui
tests. One of the examples is bug 609700. This test doesn't make use of any
keypress or mouse click, but checks the duplicateTabIn function by using the
global gBrowser object:
http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser.js#8037
--
Henrik Skupin
QA Engineer
Mozilla Corporation
Clint Talbert
the other entry points are discoverable. I think that's where the
current mechanism falls down, the entry points are not discoverable.
- Clint