Test Repository For Mozmill Tests

2 views
Skip to first unread message

Clint Talbert

unread,
Dec 30, 2008, 9:28:06 PM12/30/08
to mozmi...@googlegroups.com, q...@mozilla.com, ctal...@mozilla.com, Mark Banner, Adrian Kalla, Martin Schröder, Gary Kwong, Ted Mielczarek
We need to figure out where to put the Mozmill tests. There are quite a
few things to consider here, so first let me outline some of the goals
and benefits that the tool itself brings us.

== High Level Stuff ==

* Mozmill is a tool to enable anyone to write testcases against any
mozilla platform application.
* Mozmill strives to be a tool that even relatively non-technical people
can use to automate their manual test cases and can write testcases against.
* Mozmill is a giant lever in the eternal quest to build our community
by empowering a group of people who have traditionally been dropped into
the hole between manual testing and technical test automation. It also
has the ability to bring the people who do manual testing of non-Firefox
applications closer into the circle of the overall Project.
* Mozmill will help us to find new ways of automating and testing and
tracking test runs - perhaps by building out recording tools that people
can run once to record some unreproducible behavior, or it could be used
for L10N testing. So, whatever we do with the test repo, it has to be
flexible.

== Repo Goals ==

Okay, that's some high level stuff. Here are some things to consider
for the repository itself.
* It should be simple.
* It should be internet browsable and referencable
* All the projects that use mozmill should upload their tests there so
that we can all learn from each other's tests and test methodologies.
If one group starts writing several utility scripts, then we could
ostensibly share those between all the projects, and having a central
repo will aid in that effort.
* The tests need to be versioned, to track product versions as many of
them will be UI dependent, which might change from version to version.
* The repo should (if possible) further the goal of bringing people into
the Mozilla community while balancing against the need to keep the
barrier to entry as low as possible.

== My Proposal ==
We have the beginnings of a QA repository location in Mozilla SVN here:
http://svn.mozilla.org/qa/

The Mozilla SVN repo meets all the above goals pretty adequately: it's
simple, we can structure it so that it is versionable w.r.t. product
versions, and it's easily installed on all systems (comes by default on
mac os x 10.5, linux, and is included in the mozilla-build package on
windows). The one problem Mozilla SVN has is that it has a high bar to
entry for commit access, like all the mozilla repo's. I have some ideas
on how to mitigate this:
* A set of leaders is chosen from each project using Mozmill to shepherd
the checkin's from that project into the SVN repo. That person will get
SVN commit access first. This access will be across the entire test case
repo so that they can check in utility scripts, or tests into other
parts of the Mozmill test repo (for example, perhaps they have a unit
test for the elementlib in mozmill itself).
* That leader will also identify new members from their communities that
they wish to extend commit access to, and will help that person attain
it (filing proper bugs for it, harassing people to get them cleared etc,
whatever it takes - it's actually a fairly simple process, once the
process is started).
* The leaders will have direct ability to manipulate their project's
test directory structure so that they can easily create areas for tests
that are version dependent (according to their product version).
* These leaders will ensure (especially as new contributors are
identified and added) that there is no single person who can become a
bottleneck w.r.t. creating areas for tests against new versions of
products and/or checking in tests themselves.

Thoughts, comments, feedback, and new proposals are encouraged. I look
forward to hearing yours.

Clint

Gary

unread,
Dec 31, 2008, 12:08:18 AM12/31/08
to Al Billings, Clint Talbert, mozmi...@googlegroups.com, q...@mozilla.com, ctal...@mozilla.com, Mark Banner, Adrian Kalla, Martin Schröder, Ted Mielczarek
Are we sure that the class of users who we want to ramp up to mozmill (not super technical) will easily be able to master svn?

I think there's a difference between writing/creating the mozmill tests (which Clint emphasizes it's going to be non-technical) and checking them in to svn itself. This is similar to the structure of folks writing patches and the ones actually having commit access to check them in, the latter is a subset of the former.

That said, there should also be some form of two-way linkage between litmus tests, the relevant bug number for which the mozmill test is attached to, once it lands on the repository and working continuously by jsbridge to activate the in-testsuite flag. We can then know which litmus test is already covered in the automated test infrastructure, which can be seen both via the litmus html page and searching for bugs with a in-testsuite+ flag in bugzilla as well as the web mirroring website Clint and Al have been talking about.

-Gary

Mark Banner

unread,
Dec 31, 2008, 8:22:35 AM12/31/08
to Al Billings, mozmi...@googlegroups.com, Adrian Kalla, Martin Schröder, Gary Kwong, Ted Mielczarek
[resent so this gets on the mozmill-dev group as well]

On 12/31/2008 03:37, Al Billings wrote:
> I think that this is an excellent suggestion.


>
> Are we sure that the class of users who we want to ramp up to mozmill
> (not super technical) will easily be able to master svn?
>

> I would like to see a simple mirroring/shadowing of the repository
> files (at least the most current version) to a website for easy access.
Is svn really the good choice here? I would have thought a Mercurial
repository would be much better - it is where mozilla source control is
typically going anyway and it will be one less repository system for
devs (who will want to write mozmill tests).

The main benefit of a Mercurial repository is that if we do decide to
run tests against the current sources continuously, I think it would be
much easier for the build team to integrate pulling from an hg
repository than it would a svn one. Although it has been mentioned that
running Mozmill tests against sources continuously isn't necessarily the
best thing, as we have no UI level tests (apart from leak & bloat)
available for Thunderbird, then I'm considering it as at least a short
to medium-term option and seeing how it goes.

Standard8

Mikeal Rogers

unread,
Dec 31, 2008, 2:52:37 PM12/31/08
to mozmi...@googlegroups.com, q...@mozilla.com, ctal...@mozilla.com, Mark Banner, Adrian Kalla, Martin Schröder, Gary Kwong, Ted Mielczarek
This is a good kick off to this discussion. I have a bunch of reasons
why I don't think some of these won't work, and I don't have a good
solution either, so get ready to hate me.

First I'd like to say that this thread should really only pertain to
"functional" test repositories. unittests written using mozmill will
most likely follow patterns similar to the mochitest unittests. Most
of these problems are non-issues for unittests because the intended
audience is a contributor to the product already.

One of the big things that isn't covered is that the functional test
repository needs to be able to track the products repository (Firefox,
Thunderbird, etc). When the product branches for a release the tests
need to branch so that after the release we can run the older set of
tests against bug fix releases and run the latest and greatest tests
against the latest and greatest nightlies.

You've detailed a lot of the issues with sticking the tests in the
same repository as the product, but we do need to make sure that we
don't lose something very very important that came along with sticking
them in the product repository.

I think the only likely solution here is going to be in mercurial.
Hopefully we can find a technical solution that allows our hg test
repos to track the product repos.

-Mikeal

Mark Banner

unread,
Jan 12, 2009, 9:26:59 AM1/12/09
to Henrik Skupin, Clint Talbert, mozmi...@googlegroups.com, q...@mozilla.com, ctal...@mozilla.com, Martin Schröder, Gary Kwong, Ted Mielczarek
[resending from correct address so it gets on mozmill-dev]

On 01/12/2009 12:21, Henrik Skupin wrote:
>> == Repo Goals == * It should be simple.
>>
>
> We probably have to find a way between simplicity and running tests from
> within the build system. When bug 457265 (Integrate Mozmill into build
> system) is fixed, the build system will be able to automatically run
> MozMill tests. All tests should be accessible therefor. I would tend to
> second the simplicity here. With a good tutorial everybody should be
> able to checkout existing tests and run them locally.
>
I think you have mistaken this repository for the same thing I did. I
thought:

"This is a repository for locating tests against a product that may
eventually be run repeatedly against the source like we do for unit tests"

from the discussions so far I think this "test repository" is:

"This is a repository for locating 'random' tests written by users for a
product, that may just be run occasionally, and probably best on litmus
tests and may never see automated running"

I think we do need to have a clear description of what the repository is
for, as I think there is ambiguity here.

Standard8

Gary Kwong

unread,
Jan 12, 2009, 9:39:28 AM1/12/09
to Henrik Skupin, Clint Talbert, mozmi...@googlegroups.com, Mark Banner, Ted Mielczarek, q...@mozilla.com, "Martin Schröde"@dm-mailman01.mozilla.org, Adrian Kalla, r
Hey all,
much, much easier.  At the moment, Gary's spreadsheet that correlates 
litmus cases to Mozmill testcases for Thunderbird smoketests are 
probably the way to go here.
    
Oh, do we have an URL? I'm not aware of such a spreadsheet.
  
A makeshift one is here: https://wiki.mozilla.org/User:Nth10sd/MozMill_TB_Litmus_Tests

However, it's heavy on manual labour. I think there was some talk about an upgraded version of Litmus that can be designed to communicate with Mozmill result test servers to do everything automatically, which is a much better and more scalable idea than the manual entry spreadsheet, possibly similar to crash-tests at the moment, to see which tests are currently failing and for what version.

-Gary
nth10sd

Gary

unread,
Jan 12, 2009, 11:22:15 AM1/12/09
to Joel Maher, Henrik Skupin, Mark Banner, Ted Mielczarek, q...@mozilla.com, r, mozmi...@googlegroups.com
wrt. https://wiki.mozilla.org/User:Nth10sd/MozMill_TB_Litmus_Tests

I'm fine with transplanting that list to a Google Spreadsheet, time is scarce on my side though, I don't think I have the time anytime soon to create that spreadsheet, help is greatly appreciated.

I'm also agreeing with Joel that not all mozmill tests have to come from litmus. I think as many litmus tests should be converted to mozmill tests where possible, and mozmill tests certainly can contain other magical stuff. :)

-Gary
nth10sd


Clint Talbert

unread,
Jan 13, 2009, 12:54:07 PM1/13/09
to Mark Banner, Henrik Skupin, mozmi...@googlegroups.com, q...@mozilla.com, ctal...@mozilla.com, Martin Schröder, Gary Kwong, Ted Mielczarek
On 1/12/09 8:21 AM, Mark Banner wrote:
> I think you have mistaken this repository for the same thing I did.
> I thought:
>
> "This is a repository for locating tests against a product that may
> eventually be run repeatedly against the source like we do for unit
> tests"
>
Mozmill can also be used for unit tests, and unit tests will continue to
be checked into the main code tree near the code they test, just as they
are now with xpcshell and mochitests.

> "This is a repository for locating 'random' tests written by users for
> a product, that may just be run occasionally, and probably best on
> litmus tests and may never see automated running"
>
The proposed repo is for this use, to contain the tests written for
functional testing, not unit testing. Whether we decide to run this set
of tests on some kind of regular automatic basis is an entirely
different discussion.

I think our confusion comes from the two types of tests. Mozmill can be
used to write both unit tests and functional tests. Unit tests run
against a very specific part of the codebase, and are used to test one
unit of code. Functional tests are more broad based, involve different
components and possibly incorporate different parts of the code in more
user-focused interaction scenarios. So, given you can't pinpoint the
part of the code that functional tests come from you can't really put
them into the development tree.

There are other differences as well, and I think that the type of
automation run we want to do with the unit tests will probably be
different from the type of automation run we will want to do with the
functional tests. For one thing, the frequency of how often the tests
will be run will probably be different, where the results are reported
will probably also be different, for example, unit tests are primarily a
tool for developers so those tests should report to tinderbox.
Functional tests are tools for QA and those should report somewhere
else, such as the new results server that we are working on.

Hope this clears up the confusion.

Clint

Clint Talbert

unread,
Jan 13, 2009, 1:00:24 PM1/13/09
to Henrik Skupin, mozmi...@googlegroups.com, q...@mozilla.com, ctal...@mozilla.com, Mark Banner, Martin Schröder, Gary Kwong, Ted Mielczarek
On 1/12/09 6:21 AM, Henrik Skupin wrote:
> * All the projects that use mozmill should upload their tests there
>> so that we can all learn from each other's tests and test
>> methodologies.
>>
>
> Should we also have a review scheme? It would be great to have tests
> look similar. Everybody has his own coding style, which will make it
> sometimes hard to follow the test.
>
>
We probably will have something of a review scheme, but that discussion
is separate from the discussion of what repo to use and how to structure it.

>> If one group starts writing several utility scripts, then we could
>> ostensibly share those between all the projects, and having a central
>> repo will aid in that effort.
>>
>

> Shouldn't such utilities go directly into the MozMill source?
>
>
Possibly so. It really depends on what they are. I can imagine use
cases where we'd take something and put it into the source because it
works across all different applications, but we might not do that if it
is a small application specific API that we don't really want to have
the main product depend on (for instance, if we find some creative use
for gBrowser in the Firefox test suite, then that would probably never
get placed in the standard Mozmill API but it could be shared among
Firefox tests.) This is one of those "design for flexibility" things.

> Differentiating between products, versions and components would be a
> good idea, like we already do in hg:
>
> http://hg.mozilla.org/mozilla-central/file/ca9d3c35fe47/browser/components/privatebrowsing/test/
>
> A subfolder mozmill under the test folder of each neccessary product to
> test, would be a nice addition (also for the build system). But no idea
> how much work it needs to get fully integrated.
>
>
That makes sense for unit tests (and is probably what we'll do), but not
for functional tests. Please see my reply to Mark.

Clint

Clint Talbert

unread,
Jan 14, 2009, 10:22:57 PM1/14/09
to mozmi...@googlegroups.com, Adrian Kalla, Martin Schröder
Ok I think that everyone really seems to be leaning on the side of one
repository. The only question then left remaining is how to handle
future changes to the one repository. I have two solutions for that.

= Solution 1: The Cloning Solution =

In this solution we have a repo that has several directories in it for
the different products:
mozmill-tests
|
|____FirefoxTests
|
|____TbirdTests
|
|____<and so on>

Now, when one product releases a milestone and wants to branch their
test cases, they tag the entire tree and clone it. This will create
them a snapshot of all mozmill tests at that time. I think this is
really desirable, especially if we try to reuse common test modules.

The process for this is quite easy:
cd mozmill-tests
hg tag <myproj>-<myrelease>
cd ..
hg clone mozmill-tests <myproj>-<myrelease>

And that's it. Other people will be able to check out the cloned repo
and will be able to push changes to it.

= Solution 2: The Branching Solution =

The repository is the same structure. However, we use in-repository
branching to control the branching of the products. This is covered in
the HG book, chapter 8.5:
http://hgbook.red-bean.com/hgbookch8.html#x12-1650008.5

I've read this section a couple of times. It seems like the same thing
as above (but without the extra repos) for a whole lot more complexity.
I'm not confident this is the way to go as it seems just overly
complicated for no real gain. If we have any Hg gurus who can explain
its advantages better, I'm all ears.

= Conclusion =

Since we seem to be on the same page for one repo to rule them all and
that repo shall be Hg, then I'll file a bug tomorrow to get that repo
created for us. We can figure out the branch mechanics as we go. If
you have any reservations about the one repo strategy, speak up before
tomorrow afternoon PST.

Thanks,

Clint

Joel Maher

unread,
Jan 15, 2009, 6:36:04 AM1/15/09
to mozmi...@googlegroups.com, Adrian Kalla, Martin Schröder
I vote for Solution 1.

Henrik Skupin

unread,
Jan 15, 2009, 9:17:02 AM1/15/09
to mozmi...@googlegroups.com
On Thu, Jan 15, 2009 at 4:22 AM, Clint Talbert <ctal...@mozilla.com> wrote:

> In this solution we have a repo that has several directories in it for
> the different products:
> mozmill-tests
> |
> |____FirefoxTests
> |
> |____TbirdTests
> |
> |____<and so on>

We know that we only have tests located here. So maybe we don't need
"Tests" in the folder names, e.g.

mozmill-tests
Firefox
Thunderbird
Sunbird
...
SharedModules


> Since we seem to be on the same page for one repo to rule them all and
> that repo shall be Hg, then I'll file a bug tomorrow to get that repo
> created for us. We can figure out the branch mechanics as we go. If
> you have any reservations about the one repo strategy, speak up before
> tomorrow afternoon PST.

How do we handle that on hg.mozilla.org? Was solution 1 or 2 chosen?
Asking because of my limited background in hg.

Henrik

Clint Talbert

unread,
Jan 15, 2009, 9:39:21 AM1/15/09
to mozmi...@googlegroups.com
On 1/15/09 8:17 AM, Henrik Skupin wrote:
> On Thu, Jan 15, 2009 at 4:22 AM, Clint Talbert<ctal...@mozilla.com> wrote:
>
>
> We know that we only have tests located here. So maybe we don't need
> "Tests" in the folder names, e.g.
>
> mozmill-tests
> Firefox
> Thunderbird
> Sunbird
> ...
> SharedModules
>
>
Good point.

> e handle that on hg.mozilla.org? Was solution 1 or 2 chosen?
> Asking because of my limited background in hg.
>
>
On hg.mozilla.org, we clone when we branch.

Clint

Henrik Skupin

unread,
Jan 15, 2009, 9:57:04 AM1/15/09
to mozmi...@googlegroups.com
On Thu, Jan 15, 2009 at 3:39 PM, Clint Talbert <ctal...@mozilla.com> wrote:

We should also use a module-wise structure to have a good orientation, e.g.

>> mozmill-tests
>> Firefox
DownloadManager
Places
..
>> Thunderbird
Account Creation
Backend
..
>> Sunbird
>> ...
>> SharedModules

>> e handle that on hg.mozilla.org? Was solution 1 or 2 chosen?
>> Asking because of my limited background in hg.
>>
> On hg.mozilla.org, we clone when we branch.

I vote for that.

Henrik

Clint Talbert

unread,
Jan 15, 2009, 7:58:14 PM1/15/09
to mozmi...@googlegroups.com
Ok, I have filed bug *Bug 473859*
<https://bugzilla.mozilla.org/show_bug.cgi?id=473859> to create the repo.

Clint
Reply all
Reply to author
Forward
0 new messages