== High Level Stuff ==
* Mozmill is a tool to enable anyone to write testcases against any
mozilla platform application.
* Mozmill strives to be a tool that even relatively non-technical people
can use to automate their manual test cases and can write testcases against.
* Mozmill is a giant lever in the eternal quest to build our community
by empowering a group of people who have traditionally been dropped into
the hole between manual testing and technical test automation. It also
has the ability to bring the people who do manual testing of non-Firefox
applications closer into the circle of the overall Project.
* Mozmill will help us to find new ways of automating and testing and
tracking test runs - perhaps by building out recording tools that people
can run once to record some unreproducible behavior, or it could be used
for L10N testing. So, whatever we do with the test repo, it has to be
flexible.
== Repo Goals ==
Okay, that's some high level stuff. Here are some things to consider
for the repository itself.
* It should be simple.
* It should be internet browsable and referencable
* All the projects that use mozmill should upload their tests there so
that we can all learn from each other's tests and test methodologies.
If one group starts writing several utility scripts, then we could
ostensibly share those between all the projects, and having a central
repo will aid in that effort.
* The tests need to be versioned, to track product versions as many of
them will be UI dependent, which might change from version to version.
* The repo should (if possible) further the goal of bringing people into
the Mozilla community while balancing against the need to keep the
barrier to entry as low as possible.
== My Proposal ==
We have the beginnings of a QA repository location in Mozilla SVN here:
http://svn.mozilla.org/qa/
The Mozilla SVN repo meets all the above goals pretty adequately: it's
simple, we can structure it so that it is versionable w.r.t. product
versions, and it's easily installed on all systems (comes by default on
mac os x 10.5, linux, and is included in the mozilla-build package on
windows). The one problem Mozilla SVN has is that it has a high bar to
entry for commit access, like all the mozilla repo's. I have some ideas
on how to mitigate this:
* A set of leaders is chosen from each project using Mozmill to shepherd
the checkin's from that project into the SVN repo. That person will get
SVN commit access first. This access will be across the entire test case
repo so that they can check in utility scripts, or tests into other
parts of the Mozmill test repo (for example, perhaps they have a unit
test for the elementlib in mozmill itself).
* That leader will also identify new members from their communities that
they wish to extend commit access to, and will help that person attain
it (filing proper bugs for it, harassing people to get them cleared etc,
whatever it takes - it's actually a fairly simple process, once the
process is started).
* The leaders will have direct ability to manipulate their project's
test directory structure so that they can easily create areas for tests
that are version dependent (according to their product version).
* These leaders will ensure (especially as new contributors are
identified and added) that there is no single person who can become a
bottleneck w.r.t. creating areas for tests against new versions of
products and/or checking in tests themselves.
Thoughts, comments, feedback, and new proposals are encouraged. I look
forward to hearing yours.
Clint
Are we sure that the class of users who we want to ramp up to mozmill (not super technical) will easily be able to master svn?
On 12/31/2008 03:37, Al Billings wrote:
> I think that this is an excellent suggestion.
>
> Are we sure that the class of users who we want to ramp up to mozmill
> (not super technical) will easily be able to master svn?
>
> I would like to see a simple mirroring/shadowing of the repository
> files (at least the most current version) to a website for easy access.
Is svn really the good choice here? I would have thought a Mercurial
repository would be much better - it is where mozilla source control is
typically going anyway and it will be one less repository system for
devs (who will want to write mozmill tests).
The main benefit of a Mercurial repository is that if we do decide to
run tests against the current sources continuously, I think it would be
much easier for the build team to integrate pulling from an hg
repository than it would a svn one. Although it has been mentioned that
running Mozmill tests against sources continuously isn't necessarily the
best thing, as we have no UI level tests (apart from leak & bloat)
available for Thunderbird, then I'm considering it as at least a short
to medium-term option and seeing how it goes.
Standard8
On 01/12/2009 12:21, Henrik Skupin wrote:
>> == Repo Goals == * It should be simple.
>>
>
> We probably have to find a way between simplicity and running tests from
> within the build system. When bug 457265 (Integrate Mozmill into build
> system) is fixed, the build system will be able to automatically run
> MozMill tests. All tests should be accessible therefor. I would tend to
> second the simplicity here. With a good tutorial everybody should be
> able to checkout existing tests and run them locally.
>
I think you have mistaken this repository for the same thing I did. I
thought:
"This is a repository for locating tests against a product that may
eventually be run repeatedly against the source like we do for unit tests"
from the discussions so far I think this "test repository" is:
"This is a repository for locating 'random' tests written by users for a
product, that may just be run occasionally, and probably best on litmus
tests and may never see automated running"
I think we do need to have a clear description of what the repository is
for, as I think there is ambiguity here.
Standard8
A makeshift one is here: https://wiki.mozilla.org/User:Nth10sd/MozMill_TB_Litmus_Testsmuch, much easier. At the moment, Gary's spreadsheet that correlates litmus cases to Mozmill testcases for Thunderbird smoketests are probably the way to go here.Oh, do we have an URL? I'm not aware of such a spreadsheet.
I think our confusion comes from the two types of tests. Mozmill can be
used to write both unit tests and functional tests. Unit tests run
against a very specific part of the codebase, and are used to test one
unit of code. Functional tests are more broad based, involve different
components and possibly incorporate different parts of the code in more
user-focused interaction scenarios. So, given you can't pinpoint the
part of the code that functional tests come from you can't really put
them into the development tree.
There are other differences as well, and I think that the type of
automation run we want to do with the unit tests will probably be
different from the type of automation run we will want to do with the
functional tests. For one thing, the frequency of how often the tests
will be run will probably be different, where the results are reported
will probably also be different, for example, unit tests are primarily a
tool for developers so those tests should report to tinderbox.
Functional tests are tools for QA and those should report somewhere
else, such as the new results server that we are working on.
Hope this clears up the confusion.
Clint
>> If one group starts writing several utility scripts, then we could
>> ostensibly share those between all the projects, and having a central
>> repo will aid in that effort.
>>
>
> Shouldn't such utilities go directly into the MozMill source?
>
>
Possibly so. It really depends on what they are. I can imagine use
cases where we'd take something and put it into the source because it
works across all different applications, but we might not do that if it
is a small application specific API that we don't really want to have
the main product depend on (for instance, if we find some creative use
for gBrowser in the Firefox test suite, then that would probably never
get placed in the standard Mozmill API but it could be shared among
Firefox tests.) This is one of those "design for flexibility" things.
> Differentiating between products, versions and components would be a
> good idea, like we already do in hg:
>
> http://hg.mozilla.org/mozilla-central/file/ca9d3c35fe47/browser/components/privatebrowsing/test/
>
> A subfolder mozmill under the test folder of each neccessary product to
> test, would be a nice addition (also for the build system). But no idea
> how much work it needs to get fully integrated.
>
>
That makes sense for unit tests (and is probably what we'll do), but not
for functional tests. Please see my reply to Mark.
Clint
= Solution 1: The Cloning Solution =
In this solution we have a repo that has several directories in it for
the different products:
mozmill-tests
|
|____FirefoxTests
|
|____TbirdTests
|
|____<and so on>
Now, when one product releases a milestone and wants to branch their
test cases, they tag the entire tree and clone it. This will create
them a snapshot of all mozmill tests at that time. I think this is
really desirable, especially if we try to reuse common test modules.
The process for this is quite easy:
cd mozmill-tests
hg tag <myproj>-<myrelease>
cd ..
hg clone mozmill-tests <myproj>-<myrelease>
And that's it. Other people will be able to check out the cloned repo
and will be able to push changes to it.
= Solution 2: The Branching Solution =
The repository is the same structure. However, we use in-repository
branching to control the branching of the products. This is covered in
the HG book, chapter 8.5:
http://hgbook.red-bean.com/hgbookch8.html#x12-1650008.5
I've read this section a couple of times. It seems like the same thing
as above (but without the extra repos) for a whole lot more complexity.
I'm not confident this is the way to go as it seems just overly
complicated for no real gain. If we have any Hg gurus who can explain
its advantages better, I'm all ears.
= Conclusion =
Since we seem to be on the same page for one repo to rule them all and
that repo shall be Hg, then I'll file a bug tomorrow to get that repo
created for us. We can figure out the branch mechanics as we go. If
you have any reservations about the one repo strategy, speak up before
tomorrow afternoon PST.
Thanks,
Clint
> In this solution we have a repo that has several directories in it for
> the different products:
> mozmill-tests
> |
> |____FirefoxTests
> |
> |____TbirdTests
> |
> |____<and so on>
We know that we only have tests located here. So maybe we don't need
"Tests" in the folder names, e.g.
mozmill-tests
Firefox
Thunderbird
Sunbird
...
SharedModules
> Since we seem to be on the same page for one repo to rule them all and
> that repo shall be Hg, then I'll file a bug tomorrow to get that repo
> created for us. We can figure out the branch mechanics as we go. If
> you have any reservations about the one repo strategy, speak up before
> tomorrow afternoon PST.
How do we handle that on hg.mozilla.org? Was solution 1 or 2 chosen?
Asking because of my limited background in hg.
Henrik
Clint
We should also use a module-wise structure to have a good orientation, e.g.
>> mozmill-tests
>> Firefox
DownloadManager
Places
..
>> Thunderbird
Account Creation
Backend
..
>> Sunbird
>> ...
>> SharedModules
>> e handle that on hg.mozilla.org? Was solution 1 or 2 chosen?
>> Asking because of my limited background in hg.
>>
> On hg.mozilla.org, we clone when we branch.
I vote for that.
Henrik