It is now possible to apply arbitrary tags to tests/manifests and run all tests with a given tag

[Andrew Halberstadt]

As of bug 987360, you can now run all tests with a given tag for
mochitest (and variants), xpcshell and marionette based harnesses. Tags
can be applied to either individual tests, or the DEFAULT section in
manifests. Tests can have multiple tags, in which case they should be
comma delimited. To run all tests with a given tag, pass in --tag <tag
name> to the mach command.

For example, let's say we want to group all mochitest-plain tests
related to canvas together. First we'd add a 'canvas' tag to the DEFAULT
section in
https://dxr.mozilla.org/mozilla-central/source/dom/canvas/test/mochitest.ini

[DEFAULT]
tags = canvas

We notice there is also a canvas related test under dom/media, namely:
https://dxr.mozilla.org/mozilla-central/source/dom/media/test/mochitest.ini#541

Let's pretend it is already tagged with the 'media' tag, but that's ok,
we can add a second tag no problem:

[test_video_to_canvas.html]
tags = media,canvas

Repeat above for any other tests or manifests scattered in the tree that
are related to canvas. Now we can run all mochitest-plain tests with:

./mach mochitest-plain --tag canvas

You can also run the union of two tags by specifying --tag more than
once (though the intersection of two tags is not supported):

./mach mochitest-plain --tag canvas --tag media

So far the xpcshell (./mach xpcshell-test --tag name) and marionette
(./mach marionette-test --tag name) commands are also supported. Reftest
is not supported as it has its own special manifest format.

Applying tags to tests will not affect automation or other people's
tags. So each organization or team should feel free to use tags in
whatever creative ways they see fit. Eventually, we'll start using tags
as a foundation for some more advanced features and analysis. For
example, we may implement a way to run all tests with a given tag across
multiple different suites.

If you have any questions or things aren't working, please let me know!

Cheers,
Andrew

[Nick Fitzgerald]

On Tue, Mar 31, 2015 at 9:30 AM, Andrew Halberstadt <
ahalbe...@mozilla.com> wrote:

>
> ​For example, we may implement a way to run all tests with a given tag
> across multiple different suites.
>
>

​Does this mean that I can't do `./mach​ test --tag foobar` to run both
xpcshell and mochitests tagged with foobar yet? Do I have to do `./mach
xpcshell-test --tag foobar` and `./mach mochitest-whatever --tag foobar`
separately?

Anyways, this looks like a very cool capability. Looking forward to using
it!

[Bobby Holley]

This sounds awesome! Is there an estimate of when we'll be able to use it
for try pushes?

On Tue, Mar 31, 2015 at 9:30 AM, Andrew Halberstadt <
ahalbe...@mozilla.com> wrote:

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Andrew Halberstadt]

It's technically already possible by modifying the in-tree mozharness
configs here:
https://dxr.mozilla.org/mozilla-central/source/testing/config/mozharness

However it's not easy to figure out what needs to be modified to get the
desired results. Some sort of |mach try| like command is going to be
worked on in Q2 to make pushing various test configurations to try
easier, I'm not 100% sure how it will be implemented yet though.

In the meantime, if you get stuck trying to modify the mozharness
configs, let me know and I can come up with an example patch.

-Andrew

On 31/03/15 01:06 PM, Bobby Holley wrote:
> This sounds awesome! Is there an estimate of when we'll be able to use
> it for try pushes?
>
> On Tue, Mar 31, 2015 at 9:30 AM, Andrew Halberstadt

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org <mailto:dev-pl...@lists.mozilla.org>
> https://lists.mozilla.org/listinfo/dev-platform
>
>

[Andrew Halberstadt]

Not yet, you still need to run the two suites separately. But we are
working towards that. Eventually ./mach test will be able to not only
run tags across suites, but run directories across suites and
auto-determine which suite a test path belongs to. In other words,
./mach test will become the norm, and running a specific test harness
directly will become the edge case. There is some work around this
specifically slotted in our Q2 goals.

-Andrew

[Andrew Halberstadt]

Not yet, you still need to run the two suites separately. But we are
working towards that. Eventually ./mach test will be able to not only
run tags across suites, but run directories across suites and
auto-determine which suite a test path belongs to. In other words,
./mach test will become the norm, and running a specific test harness
directly will become the edge case. There is some work around this
specifically slotted in our Q2 goals.

-Andrew

On 31/03/15 12:52 PM, Nick Fitzgerald wrote:

[Christopher Manchester]

I filed bug 1149670 for the mach try feature.

On Tue, Mar 31, 2015 at 10:39 AM, Andrew Halberstadt <
ahalbe...@mozilla.com> wrote:

> It's technically already possible by modifying the in-tree mozharness
> configs here:
> https://dxr.mozilla.org/mozilla-central/source/testing/config/mozharness
>
> However it's not easy to figure out what needs to be modified to get the
> desired results. Some sort of |mach try| like command is going to be worked
> on in Q2 to make pushing various test configurations to try easier, I'm not
> 100% sure how it will be implemented yet though.
>
> In the meantime, if you get stuck trying to modify the mozharness configs,
> let me know and I can come up with an example patch.
>
> -Andrew
>
>
> On 31/03/15 01:06 PM, Bobby Holley wrote:
>
>> This sounds awesome! Is there an estimate of when we'll be able to use it
>> for try pushes?
>>

>> On Tue, Mar 31, 2015 at 9:30 AM, Andrew Halberstadt <

>> _______________________________________________
>> dev-platform mailing list
>> dev-pl...@lists.mozilla.org <mailto:dev-pl...@lists.mozilla.org
>> >
>> https://lists.mozilla.org/listinfo/dev-platform
>>
>>
>>
> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org

> https://lists.mozilla.org/listinfo/dev-platform
>

[Bobby Holley]

Is there any reason we couldn't put it in the try syntax instead?

On Tue, Mar 31, 2015 at 10:39 AM, Andrew Halberstadt <
ahalbe...@mozilla.com> wrote:

> It's technically already possible by modifying the in-tree mozharness
> configs here:
> https://dxr.mozilla.org/mozilla-central/source/testing/config/mozharness
>
> However it's not easy to figure out what needs to be modified to get the
> desired results. Some sort of |mach try| like command is going to be worked
> on in Q2 to make pushing various test configurations to try easier, I'm not
> 100% sure how it will be implemented yet though.
>
> In the meantime, if you get stuck trying to modify the mozharness configs,
> let me know and I can come up with an example patch.
>
> -Andrew
>
>
>
> On 31/03/15 01:06 PM, Bobby Holley wrote:
>
> This sounds awesome! Is there an estimate of when we'll be able to use it
> for try pushes?
>
> On Tue, Mar 31, 2015 at 9:30 AM, Andrew Halberstadt <

[Andrew Halberstadt]

The main problem is that the current syntax is very inflexible. And tags
are just one of many things we want to make easier when pushing to try
(think of things like running directories or individual tests and all
the debug options like --run-until-failure). A try syntax that knows how
to handle all those cases will be just as convoluted as editing the
mozharness configs.

Also, how things are currently set up, we'd need to get the try syntax
to somehow modify how buildbot runs tests. This would be difficult if
not impossible. Taskcluster may make this easier, but I wouldn't count
on it..

[Gregory Szorc]

On Tue, Mar 31, 2015 at 10:44 AM, Andrew Halberstadt <
ahalbe...@mozilla.com> wrote:

> Not yet, you still need to run the two suites separately. But we are
> working towards that. Eventually ./mach test will be able to not only run
> tags across suites, but run directories across suites and auto-determine
> which suite a test path belongs to. In other words, ./mach test will become
> the norm, and running a specific test harness directly will become the edge
> case. There is some work around this specifically slotted in our Q2 goals.

The ability to `mach test <dir>` and have it run multiple suites is already
supported.
https://hg.mozilla.org/mozilla-central/file/0aa30282f496/testing/mach_commands.py#l166

[Andrew Halberstadt]

Cool, in that case getting it to run tags across suites that support
--tag shouldn't be terribly difficult. I filed bug 1149695.

[Andrew Halberstadt]

Minor update. It was pointed out that other list-like manifestparser
attributes (like head and support-files) are whitespace delimited
instead of comma delimited. To be consistent I switched tags to
whitespace delimitation as well.

E.g both these forms are ok:

[test_foo.html]
tags = foo bar baz

[test_bar.html]
tags =
foo
bar
baz

-Andrew

[Christopher Manchester]

You can now add "--tag" arguments to try syntax and they will get passed to
test harnesses in your try push. Details of the implementation are in bug
978846, but if you're interested in passing other arguments from try syntax
to a test harness, this can be done by adding those arguments to
testing/config/mozharness/try_arguments.py. Note this is still rather
coarse in the sense that arguments are forwarded without regard for whether
a harness supports a particular argument, but I can imagine it being useful
in a number of cases (for instance, when testing the feature with xpcshell
and "--tag devtools", I was able to get feedback in about ten minutes
whether things were working rather than waiting for every xpcshell test to
run).

Chris

[Gregory Szorc]

Wait - you're telling me that it is now possible to limit try pushes but
not just jobs but tests within jobs?! Stop the presses: this is huge! If
used by the masses, this could drastically reduce try turnaround times and
decrease automation load and costs.

Could we encourage use of --tag by having the automation scheduler
up-weight jobs that opt in to reduced load?

On Thu, Apr 30, 2015 at 4:21 PM, Christopher Manchester <
chmanc...@gmail.com> wrote:

> You can now add "--tag" arguments to try syntax and they will get passed to
> test harnesses in your try push. Details of the implementation are in bug
> 978846, but if you're interested in passing other arguments from try syntax
> to a test harness, this can be done by adding those arguments to
> testing/config/mozharness/try_arguments.py. Note this is still rather
> coarse in the sense that arguments are forwarded without regard for whether
> a harness supports a particular argument, but I can imagine it being useful
> in a number of cases (for instance, when testing the feature with xpcshell
> and "--tag devtools", I was able to get feedback in about ten minutes
> whether things were working rather than waiting for every xpcshell test to
> run).
>
> Chris
>
> On Thu, Apr 2, 2015 at 2:22 PM, Andrew Halberstadt <
> ahalbe...@mozilla.com
> > wrote:
>

[Chris AtLee]

Sounds great! I've filed
https://bugzilla.mozilla.org/show_bug.cgi?id=1161282 for this.

According to
https://secure.pub.build.mozilla.org/builddata/reports/reportor/daily/highscores/highscores.html,
we still have a ton of people using '-p all -u all' on try

[kgu...@mozilla.com]

On Thursday, April 30, 2015 at 7:22:26 PM UTC-4, Christopher Manchester wrote:
> You can now add "--tag" arguments to try syntax and they will get passed to
> test harnesses in your try push. Details of the implementation are in bug
> 978846, but if you're interested in passing other arguments from try syntax
> to a test harness, this can be done by adding those arguments to
> testing/config/mozharness/try_arguments.py. Note this is still rather
> coarse in the sense that arguments are forwarded without regard for whether
> a harness supports a particular argument, but I can imagine it being useful
> in a number of cases (for instance, when testing the feature with xpcshell
> and "--tag devtools", I was able to get feedback in about ten minutes
> whether things were working rather than waiting for every xpcshell test to
> run).

Can you provide a sample complete try syntax that uses this? I just did a try run with |try: -b do -p linux,macosx64,win32,win64 -u all -t none --tag apz| (after adding the apz tag to a mochitest.ini file) and it still ran all the mochitests instead of just the ones in the mochitest.ini file. Does it only work in some scenarios?

Cheers,
kats

[Christopher Manchester]

That looks like a valid use of the feature assuming the manifest was
annotated correctly. I'm not sure what went wrong, I'll reply here when I
figure out what went wrong or file a bug and investigate further.

Chris

[Christopher Manchester]

It looks like I inadvertently landed this change on an the wrong branch, so
we aren't pointing to it in production (although we were for a short time
when I tested last week). I'll straighten this out in the morning. Sorry
for the inconvenience.

Chris

On Mon, May 4, 2015 at 4:43 PM, Christopher Manchester <
chmanc...@gmail.com> wrote:

> That looks like a valid use of the feature assuming the manifest was
> annotated correctly. I'm not sure what went wrong, I'll reply here when I
> figure out what went wrong or file a bug and investigate further.
>
> Chris
>
> On Mon, May 4, 2015 at 4:23 PM, <kgu...@mozilla.com> wrote:
>

[Christopher Manchester]

This landed as the mach try command. For more details run |./mach help try|
(excerpted below) and see the discussion in bug 1149670. The highlights
here are partly automated try syntax generation for testing changes
relevant to tests in certain directories, and pushing try syntax to the try
server without the benefit of mq.

Chris


"""
Pushes the specified tests to try. The simplest way to specify tests is
by using the -u argument, which will behave as usual for try syntax.
This command also provides a mechanism to select test jobs and tests
within a job by path based on tests present in the tree under that
path. Mochitests, xpcshell tests, and reftests are eligible for
selection by this mechanism. Selected tests will be run in a single
chunk of the relevant suite, at this time in chunk 1.

Specifying platforms is still required with the -p argument (a default
is taken from the AUTOTRY_PLATFORM_HINT environment variable if set).

Tests may be further filtered by passing one or more --tag to the
command.

To run suites in addition to those determined from the tree, they
can be passed to the --extra arguent.

The command requires either its own mercurial extension ("push-to-try",
installable from mach mercurial-setup) or a git repo using git-cinnabar
(available at https://github.com/glandium/git-cinnabar).

Global Arguments:
-v, --verbose Print verbose output.
-l FILENAME, --log-file FILENAME
Filename to write log data to.
--log-interval Prefix log line with interval from last message
rather
than relative time. Note that this is NOT execution
time if there are parallel operations.
--log-no-times Do not prefix log lines with times. By default, mach
will prefix each output line with the time since
command start.
-h, --help Show this help message.
--debug-command Start a Python debugger when command is dispatched.

Command Arguments:
paths Paths to search for tests to run on try.
-n Print detailed information about the resulting test
selection and commands performed.
-p PLATFORMS Platforms to run. (required if not found in the
environment)
-u TESTS Test jobs to run. These will be used in place of
suites
determined by test paths, if any.
--extra EXTRA_TESTS Additional tests to run. These will be added to
suites
determined by test paths, if any.
-b BUILDS Build types to run (d for debug, o for optimized)
--tag TAGS Restrict tests to the given tag (may be specified
multiple times)
--no-push Do not push to try as a result of running this
command
(if specified this command will only print calculated
try syntax and selection info).

"""

On Tue, Mar 31, 2015 at 10:57 AM, Christopher Manchester <
chmanc...@gmail.com> wrote:

> I filed bug 1149670 for the mach try feature.
>

> On Tue, Mar 31, 2015 at 10:39 AM, Andrew Halberstadt <
> ahalbe...@mozilla.com> wrote:
>
>> It's technically already possible by modifying the in-tree mozharness
>> configs here:
>> https://dxr.mozilla.org/mozilla-central/source/testing/config/mozharness
>>
>> However it's not easy to figure out what needs to be modified to get the
>> desired results. Some sort of |mach try| like command is going to be worked
>> on in Q2 to make pushing various test configurations to try easier, I'm not
>> 100% sure how it will be implemented yet though.
>>
>> In the meantime, if you get stuck trying to modify the mozharness
>> configs, let me know and I can come up with an example patch.
>>
>> -Andrew
>>
>>
>> On 31/03/15 01:06 PM, Bobby Holley wrote:
>>
>>> This sounds awesome! Is there an estimate of when we'll be able to use
>>> it for try pushes?
>>>
>>> On Tue, Mar 31, 2015 at 9:30 AM, Andrew Halberstadt <

>>> _______________________________________________
>>> dev-platform mailing list
>>> dev-pl...@lists.mozilla.org <mailto:dev-platform@lists.
>>> mozilla.org>
>>> https://lists.mozilla.org/listinfo/dev-platform