Intent to remove: mochitest-<flavor> mach commands

[Andrew Halberstadt]

As mentioned previously in another post, work is under way to remove the
flavor specific mochitest commands (e.g mach mochitest-plain, mach
mochitest-browser etc.). This is being tracked in bug 1164597 [1].
Bug description pasted below:

> As part of an effort to make running tests as easy and intuitive as
> possible, I'd like to consolidate all of the mochitest-<flavor> mach
> commands into the single |mach mochitest| command.
>
> Currently |mach mochitest| will automatically detect the flavor of
> mochitests you wish to run. If you specify a directory that contains
> multiple flavors, each flavor will be run in sequence. For example:
>
> ./mach mochitest dom/indexedDB
>
> runs all the mochitest-chrome, mochitest-browser-chrome and
> mochitest-plain tests in that directory. The flavor can be limited
> by passing in -f (--flavor). So to only run the chrome tests in that
> directory you'd use:
>
> ./mach mochitest -f chrome dom/indexedDB
>
> Commands of the form |mach mochitest-<flavor>| will be removed:
>
> ./mach mochitest-plain -> ./mach mochitest -f plain ./mach
> mochitest-browser -> ./mach mochitest -f browser etc..
>
> I believe this will make running tests easier and more intuitive.
> The fact that different flavors of mochitest exist will become a
> background implementation detail. Instead, an emphasis will be
> placed on running directories of tests, or tests related to certain
> features. The knowledge of which test belongs to which flavor will
> no longer be required, but of course the ability to run specific
> flavors will still be there for those who need it.
>
> The described behaviour for |mach mochitest| above, already exists
> today. This bug is about making it the defacto way to run mochitests
> (maybe aside from |mach test|). This bug will also add b2g and
> android support to |mach mochitest|.
>
> Exceptions to this will be robocop, webapprt-chrome and
> webapprt-content which for technical reasons will not be rolled into
> |mach mochitest| (for now).


Please let me know if you have any questions or concerns,
-Andrew

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1164597

[Brian Grinstead]

Does this include subsuites like devtools? There is currently a bug on file in which `mach mochitest` doesn’t work with devtools tests (1122590). Until that is resolved, if mochitest-devtools was removed then running these tests would become the harder-to-remember `./mach mochitest browser/devtools --subsuite devtools` as opposed to `./mach mochitest-devtools browser/devtools`. I'd request that mochitest-devtools remains in place until it's possible to run simply `mach mochitest browser/devtools`

Thanks,
Brian

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform

[Andrew Halberstadt]

On 13/05/15 04:13 PM, Brian Grinstead wrote:
> Does this include subsuites like devtools? There is currently a bug
> on file in which `mach mochitest` doesn’t work with devtools tests
> (1122590). Until that is resolved, if mochitest-devtools was removed
> then running these tests would become the harder-to-remember `./mach
> mochitest browser/devtools --subsuite devtools` as opposed to `./mach
> mochitest-devtools browser/devtools`. I'd request that
> mochitest-devtools remains in place until it's possible to run simply
> `mach mochitest browser/devtools`
>
> Thanks, Brian

Yes it does. I actually already have ./mach mochitest browser/devtools
working in my local patch (posted in bug 1164597). We'll be able to
resolve bug 1122590 automatically once it lands.

Cheers,
Andrew

p.s ./mach mochitest browser/devtools --subsuite devtools works currently.

[Ms2ger]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/13/2015 09:54 PM, Andrew Halberstadt wrote:
> As mentioned previously in another post, work is under way to
> remove the flavor specific mochitest commands (e.g mach
> mochitest-plain, mach mochitest-browser etc.).
>

Does this include `mochitest-1` and friends? These are a lot easier to
use than having to figure out the exact incantation for the chunked
suites.

HTH
Ms2ger

-----BEGIN PGP SIGNATURE-----

iQEcBAEBAgAGBQJVU7vsAAoJEOXgvIL+s8n229EH/A+dFtY8nqDOCywDGBjgU8cL
V32tfYfv3tISII8YCP9Jsl58SW3YFpUk+inneUdVKXYce5OAkr4furpE64JbA7Kq
RInF56lkdKagTvJyKS8VPsTHZ/kEOGFAfSPDD3pxlUjsZgqwuBxwtUPLR7NkLkWx
/xViL9ZNqgXiveWPsrpYKDEYqxVgmm9/XNVCqXX8R0z3r3jI4BWCXiwjb5ELE7sD
S5sqknCBezlO2WYNUG4Fir8hRlESjQeJtqoCfhJ2x1MJotRWoGCuk4ktEU6QIYRS
zmeY/3miXbpUOhqNSCC/CFasRcF9jLwzs8OQChFTMDW3DMSMDgtdNj1xx+FV9z4=
=C348
-----END PGP SIGNATURE-----

[Gregory Szorc]

On Wed, May 13, 2015 at 2:03 PM, Ms2ger <ms2...@gmail.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 05/13/2015 09:54 PM, Andrew Halberstadt wrote:
> > As mentioned previously in another post, work is under way to
> > remove the flavor specific mochitest commands (e.g mach
> > mochitest-plain, mach mochitest-browser etc.).
> >
>
> Does this include `mochitest-1` and friends? These are a lot easier to
> use than having to figure out the exact incantation for the chunked
> suites.
>

IIRC `mochitest-1` and other job-name based arguments are only
exposed/supported via `mach test`. I don't believe Andrew's work will
impact these.

[Andrew Halberstadt]

Yes, those will not be impacted. Those may even be exposed to the |mach
mochitest| command in the future.

Unrelated side note: I think it would be nice if the harness specific
mach commands did all of the leg work and |mach test| just delegated to
them. I'm looking into it but haven't yet figured out a good way of
doing it. Until then, there will be a slightly different feature set
between |mach test| and |mach mochitest|.

[Andrew Halberstadt]

This has now been merged to central. I've update the docs at:
https://developer.mozilla.org/en-US/docs/Mochitest
https://developer.mozilla.org/en-US/Firefox_OS/Automated_testing/Mochitests

In most cases just running |mach mochitest path/to/test/or/dir| will do
what you want. But if for some reason you need to restrict tests to a
specific flavor or subsuite, you can use --flavor (-f) or --subsuite
respectively.

E.g:
|mach mochitest-plain| becomes |mach mochitest -f plain|

Bug 1122590 is also fixed, so running |mach mochitest browser/devtools|
should now work. To exactly mimic the old |mach mochitest-devtools|
command, you'd use:
|mach mochitest -f browser --subsuite devtools|

Trying to run one of the old removed commands will print out a helpful
error message that provides examples using the new syntax.

If you have questions or things aren't quite working as expected, please
let me know.

-Andrew