Re: ./mach try fuzzy: A Try Syntax Alternative

[Andrew Halberstadt]

Btw, there's now a non-interactive mode landed with -q/--query. E.g to run
all browser-chrome:
./mach try fuzzy -q "'browser-chrome"

(You may want to first enter interactive mode to make sure your query will
actually select what you want, ctrl-c, then retype the query with -q so
it's in your shell history)

-Andrew

On Wed, Aug 2, 2017 at 10:30 AM Andrew Halberstadt <ahalbe...@mozilla.com>
wrote:

> I'm pleased to announce an alternative method for scheduling tasks on try
> is now landed on mozilla-central. It makes use of the awesome fzf [1]
> project to filter down the list of all task labels with a fuzzy matching
> algorithm.
>
> It works both with mercurial or git. If using mercurial, you'll need to
> make sure you're updated to the latest version-control-tools:
>
> $ ./mach mercurial-setup --update
>
> To push to try, run:
>
> $ ./mach try fuzzy
>
> This will prompt you to install fzf. After bootstrapping is complete,
> it'll generate the task list. This takes around ~10-20 seconds, but will be
> cached so subsequent invocations won't incur this penalty again.
>
> Now you'll be in the fzf interface. Basic usage is to start typing to
> filter down the task list. You can use the arrow keys to move the cursor up
> or down, <tab> to select a task, <ctrl-a> to select all tasks and <enter>
> to schedule the current selection (and their dependencies) on try.
>
> There are a ton more keyboard shortcuts and features you can use to tweak
> fzf just to your liking. For extra help, see:
>
> $ ./mach try fuzzy --help
> or
> $ man fzf
>
> For a demo and more information on implementation details, see:
> https://ahal.ca/blog/2017/mach-try-fuzzy/
>
> I expect this to work on all platforms including Windows for both
> mercurial (with push-to-try) and git (with git-cinnabar). But it's a new
> tool without a lot of real world testing, so a few bumps are likely. If you
> find any bugs or bad UX, please file a bug under Testing :: General and let
> me know (we should find a better component for this).
>
> Cheers,
> Andrew
>
> p.s When running the fzf bootstrap, you'll be prompted to install some
> shell extensions. These aren't necessary for |mach try fuzzy| to work so
> feel free to hit 'N' at each prompt. That being said, the fzf shell
> extensions are awesome. I highly encourage you to look into setting it up
> for your shell. The fzf vim plugin is also great. See the project [1] for
> more details.
>
> [1] https://github.com/junegunn/fzf
>

[Andrew Halberstadt]

+to...@lists.mozilla.org

There have been a bunch of new features added here I'd like to highlight:

- --artifact/--no-artifact: This works similarly to try syntax. If a
local build with artifact builds is detected, --artifact will be used
automatically. Unlike try syntax, this will change your treeherder symbols
to 'Ba' to make it more obvious artifact builds are being used.
- --env: You can now set environment variables in your tasks directly
from the command line, e.g:
- ./mach try fuzzy --env XPCOM_DEBUG_BREAK=stack --env
MOZ_LOG="example_logger:3"|
- --full: Populate the fuzzy interface with the "full" set of tasks.
This will let you choose tasks that were previously hard to select (e.g
nightlies, l10n, signing tasks, etc)
- --save/--preset: Works the same as try syntax, using the --query
argument. E.g:
- ./mach try fuzzy --save reftest -q "!cov !pgo 'reftest !jsreftest"
- ./mach try --preset reftest

See |mach try fuzzy --help| for more information.

At this point I consider |mach try fuzzy| to be superior to try syntax in
almost every way [1], and would encourage people to start using it by
default. To do this, create a ~/.mozbuild/machrc file (if you don't already
have one) and add:
[try]
default = fuzzy

Now when you run |mach try| it will default to the fuzzy interface instead
of try syntax. If you need to use try syntax for some reason, you can still
run |mach try syntax -b do ... |. By the end of Q3 I plan on writing up an
outline to deprecate try syntax (it'll likely never be removed completely)
in favour of 'try_task_config.json' mechanisms. Nothing major will change
anytime soon, but it's worth being aware that there are now better methods
for scheduling tasks and finding people willing to maintain try syntax is
going to be even harder than it was before.

-Andrew

[1] Try syntax still has a few benefits over fuzzy. You can specify paths,
use --rebuild and it will detect compiled tests and error out if you try to
use artifact builds. There are bugs on file to reach parity here.

[Randell Jesup]

>+to...@lists.mozilla.org
>
>There have been a bunch of new features added here I'd like to highlight:

> - --env: You can now set environment variables in your tasks directly
> from the command line, e.g:
> - ./mach try fuzzy --env XPCOM_DEBUG_BREAK=stack --env
> MOZ_LOG="example_logger:3"|

This is *awesome*; I've been wanting this for YEARS. Does this work
without 'fuzzy'?

> - --save/--preset: Works the same as try syntax, using the --query
> argument. E.g:
> - ./mach try fuzzy --save reftest -q "!cov !pgo 'reftest !jsreftest"
> - ./mach try --preset reftest

Also really great.

--
Randell Jesup, Mozilla Corp
remove "news" for personal email

[Andrew Halberstadt]

On Sat, Sep 2, 2017 at 1:00 PM Randell Jesup <rje...@jesup.org> wrote:

> >+to...@lists.mozilla.org
> >
> >There have been a bunch of new features added here I'd like to highlight:
>
> > - --env: You can now set environment variables in your tasks directly
> > from the command line, e.g:
> > - ./mach try fuzzy --env XPCOM_DEBUG_BREAK=stack --env
> > MOZ_LOG="example_logger:3"|
>
> This is *awesome*; I've been wanting this for YEARS. Does this work
> without 'fuzzy'?
>

Yes and no :).

There has been a --setenv option to try syntax for a fair while now, but
I don't think it works with all tasks (likely just test tasks like
mochitest,
reftest, xpcshell, etc). I haven't tried this myself though, I'm not even
sure
if it's still working or not.

The |mach try fuzzy| implementation is built on top of a much more general
purpose mechanism that isn't (and won't be) available to try syntax. The
main
benefit of the |mach try fuzzy| implementation is that it will set the env
in
every task that gets scheduled no matter what kind/type.

> - --save/--preset: Works the same as try syntax, using the --query
> > argument. E.g:
> > - ./mach try fuzzy --save reftest -q "!cov !pgo 'reftest !jsreftest"
> > - ./mach try --preset reftest
>
> Also really great.
>
> --
> Randell Jesup, Mozilla Corp
> remove "news" for personal email

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Dustin Mitchell]

That general mechanism is open-ended, though -- fuzzy is just one
interface to it. The possibilities for shortcuts, purpose-specific
commands, and automation are enormous.

https://firefox-source-docs.mozilla.org/taskcluster/taskcluster/try.html

Dustin

2017-09-02 23:04 GMT-04:00 Andrew Halberstadt <ahalbe...@mozilla.com>:
> On Sat, Sep 2, 2017 at 1:00 PM Randell Jesup <rje...@jesup.org> wrote:
>

>> >+to...@lists.mozilla.org
>> >
>> >There have been a bunch of new features added here I'd like to highlight:
>>
>> > - --env: You can now set environment variables in your tasks directly
>> > from the command line, e.g:
>> > - ./mach try fuzzy --env XPCOM_DEBUG_BREAK=stack --env
>> > MOZ_LOG="example_logger:3"|
>>
>> This is *awesome*; I've been wanting this for YEARS. Does this work
>> without 'fuzzy'?
>>
>

> Yes and no :).
>
> There has been a --setenv option to try syntax for a fair while now, but
> I don't think it works with all tasks (likely just test tasks like
> mochitest,
> reftest, xpcshell, etc). I haven't tried this myself though, I'm not even
> sure
> if it's still working or not.
>
> The |mach try fuzzy| implementation is built on top of a much more general
> purpose mechanism that isn't (and won't be) available to try syntax. The
> main
> benefit of the |mach try fuzzy| implementation is that it will set the env
> in
> every task that gets scheduled no matter what kind/type.
>
>

>> - --save/--preset: Works the same as try syntax, using the --query
>> > argument. E.g:
>> > - ./mach try fuzzy --save reftest -q "!cov !pgo 'reftest !jsreftest"
>> > - ./mach try --preset reftest
>>
>> Also really great.
>>
>> --
>> Randell Jesup, Mozilla Corp
>> remove "news" for personal email

[William Lachance]

On 2017-09-02 11:04 PM, Andrew Halberstadt wrote:
>>> - --env: You can now set environment variables in your tasks directly
>>> from the command line, e.g:
>>> - ./mach try fuzzy --env XPCOM_DEBUG_BREAK=stack --env
>>> MOZ_LOG="example_logger:3"|

>> This is*awesome*; I've been wanting this for YEARS. Does this work

>> without 'fuzzy'?
>>
> Yes and no:).

> ...

There's also the "retrigger job with extra options" action which you can
trigger directly from treeherder, that lets you set the environment,
preferences, and a few other things:

https://wlach.github.io/blog/2017/04/easier-reproduction-of-intermittent-test-failures-in-automation/

That requires having an existing job to use as a template though.

Will

[Dustin Mitchell]

2017-09-03 13:51 GMT-04:00 William Lachance <wlac...@mozilla.com>:
> There's also the "retrigger job with extra options" action which you can
> trigger directly from treeherder, that lets you set the environment,
> preferences, and a few other things:
>
> https://wlach.github.io/blog/2017/04/easier-reproduction-of-intermittent-test-failures-in-automation/
>
> That requires having an existing job to use as a template though.

Actions, too, are defined in-tree, and it's easy to define your own.
So if you have a common need to re-run a task with certain env vars
set, or extra command-line options, or whatever -- you can do that! A
"retrigger with verbose logging" action would be trivial to develop,
for example.

Maybe I'm sounding like a broken record, but the I think this
particular sample deserves repeating (and maybe a bass drop for
emphasis).

Dustin

[Botond Ballo]

Is there an easy way to do a T-push (builds on all platforms, tests on
one platform only) with |mach try fuzzy|?

I usually do T-pushes using try syntax, by Trychooser seems to be out
of date when it comes to building a T-push syntax for Android, so I'm
at a loss as to how to do a T-push for Android right now.

Thanks,
Botond

On Wed, Aug 2, 2017 at 10:30 AM, Andrew Halberstadt

[James Graham]

On 06/08/2018 01:25, Botond Ballo wrote:
> Is there an easy way to do a T-push (builds on all platforms, tests on
> one platform only) with |mach try fuzzy|?
>
> I usually do T-pushes using try syntax, by Trychooser seems to be out
> of date when it comes to building a T-push syntax for Android, so I'm
> at a loss as to how to do a T-push for Android right now.

There are a couple of options. Interactively you can select all the
builds you want, press ctrl+a (or whatever the select-all keybinding you
have configured is), then do the same again with the tests you want,
then accept all your choices.

If you want to construct a single query string that can be reused with
--save, something like 'test-linux64 | build !ccov !pgo !msvc' seems to
select all builds and tests just on linux64. Unfortunately I can't
figure out any way to logically group expressions, which does make
composing multiple terms more tricky.

[Andrew Halberstadt]

I recently added the ability to specify --query multiple
times (where the set of tasks is the union of each
individual query). So something like:

./mach try fuzzy -q "'android !pgo !cov" -q "'build !pgo !cov"

Should also accomplish what you want. It's still a bit clunky
as multiple queries don't work with --save yet (just the first
query will be saved). But at least this will get the command
in your shell history (and you can also use |mach try again| to
rerun it).