PSA: "cros tryjob" preview available

[Don Garrett]

TLDR:

There is a testing version of the "cros tryjob" command available for submitting tryjobs.

Details:

I'm planning to move control of both local and remote tryjobs out of cbuildbot and into a dedicated command. The new command now exists and is ready for testing. After we're all convinced it works, I'll remove the tryjob behavior from the cbuildbot command.

The syntax is similar to, but not identical to cbuildbot tryjobs. The behavior after the tryjob is launched should be identical. One important difference (which mostly only affects TPMs) is that you should never use "--buildbot" with this command. Use "--production" instead. 

Please try this out, and let me know about any issues you hit.

cros tryjob --help

usage: cros tryjob [-h]

                   [--log-level {fatal,critical,error,warning,notice,info,debug}]

                   [--log_format LOG_FORMAT] [--debug] [--nocolor] [-b BRANCH]

                   [--hwtest] [--yes] [--production] [--passthrough ...]

                   [--local | --remote] [-r BUILDROOT] [-g GERRIT_PATCHES]

                   [-G RIETVELD_PATCHES] [-p LOCAL_PATCHES]

                   [--committer-email COMMITTER_EMAIL]

                   [--remote-description REMOTE_DESCRIPTION]

                   [--version VERSION] [--channel CHANNELS]

                   build_configs [build_configs ...]

Schedule a tryjob.

positional arguments:

  build_configs         One or more configs to build.

optional arguments:

  -h, --help            show this help message and exit

  -b BRANCH, --branch BRANCH

                        The manifest branch to test. The branch to check the

                        buildroot out to.

  --hwtest              Enable hwlab testing. Default false, except for

                        production.

  --yes                 Never prompt to confirm.

  --production          This is a production build, NOT a test build. Use with

                        care.

  --passthrough ...     Arguments to pass to cbuildbot.

Debug options:

  --log-level {fatal,critical,error,warning,notice,info,debug}

                        Set logging level to report at.

  --log_format LOG_FORMAT

                        Set logging format to use.

  --debug               Alias for `--log-level=debug`. Useful for debugging

                        bugs/failures.

  --nocolor             Do not use colorized output (or `export NOCOLOR=true`)

Where:

  Where do we run the tryjob?

  --local               Run the tryjob on your local machine.

  --remote              Run the tryjob on a remote builder. (default)

  -r BUILDROOT, --buildroot BUILDROOT

                        Root directory to use for the local tryjob. NOT the

                        current checkout.

Patch:

  Which patches should be included with the tryjob?

  -g GERRIT_PATCHES, --gerrit-patches GERRIT_PATCHES

                        Space-separated list of short-form Gerrit Change-Id's

                        or change numbers to patch. Please prepend '*' to

                        internal Change-Id's

  -G RIETVELD_PATCHES, --rietveld-patches RIETVELD_PATCHES

                        Space-separated list of short-form Rietveld issue

                        numbers to patch. If no subdir is specified, the src

                        directory is used.

  -p LOCAL_PATCHES, --local-patches LOCAL_PATCHES

                        Space-separated list of project branches with patches

                        to apply. Projects are specified by name. If no branch

                        is specified the current branch of the project will be

                        used.

Requestor:

  Who is submitting the jobs?

  --committer-email COMMITTER_EMAIL

                        Override default git committer email.

  --remote-description REMOTE_DESCRIPTION

                        Attach an optional description to a --remote run to

                        make it easier to identify the results when it

                        finishes

Specialty:

  Options only used by specific tryjobs.

  --version VERSION     Specify the release version for payload regeneration.

                        Ex: 9799.0.0

  --channel CHANNELS    Specify a channel for a payloads trybot. Can be

                        specified multiple times. No valid for non-payloads

                        configs.

Remote Examples:

  cros tryjob -g 123 lumpy-compile-only-pre-cq

  cros tryjob -g 123 -g 456 lumpy-compile-only-pre-cq daisy-pre-cq

  cros tryjob -g 123 --hwtest daisy-paladin

Local Examples:

  cros tryjob --local -g 123 daisy-paladin

  cros tryjob --local --buildroot /my/cool/path -g 123 daisy-paladin

Production Examples (danger, can break production if misused):

  cros tryjob --production --branch release-R61-9765.B asuka-release

  cros tryjob --production --version 9795.0.0 --channel canary  lumpy-payloads

[Mike Frysinger]

On Mon, Aug 7, 2017 at 3:32 PM, Don Garrett wrote:

optional arguments:

  -h, --help            show this help message and exit

  -b BRANCH, --branch BRANCH

                        The manifest branch to test. The branch to check the

                        buildroot out to.

  --hwtest              Enable hwlab testing. Default false, except for

                        production.

imo we should create another option group for --hwtest and explicitly document the other test related flags rather than forcing people to utilize --passthrough.

    --notests           Override values from buildconfig, run no tests, and

                        build no autotest and artifacts.

    --novmtests         Override values from buildconfig, run no vmtests.

    --noimagetests      Override values from buildconfig and run no image

                        tests.

-mike

[Don Garrett]

Reasonable. I was considering removing those options based on the idea that they are rarely used.

We've generally gone towards build configs for different levels of testing.

[Don Garrett]

My intention was to remove "--notests", "--novmtests", and "--noimagetests" is there are no strong objections. You can achieve similar effects by using restricted build configurations like "<board>-compile-only-pre-cq". If people are using them, I'd like to hear about it.

On Tue, Aug 8, 2017 at 12:20 PM Mike Frysinger <vap...@chromium.org> wrote:

[Don Garrett]

My intention is to remove "cbuildbot --remote" and "cbuildbot --local" next week.

If there is any reason why "cros tryjob" is in sufficient, now is the time to speak up.

[Brian Norris]

What's the expected auth mechanism?

$ cros tryjob -g '*414309' whirlwind-test-ap

Verifying patches...

Submitting tryjob...

17:23:17: WARNING: Error getting tokens with service_account None: Not logged in. Run 'authutil login'.

Getting a refresh token with following OAuth scopes:

  * https://www.googleapis.com/auth/userinfo.email

Visit the following URL to get the authorization code and copy-paste it below.

...[URL trimmed]...

$ authutil login

-bash: authutil: command not found

I went through the copy/paste auth successfully, but the first WARNING is confusing.

--
--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en

[Don Garrett]

Improving that message is a bug we should address, but I believe that behavior is unchanged from "cbuildbot --remote".

The authentication in question is to talk to buildbucket to schedule the build. The message is worded that way, because it was originally written to run on build machines which use service accounts to perform that authentication.

[Mike Frysinger]

not all configs have such duplication.  for example, chromiumos-sdk.  i'm not sure the right answer is to force multiplication of all possible configs ...

-mike

[Manoj Gupta]

Does <board>-compile-pre-cq also builds chrome? 

--

[Don Garrett]

If people really use those flags, I'll maintain them. Just looking for unused stuff to remove during the transition.

[Mike Frysinger]

i think the option groups help quite a bit, but can't argue with that only going so far

btw, i dig the reflective questions included in the option groups

-mike

[Don Garrett]

On Mon, Aug 14, 2017 at 8:57 PM Manoj Gupta <manoj...@chromium.org> wrote:

Does <board>-compile-pre-cq also builds chrome? 

To the best of my knowledge, none of the PFQ configs compile Chrome.

[Mike Frysinger]

On Tue, Aug 15, 2017 at 2:13 PM, Don Garrett <dgar...@chromium.org> wrote:

On Mon, Aug 14, 2017 at 8:57 PM Manoj Gupta <manoj...@chromium.org> wrote:

Does <board>-compile-pre-cq also builds chrome? 

To the best of my knowledge, none of the PFQ configs compile Chrome.

*-chrome-pfq* and *-chromium-pfq* do, but these PFQs are not equivalent ;)

-mike

[Don Garrett]

I meant none of the PreCQ build configs compile Chrome.

One again, I confused PFQ and PreCQ when typing or speaking. It's very confusing to everyone, since I think I said/wrote the right thing.

[Ting-Yuan Huang]

Hi Don,

There are a few options that we are using not listed in the help. Any plan on supporting them in the new command? Or the only way specifying them would be --passthrough?

--latest-toolchain

--nochromesdk

Thanks,

Ting-Yuan

[Don Garrett]

I'm happy to support options people really need, and hope to avoid any "normal" usage of --passthrough. 

Is there anything else you need other than those two?

--

[Ting-Yuan Huang]

'cros tryjob' doesn't seem to work with *-llvm-next-toolchain. By the way, it looks like that --passthrough should be the last argument but it is in the middle of help message.

laszio@laszio0:~/cros2$ cros tryjob --debug --remote chell-llvm-next-toolchain

...

13:03:35: ERROR: cros tryjob failed before completing.

cros: Unhandled exception:

Traceback (most recent call last):

  File "/ssd2/cros2/chromite/bin/cros", line 168, in <module>

    DoMain()

  File "/ssd2/cros2/chromite/bin/cros", line 164, in DoMain

    commandline.ScriptWrapperMain(FindTarget)

  File "/ssd2/cros2/chromite/lib/commandline.py", line 860, in ScriptWrapperMain

    ret = target(argv[1:])

  File "/ssd2/cros2/chromite/scripts/cros.py", line 65, in main

    code = _RunSubCommand(subcommand)

  File "/ssd2/cros2/chromite/scripts/cros.py", line 50, in _RunSubCommand

    return subcommand.Run()

  File "/ssd2/cros2/chromite/cli/cros/cros_tryjob.py", line 253, in Run

    return RunRemote(self.options, patch_pool)

  File "/ssd2/cros2/chromite/cli/cros/cros_tryjob.py", line 111, in RunRemote

    tryjob.Submit(dryrun=False)

  File "/ssd2/cros2/chromite/cbuildbot/remote_try.py", line 348, in Submit

    self._Submit(ref.path, testjob, dryrun)

  File "/ssd2/cros2/chromite/cbuildbot/remote_try.py", line 225, in _Submit

    self._PostConfigsToBuildBucket(testjob, dryrun)

  File "/ssd2/cros2/chromite/cbuildbot/remote_try.py", line 288, in _PostConfigsToBuildBucket

    self._PutConfigToBuildBucket(buildbucket_client, bot, dryrun)

  File "/ssd2/cros2/chromite/cbuildbot/remote_try.py", line 261, in _PutConfigToBuildBucket

    'builder_name': self._GetBuilder(bot),

  File "/ssd2/cros2/chromite/cbuildbot/remote_try.py", line 234, in _GetBuilder

    if site_config[bot] and site_config[bot]['_template']:

KeyError: 'chell-llvm-next-toolchain'

[Mike Frysinger]

> By the way, it looks like that --passthrough should be the last argument but it is in the middle of help message

hmm i'm not sure --passthrough's nargs=REMAINDER plays well with build_configs nargs=+.  maybe we should change --passthrough to be action=split_extend ?  in which case someone should +2 my CL :).

  https://chromium-review.googlesource.com/612821

-mike

--

[Don Garrett]

There is no chell-llvm-nexttoolchain.

However, I'm not checking for valid configs and have no way to list available configs, which I'd intended to support.

These are the only next-toochain configs.

amd64-llvm-next-toolchain                     Full release build with LLVM (next) toolchain

arm-llvm-next-toolchain                       Full release build with LLVM (next) toolchain

arm64-llvm-next-toolchain                     Full release build with LLVM (next) toolchain

caroline-llvm-next-toolchain                  Full release build with LLVM (next) toolchain

daisy-llvm-next-toolchain                     Full release build with LLVM (next) toolchain

eve-llvm-next-toolchain                       Full release build with LLVM (next) toolchain

gale-llvm-next-toolchain                      Full release build with LLVM (next) toolchain

kevin-llvm-next-toolchain                     Full release build with LLVM (next) toolchain

lakitu-llvm-next-toolchain                    Full release build with LLVM (next) toolchain

link-llvm-next-toolchain                      Full release build with LLVM (next) toolchain

lumpy-llvm-next-toolchain                     Full release build with LLVM (next) toolchain

nyan_big-llvm-next-toolchain                  Full release build with LLVM (next) toolchain

peach_pit-llvm-next-toolchain                 Full release build with LLVM (next) toolchain

peppy-llvm-next-toolchain                     Full release build with LLVM (next) toolchain

reef-llvm-next-toolchain                      Full release build with LLVM (next) toolchain

sentry-llvm-next-toolchain                    Full release build with LLVM (next) toolchain

squawks-llvm-next-toolchain                   Full release build with LLVM (next) toolchain

terra-llvm-next-toolchain                     Full release build with LLVM (next) toolchain

whirlwind-llvm-next-toolchain                 Full release build with LLVM (next) toolchain

PS: --remote isn't needed, it's the default.

--

[Ting-Yuan Huang]

Thanks for the clarification.

and +1 for --notests. We use it everyday.

[Don Garrett]

I have this CL up now to revise the command line options based on discussion here.

Vapier has asked that the behavior of --passthrough be updated, but I want to hold off until the new options are in place and the toolchain team can adjust away from it's current usage.

https://chromium-review.googlesource.com/c/618510

usage: cros tryjob [-h]

                   [--log-level {fatal,critical,error,warning,notice,info,debug}]

                   [--log_format LOG_FORMAT] [--debug] [--nocolor] [-b BRANCH]

                   [--yes] [--production] [--passthrough ...]

                   [--local | --remote] [-r BUILDROOT] [-g GERRIT_PATCHES]

                   [-p LOCAL_PATCHES] [--committer-email COMMITTER_EMAIL]

                   [--remote-description REMOTE_DESCRIPTION]

                   [--latest-toolchain] [--nochromesdk] [--hwtest] [--notests]

                   [--novmtests] [--noimagetests] [--version PASSTHROUGH]

                   [--channel PASSTHROUGH]

                   build_configs [build_configs ...]

Schedule a tryjob.

positional arguments:

  build_configs         One or more configs to build.

optional arguments:

  -h, --help            show this help message and exit

  -b BRANCH, --branch BRANCH

                        The manifest branch to test. The branch to check the

                        buildroot out to.

  --yes                 Never prompt to confirm.

  --production          This is a production build, NOT a test build. Confirm

                        with Chrome OS deputy before use.

  --passthrough ...     Arguments to pass to cbuildbot. To be avoided.Confirm

                        with Chrome OS deputy before use.

Debug options:

  --log-level {fatal,critical,error,warning,notice,info,debug}

                        Set logging level to report at.

  --log_format LOG_FORMAT

                        Set logging format to use.

  --debug               Alias for `--log-level=debug`. Useful for debugging

                        bugs/failures.

  --nocolor             Do not use colorized output (or `export NOCOLOR=true`)

Where:

  Where do we run the tryjob?

  --local               Run the tryjob on your local machine.

  --remote              Run the tryjob on a remote builder. (default)

  -r BUILDROOT, --buildroot BUILDROOT

                        Root directory to use for the local tryjob. NOT the

                        current checkout.

Patch:

  Which patches should be included with the tryjob?

  -g GERRIT_PATCHES, --gerrit-patches GERRIT_PATCHES

                        Space-separated list of short-form Gerrit Change-Id's

                        or change numbers to patch. Please prepend '*' to

                        internal Change-Id's

  -p LOCAL_PATCHES, --local-patches LOCAL_PATCHES

                        Space-separated list of project branches with patches

                        to apply. Projects are specified by name. If no branch

                        is specified the current branch of the project will be

                        used.

Requestor:

  Who is submitting the jobs?

  --committer-email COMMITTER_EMAIL

                        Override default git committer email.

  --remote-description REMOTE_DESCRIPTION

                        Attach an optional description to a --remote run to

                        make it easier to identify the results when it

                        finishes

Modifiers:

  How do we change the build configuration?

  --latest-toolchain    Use the latest toolchain.

  --nochromesdk         Don't run the ChromeSDK stage which builds Chrome

                        outside of the chroot.

Testing Flags:

  How do we change testing behavior?

  --hwtest              Enable hwlab testing. Default false.

  --notests             Override values from buildconfig, run no tests, and

                        build no autotest artifacts.

  --novmtests           Override values from buildconfig, run no vmtests.

  --noimagetests        Override values from buildconfig and run no image

                        tests.

Payloads:

  Options only used by payloads tryjobs.

  --version PASSTHROUGH

                        Specify the release version for payload regeneration.

                        Ex: 9799.0.0

  --channel PASSTHROUGH

                        Specify a channel for a payloads trybot. Can be

                        specified multiple times. No valid for non-payloads

                        configs.

Remote Examples:

  cros tryjob -g 123 lumpy-compile-only-pre-cq

  cros tryjob -g 123 -g 456 lumpy-compile-only-pre-cq daisy-pre-cq

  cros tryjob -g 123 --hwtest daisy-paladin

Local Examples:

  cros tryjob --local -g 123 daisy-paladin

  cros tryjob --local --buildroot /my/cool/path -g 123 daisy-paladin

Production Examples (danger, can break production if misused):

  cros tryjob --production --branch release-R61-9765.B asuka-release

  cros tryjob --production --version 9795.0.0 --channel canary  lumpy-payloads

[Don Garrett]

The new commandline options are available.

[jcli...@chromium.org]

Hi Don,

There are a lot of stale references to cbuildbot try jobs on chromium.org; would you mind updating?

https://www.google.com/search?q=site%3Awww.chromium.org+%22cros+tryjob%22&oq=site%3Awww.chromium.org+%22cros+tryjob%22&aqs=chrome..69i57j69i58.9123j0j7&sourceid=chrome&ie=UTF-8

Thank you,

[Edward Jee]

Hi, I've experienced that -p option doesn't work if I don't specify the branch of a project, although the help message says "If no branch is specified the current branch of the project will be used".

Also it's not clear from the help message that the argument format is <project name>:<branch name>.

[Don Garrett]

On Thu, Sep 21, 2017 at 3:51 PM 'Edward Jee' via Chromium OS dev <chromiu...@chromium.org> wrote:

Hi, I've experienced that -p option doesn't work if I don't specify the branch of a project, although the help message says "If no branch is specified the current branch of the project will be used".

Also it's not clear from the help message that the argument format is <project name>:<branch name>.

I forgot to reply here, but there was a bug and the behavior should be fixed. Feel free to send a CL with a better argument help message. I'm not very good at writing those.

[Junichi Uekawa (上川純一)]

Hi,

I've just sent out my first tryjob with cros tryjob.

My feedback: 

This command should give me a better error message and point to a useful documentation

$ cbuildbot --remote -g 694404 kevin-pre-cq --hwtest

13:22:43: ERROR: Deprecated usage. Please use cros tryjob instead.

If the supported command-line options are going to be the same, it should say so ...

$  cros tryjob --remote -g 694404 --hwtest kevin-pre-cq

if they are different the documentation should tell me.

--
--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en

---

You received this message because you are subscribed to the Google Groups "Chromium OS dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-os-dev+unsubscribe@chromium.org.

[Ting-Yuan Huang]

May I know how to specify a patch for chromium? IIRC there was a -G but Rietveld is deprecated. I uploaded a chromium patch to Gerrit and tried to specify it with -g but failed in the PatchChanges stage.

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/chrome_pfq/builds/196

[Mike Frysinger]

i'm fairly certain we never updated things once Chromium moved from rietveld to GoB.  we dropped the rietveld logic entirely.

this applies both to cbuildbot & cros tryjob ... inability to patch in browser changes is unrelated to this migration.

i've filed https://crbug.com/776956 to track.

-mike