RFC: migrating infra builders to kitchen_run/remote_run

[Paweł Hajdan, Jr.]

I'm planning to migrate infra builders to kitchen_run/remote_run, as part of effort to test things before migrating chromium and recipes-in-repos. This will also make it easier to move to swarming later (need cipd and logdog on swarming first).

From end-user POV, this change should be no-op.

You can see a builder using this at https://build.chromium.org/p/chromium.fyi/builders/Linux%20Kitchen%20%28kitchen_run%29 . As you can see, it uses logdog and looks pretty much the same as any other builder. Can you actually see the differences? ;-)

The candidates for migration are:

https://build.chromium.org/p/chromium.infra/builders

https://build.chromium.org/p/chromium.infra.cron/builders

https://build.chromium.org/p/tryserver.infra/builders

I plan to start with https://build.chromium.org/p/chromium.infra.cron/builders/publish_tarball tomorrow because it's only used by me and open source distro packagers.

I'd aim for migrating some next batches over the next week.

Please let me know if you have any questions or concerns. This is mostly a heads-up to ensure we have a consensus here.

Paweł

[Erik Staab]

LGTM, thanks!

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAATLsPb%2BTao7jfKxm81wUaA7%3DTYOrVAWeAmg89Ngs2yTKUF-MA%40mail.gmail.com.

[Paweł Hajdan, Jr.]

Awesome!

I've successfully moved publish_tarball - see https://build.chromium.org/p/chromium.infra.cron/builders/publish_tarball/builds/3626 .

I've realized the recipe behaves slightly differently when using "kitchen" path_config, see e.g. https://build.chromium.org/p/chromium.infra.cron/builders/publish_tarball/builds/3625 which I had to address with https://codereview.chromium.org/2035863004 . I suggest we add path_config=kitchen as a property to all recipes moved to remote_run. WDYT?

Please also see https://groups.google.com/a/chromium.org/d/msg/infra-dev/tW99wuWVSWQ/XZhPVCGbBAAJ for possible further steps moving infra recipes to infra repos.

Paweł

[Paweł Hajdan, Jr.]

I plan to move master.chromium.infra.cron tomorrow - please speak up if there are any concerns or questions.

Today I've moved almost entire master.chromium.infra to remote_run. First I ensured recipes would continue to work with kitchen path_config (https://codereview.chromium.org/2045493002) and then landed a master change (https://codereview.chromium.org/2043683002).

There was an unexpected breakage and I had to convert infra-continuous-win-64 back (https://codereview.chromium.org/2037423002) - just that one builder. Please see https://build.chromium.org/p/chromium.infra/builders/infra-continuous-win-64/builds/2124 for the failure - any insights would be appreciated.

Encountered error on uploading E:\b\build\slave\infra-continuous\build\.remote_run\tmpxrgo0g\run_workdir\infra\go\bin\authutil.exe to gs://chromium-luci/291989336577e45440cbd327bcf75daef17d20c5

Traceback (most recent call last):

  File "E:\b\build\slave\infra-continuous\build\.remote_run\tmpxrgo0g\remote_run_workdir\checkout\scripts\slave\.recipe_deps\depot_tools\gsutil.py", line 160, in <module>

    sys.exit(main())

  File "E:\b\build\slave\infra-continuous\build\.remote_run\tmpxrgo0g\remote_run_workdir\checkout\scripts\slave\.recipe_deps\depot_tools\gsutil.py", line 157, in main

    clean=args.clean)

  File "E:\b\build\slave\infra-continuous\build\.remote_run\tmpxrgo0g\remote_run_workdir\checkout\scripts\slave\.recipe_deps\depot_tools\gsutil.py", line 125, in run_gsutil

    gsutil_bin = ensure_gsutil(force_version, target, clean)

  File "E:\b\build\slave\infra-continuous\build\.remote_run\tmpxrgo0g\remote_run_workdir\checkout\scripts\slave\.recipe_deps\depot_tools\gsutil.py", line 109, in ensure_gsutil

    target_zip.extractall(download_dir)

  File "E:\b\depot_tools\python276_bin\lib\zipfile.py", line 1036, in extractall

    self.extract(zipinfo, path, pwd)

  File "E:\b\depot_tools\python276_bin\lib\zipfile.py", line 1024, in extract

    return self._extract_member(member, path, pwd)

  File "E:\b\depot_tools\python276_bin\lib\zipfile.py", line 1079, in _extract_member

    file(targetpath, "wb") as target:

IOError: [Errno 2] No such file or directory: 'E:\\b\\build\\slave\\infra-continuous\\build\\.remote_run\\tmpxrgo0g\\remote_run_workdir\\checkout\\scripts\\slave\\.recipe_deps\\depot_tools\\external_bin\\gsutil\\gsutil_pyfgpwuf\\download\\gsutil\\third_party\\gcs-oauth2-boto-plugin\\gcs_oauth2_boto_plugin\\test_oauth2_client.py'

Paweł