RFC: moving infra recipes to infra repo

[Paweł Hajdan, Jr.]

Related to the effort moving builders to remote_run (https://groups.google.com/a/chromium.org/d/msg/infra-dev/pt70Gth16KM/vn6oNolSBAAJ), I'm also planning to move recipes to the infra repo. The former is a prerequisite, i.e. annotated_run which most of the builders use can only use recipes in the build repo, while remote_run can reference any repo.

I'd like to ask for your input. Some more specific questions:

1. Do you have any concerns about moving forward? I'd like to start with infra/publish_tarball which is only used by me and open source distros, so impact on Google and rest of Chrome should be minimal. Once that's ready, we can evaluate next steps.

2. Where in the infra repo do we want to add recipes? I could add recipes, recipe_modules and recipes.py directly under repo root. If you'd prefer a different directory, I'm also fine with that - let me know your suggestions. Please note that putting them e.g. under infra/infra might interfere with test.py's coverage - I haven't tried that yet. It might be best to go for simplicity and use the repo root.

3. Which places need to be updated to register the infra repo? I know about public recipe autoroller, and likely downstream recipe tryjobs. Is there anything else?

Paweł

[Erik Staab]

replies inline

On Fri, Jun 3, 2016 at 7:24 AM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

Related to the effort moving builders to remote_run (https://groups.google.com/a/chromium.org/d/msg/infra-dev/pt70Gth16KM/vn6oNolSBAAJ), I'm also planning to move recipes to the infra repo. The former is a prerequisite, i.e. annotated_run which most of the builders use can only use recipes in the build repo, while remote_run can reference any repo.

I'd like to ask for your input. Some more specific questions:

1. Do you have any concerns about moving forward? I'd like to start with infra/publish_tarball which is only used by me and open source distros, so impact on Google and rest of Chrome should be minimal. Once that's ready, we can evaluate next steps.

sgtm

2. Where in the infra repo do we want to add recipes? I could add recipes, recipe_modules and recipes.py directly under repo root. If you'd prefer a different directory, I'm also fine with that - let me know your suggestions. Please note that putting them e.g. under infra/infra might interfere with test.py's coverage - I haven't tried that yet. It might be best to go for simplicity and use the repo root.

I've always felt it odd that recipes get three top-level files - could we have everything under a top-level "recipes"? This would be more in line with the direction we're going with fetching subdirectories of repos using gitiles tarball API. We could then have:

recipes/

  recipes.py

  recipes/

  recipe_modules/

  <whatever supporting files are needed with gitiles tarball fetches>

 

3. Which places need to be updated to register the infra repo? I know about public recipe autoroller, and likely downstream recipe tryjobs. Is there anything else?

[+cc martiniss]

Erik

Paweł

[Robert Iannucci]

This sgtm too.

I agree with Erik that there should be some folder which contains recipes (and only recipes). On that note, we should restructure expectations so they live in a parallel directory structure. So something like:

recipes/

  recipes.py

  recipes/

    foo.py

  recipe_modules/

    foo/

      api.py

      example.py

recipes.expected/

  recipes/

    foo/

      test1.json

  recipe_modules/

    foo/

      example_test.json

This can happen later, but we should keep it in mind. Fetching all the json files is a non-trivial amount of data to fetch and ignore, especially in repos like chromium.

R

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAN60ya%2B0ZN%2BM%2BmJ9c42g6U_mRsu5tvZ9oFSPu79XM1iEofn8jQ%40mail.gmail.com.

[Dirk Pranke]

On Fri, Jun 3, 2016 at 7:24 AM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

Related to the effort moving builders to remote_run (https://groups.google.com/a/chromium.org/d/msg/infra-dev/pt70Gth16KM/vn6oNolSBAAJ), I'm also planning to move recipes to the infra repo. The former is a prerequisite, i.e. annotated_run which most of the builders use can only use recipes in the build repo, while remote_run can reference any repo.

Are you talking about moving just infra-specific recipes from build to infra, or all recipes (e.g., chromium.py)?

--  Dirk

I'd like to ask for your input. Some more specific questions:

1. Do you have any concerns about moving forward? I'd like to start with infra/publish_tarball which is only used by me and open source distros, so impact on Google and rest of Chrome should be minimal. Once that's ready, we can evaluate next steps.

2. Where in the infra repo do we want to add recipes? I could add recipes, recipe_modules and recipes.py directly under repo root. If you'd prefer a different directory, I'm also fine with that - let me know your suggestions. Please note that putting them e.g. under infra/infra might interfere with test.py's coverage - I haven't tried that yet. It might be best to go for simplicity and use the repo root.

3. Which places need to be updated to register the infra repo? I know about public recipe autoroller, and likely downstream recipe tryjobs. Is there anything else?

Paweł

--

You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAATLsPYY7OkuSQnVivW3PYezUtKLSjYWrnnk4H0dW5RttKxZFA%40mail.gmail.com.

[Erik Staab]

On Fri, Jun 3, 2016 at 1:39 PM, Dirk Pranke <dpr...@chromium.org> wrote:

On Fri, Jun 3, 2016 at 7:24 AM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

Related to the effort moving builders to remote_run (https://groups.google.com/a/chromium.org/d/msg/infra-dev/pt70Gth16KM/vn6oNolSBAAJ), I'm also planning to move recipes to the infra repo. The former is a prerequisite, i.e. annotated_run which most of the builders use can only use recipes in the build repo, while remote_run can reference any repo.

Are you talking about moving just infra-specific recipes from build to infra, or all recipes (e.g., chromium.py)?

Just infra-specific recipes, e.g. recipes running on chromium.infra and tryserver.infra. Chromium recipes will go to the chromium repo in a later step.

Erik

 

--  Dirk

I'd like to ask for your input. Some more specific questions:

1. Do you have any concerns about moving forward? I'd like to start with infra/publish_tarball which is only used by me and open source distros, so impact on Google and rest of Chrome should be minimal. Once that's ready, we can evaluate next steps.

2. Where in the infra repo do we want to add recipes? I could add recipes, recipe_modules and recipes.py directly under repo root. If you'd prefer a different directory, I'm also fine with that - let me know your suggestions. Please note that putting them e.g. under infra/infra might interfere with test.py's coverage - I haven't tried that yet. It might be best to go for simplicity and use the repo root.

3. Which places need to be updated to register the infra repo? I know about public recipe autoroller, and likely downstream recipe tryjobs. Is there anything else?

Paweł

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAATLsPYY7OkuSQnVivW3PYezUtKLSjYWrnnk4H0dW5RttKxZFA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAEoffTApq-nyLv-i09Q0gGZMdOBjfmJsTMYtF5MHUyA4LBOF%3Dg%40mail.gmail.com.

[Stephen Martinis]

On Fri, Jun 3, 2016 at 8:53 AM Erik Staab <est...@chromium.org> wrote:

replies inline

On Fri, Jun 3, 2016 at 7:24 AM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

Related to the effort moving builders to remote_run (https://groups.google.com/a/chromium.org/d/msg/infra-dev/pt70Gth16KM/vn6oNolSBAAJ), I'm also planning to move recipes to the infra repo. The former is a prerequisite, i.e. annotated_run which most of the builders use can only use recipes in the build repo, while remote_run can reference any repo.

I'd like to ask for your input. Some more specific questions:

1. Do you have any concerns about moving forward? I'd like to start with infra/publish_tarball which is only used by me and open source distros, so impact on Google and rest of Chrome should be minimal. Once that's ready, we can evaluate next steps.

sgtm

sgtm 

2. Where in the infra repo do we want to add recipes? I could add recipes, recipe_modules and recipes.py directly under repo root. If you'd prefer a different directory, I'm also fine with that - let me know your suggestions. Please note that putting them e.g. under infra/infra might interfere with test.py's coverage - I haven't tried that yet. It might be best to go for simplicity and use the repo root.

I've always felt it odd that recipes get three top-level files - could we have everything under a top-level "recipes"? This would be more in line with the direction we're going with fetching subdirectories of repos using gitiles tarball API. We could then have:

recipes/

  recipes.py

  recipes/

  recipe_modules/

  <whatever supporting files are needed with gitiles tarball fetches>

I would agree with this. But I think the recipes/recipes/chromium.py is a bit strange.... not sure what a good alternative would be though.  

 

3. Which places need to be updated to register the infra repo? I know about public recipe autoroller, and likely downstream recipe tryjobs. Is there anything else?

[+cc martiniss]

Theoretically it should "just work" if it's in luci-config. But yes, right now there's a whitelist in the autoroller recipe module which we would need to update.

Erik

Paweł

[Paweł Hajdan, Jr.]

+agable,nodir

I realized we have more issues here than cosmetic naming.

infra repo is a bit special, and we store configs in config/ instead of infra/config/ . Most likely that's because infra/ contains python code in infra namespace.

IMO it'd be great not to have that special case. Even though it seems possible to get info about it from luci-config, it'd be much simpler to have all systems behave in a consistent way. FWIW, the path I found is https://apis-explorer.appspot.com/apis-explorer/?base=https://luci-config.appspot.com/_ah/api#p/config/v1/config.get_config_sets?_h=2& and then https://chromium.googlesource.com/infra/infra/+/refs/heads/infra/config/refs.cfg . Aaron/Nodir, could you share more context about https://codereview.chromium.org/1183693003 (move the luci-cfg config directory out of the infra/ python package)?

Actually there might be an easy way to do this, which I found in https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/recipe_tryjob/api.py?q=recipes.cfg&sq=package:chromium&l=160&dr=C :

result = self.m.luci_config.get_project_config(project, 'recipes.cfg')

There are several places which assume infra/config/recipes.cfg :

https://cs.chromium.org/chromium/infra/recipes-py/recipe_engine/package.py?q=recipes.cfg&sq=package:chromium&dr=CSs&l=50 :

class InfraRepoConfig(object):

  def to_recipes_cfg(self, repo_root):

    # TODO(luqui): This is not always correct.  It can be configured in

    # infra/config:refs.cfg.

    return os.path.join(repo_root, 'infra', 'config', 'recipes.cfg')

  def from_recipes_cfg(self, recipes_cfg):

    return os.path.dirname( # <repo root>

            os.path.dirname( # infra

              os.path.dirname( # config

                os.path.abspath(recipes_cfg)))) # recipes.cfg

https://cs.chromium.org/chromium/infra/recipes-py/recipe_engine/fetch.py?q=recipes.cfg&sq=package:chromium&dr=C&l=96 :

def ensure_gitiles_checkout(repo, revision, checkout_dir, allow_fetch):

  ...

  recipes_cfg_url = '%s/+/%s/infra/config/recipes.cfg?format=TEXT' % (

      repo, urllib.quote(revision))

https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/recipe_autoroller/api.py?q=recipes.cfg&sq=package:chromium&dr=C&l=167 :

recipes_cfg_path = workdir.join('infra', 'config', 'recipes.cfg')

Note that for from_recipes_cfg code above, luci-config might not be viable - e.g. if we only have code without URL of the repo. One way to handle this could be to change recipes_path in recipes.cfg to be relative to recipes.cfg (which we know where it is), as opposed to repo root (which we might not know). There are several ways to handle that, e.g. just make a change in recipe engine and fix downstream to make rolls succeed; or add a new field name deprecating recipes_path; or bump api_version, and keep compatibility code in recipe engine until all repos migrate to it.

WDYT?

Paweł

[Robert Iannucci]

Homogeneity is good, but I don't think it's something we should assume is available in every repo we support. The whole reason we have luci-config is so that we have one homogenous API for configuration, even though we can't assume homogenous repo layouts. I would much rather update the hard coded locations to use the homogenous API that we built specifically for this purpose than to paper over it by making infra conform to the hard coded path assumptions.

The fact that we still have a separate list of packages in the roller recipe is also a historical bug, but has the same solution.

If there's deficiencies that we need to remedy to make luci-config's api work offline (like when we have code but no URL) we should solve that problem (e.g. a consistent way to bundle the luci-config data statically with the e.g. the isolated code). Since luci-config is already a virtual filesystem (but with a weird API), we could easily have an api call to retrieve a zip'd snapshot of config as a folder structure, which we could bundle with the isolated code. However, I don't think we need to solve that problem right away.

R

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAATLsPbSKDtJM0GmaCUKfASFpfTBe3cgh%3DyYe793GAeL99nvPg%40mail.gmail.com.

[Nodir Turakulov]

FWIU path to recipes.cfg is needed during deps resolution. Currently recipes.cfg lists all of its deps via absolute location (url, branch/rev). Adding luci-config the way we are discussing will make this location reference non-absolute. In other words it will add an implicit variable stored on a separate service. This is NOT simple.

What do you think about adding recipe.cfg path variable to recipes.cfg which defaults to "infra/config/recipes.cfg"?

deps {

  project_id: "infra"

  url: "https://chromium.googlesource.com/infra/infra.git"

  config: "config/recipes.cfg"

  branch: "master"

  revision: "a2f99b57a04aea317bc02bb10ab570e1dc337218"

}

We get the same gain "no hardcoded path" and at the same time

- no luci-config dependency

- no problem with authentication (some projects are internal and recipes-py have to provide creds to luci-config)

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CA%2Bq_oBdLkT6pR3hFGfGK%2BUY%3DLMwmRFmQ0c4W-sAaWX7X%3DqzHKQ%40mail.gmail.com.

[Nodir Turakulov]

I've forked this thread with new subject "hardcoded infra/config dir of recipes.cfg" because "RFC: moving infra  recipes to infra repo" is too broad for this topic.

On Mon, Jun 6, 2016 at 9:25 AM Robert Iannucci <iann...@chromium.org> wrote:

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CA%2Bq_oBdLkT6pR3hFGfGK%2BUY%3DLMwmRFmQ0c4W-sAaWX7X%3DqzHKQ%40mail.gmail.com.

[Nodir Turakulov]

This may look like data duplication, but repo URLs are already duplicated, and it is a bigger (but not big) problem than duplicating recipes.cfg path

[Robert Iannucci]

The repo URLs are already a bug (for the same historical reason that the hard coded paths and project lists are bugs). It should just be using the luci-config ID to get all the information it needs.

Afaik, we have two modes, online and offline. Online is e.g. remote_run or remote_bundle. These modes can talk to luci-config. The offline modes (e.g. running a bundle) should have any luci-config data captured for them at the time the online mode was run.

Right?

[Stephen Martinis]

I agree with Robbie. I think we should teach the recipe engine about luci-config at some level. I was planning on working on this next.

I'm a bit hesitant about putting the config path in the recipes.cfg file; it means if they ever move, it's one more place to move.

Why can't we just put the luci-config URL inside of the dependency?

I think it'd be fair to make the online mode robbie mentioned depend on luci-config.

[Nodir Turakulov]

if luci-config URL is explicitly specified in deps, then it resolves my original concern - implicit dependency on luci-config

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CANheWB%3DzyvZ-dJCSC1q0u2b4tqXkqm2Fo5tvt9LCR2nBTbmhrQ%40mail.gmail.com.

[Robert Iannucci]

Yes URL for luci-config service in .cfg sgtm

[Paweł Hajdan, Jr.]

How would we handle overriding recipe directories, i.e. recipes.py -O depot_tools=~/depot_tools? I use that very frequently when developing cross-repo changes. Specifically, how would the recipe engine know where to look for in the overridden depot_tools location?

I'd be fine with putting path to deps recipes.cfg in downstream recipes.cfg as suggested in the original post. It's probably important to ensure it's the same in the entire dependency tree just like we ensure the same revision of each repo.

For luci-config - that's probably also fine. Could you please provide "pseudo-code" for how to resolve location of recipes.cfg for any repo? Ideally point to specific API calls in luci-config. I may be very interested in working on this if we decide to go this way, since it seems to be on the critical path to getting further recipes-in-repos work done.

I'd still like us to consider requiring infra/config/recipes.cfg everywhere. I wonder how an evaluation of pros and cons of each of these alternatives could look like.

Paweł

[Paweł Hajdan, Jr.]

I'd still like to understand why infra repo can't have the file under infra/config/recipes.cfg . It sounds like the simplest thing that would work.

Moving recipes into repos allows us to do significant cleanups of chromium_tests infrastructure, and also various historical hacks or things that were considered too risky and untestable. Do benefits of the longer way here outweigh just using infra/config/recipes.cfg for the infra repo? I'm wondering if we can work on the nicer things in parallel.

I think I'm actually going to try adding infra/config/recipes.cfg . Please let me know if you have any questions/suggestions/concerns about that - provided it'd pass the trybots.

Paweł

[Nodir Turakulov]

pawel, your message has old subject, so it ends up in the old thread in my Inbox, not in the fork

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAATLsPbc%2B1bS%3DtKwjyM%3DNKKUuxLSc8EZ-2XFzsn-ELGQMfK9pw%40mail.gmail.com.

[Stephen Martinis]

I'm fine with short term infra/config hard-coding everywhere. It solves our immediate problems with no required change.

We would need to change the recipe tryjob to not use luci-config, if we hardcode. 

I think long term we should standardize on luci-config. This allows projects to change URL and config location easily. It also allows any appengine services to access recipes.cfg information, which is the problem luci-config originally solved.

On Mon, Jun 6, 2016 at 1:32 PM Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

How would we handle overriding recipe directories, i.e. recipes.py -O depot_tools=~/depot_tools? I use that very frequently when developing cross-repo changes. Specifically, how would the recipe engine know where to look for in the overridden depot_tools location?

With luci-config, you would cache this location (from luci-config, by running it online), and then use that information when offline. Or something like that; the information is in luci-config, you just have to figure out how to get it.

I'd be fine with putting path to deps recipes.cfg in downstream recipes.cfg as suggested in the original post. It's probably important to ensure it's the same in the entire dependency tree just like we ensure the same revision of each repo.

I'd prefer not to do this, since that means that if it ever changes, you have to update all dependencies on your repo, semi-atomically, which you can't do for multiple git commits.  

For luci-config - that's probably also fine. Could you please provide "pseudo-code" for how to resolve location of recipes.cfg for any repo? Ideally point to specific API calls in luci-config. I may be very interested in working on this if we decide to go this way, since it seems to be on the critical path to getting further recipes-in-repos work done.

https://apis-explorer.appspot.com/apis-explorer/?base=https://luci-config.appspot.com/_ah/api#p/config/v1/config.get_project_configs?path=refs.cfg&_h=1&

is the luci-config call to get the refs.cfg file, which gives you the config location for all projects. https://apis-explorer.appspot.com/apis-explorer/?base=https://luci-config.appspot.com/_ah/api#p/config/v1/config.get_config?config_set=projects%252Fbuild%252Frefs%252Fheads%252Fmaster&path=recipes.cfg&_h=2& is the API call to get you the recipes.cfg for build. 

You can just curl those two URLs (not the api explorer, but the actual api endpoints).

I don't think it's currently blocking though.

 

I'd still like us to consider requiring infra/config/recipes.cfg everywhere. I wonder how an evaluation of pros and cons of each of these alternatives could look like.

There are currently 4 repos which have config files not at infra/config path. See this query.

I didn't get a chance to write up a document about these pros and cons, unfortunately. 

[Erik Staab]

I've spoken to people offline but I should chime in here as well. luci-config integration sounds like the right mid-term approach and hardcoding the path for now to unblock sounds fine. I'd suggest adding a TODO with a reference to a bug to do the conversion to luci-config. If we can follow the progress of the work in the bug we can put a bound on the technical debt we're introducing.

[Paweł Hajdan, Jr.]

Nodir, my last message was intended for this thread.

Everyone: I'm excited to announce with https://codereview.chromium.org/2041313002 we now have recipe code in the infra repo. https://codereview.chromium.org/2042243002 and https://codereview.chromium.org/2045933002 add relevant tests to infra tryserver. https://codereview.chromium.org/2048653002 adds infra repo to the recipe roller. https://codereview.chromium.org/2045623004 switched the buildbot master.

I've tested all affected bots on chromium.infra, either with a whitespace CL or forcing re-run of their last build (for builders that trigger from repos other than infra). All of the builds are green.

I intend to migrate remaining masters using the recipes that are now in infra repo (e.g. tryserver, internal infra master) and delete the copy of recipes in the build repo.

Any feedback would be very welcome.

Paweł

[Erik Staab]

This is awesome! Thanks for making all of the changes to get them moved over and working.

Maybe we should add a restrictive OWNERS file in build/scripts/slave/recipes/infra/ with a note to edit the files in infra.git instead until the old files are deleted? We're probably ok if it's just a few days.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAATLsPYry2eCZK%3DCymEPFK%2BZUZ52SF0VMWpghQ18LpyoRJ8HnA%40mail.gmail.com.

[Stephen Martinis]

Great, that's awesome! Thanks for all that work; are you planning on moving the rest of the recipes over? I'd be glad to help with some of it.

I've uploaded https://codereview.chromium.org/2046453004 per estaab@'s suggestion.

[Paweł Hajdan, Jr.]

Thanks, folks!

Today I switched remaining builders to the recipes now in infra repo and deleted the build copy (https://codereview.chromium.org/2047263003), except infra_continuous which is still used by master.chromium.infra:infra-continuous-win-64 . I had to convert that one back to annotated_run (https://codereview.chromium.org/2037423002) because of a build breakage: https://build.chromium.org/p/chromium.infra/builders/infra-continuous-win-64/builds/2124 . This is somewhat mysterious to me, and it might work on a retry. I didn't hit this issue with internal waterfalls.

I also moved remote_run's workdir to tempfile.gettempdir() - https://codereview.chromium.org/2049053002 - to avoid path too long issues on at least Windows and Linux. See https://goto.google.com/guoqq and https://goto.google.com/nssym for the failing builds (Google-only).

With https://codereview.chromium.org/2050683002 and https://codereview.chromium.org/2044423002 I switched master.tryserver.infra to remote_run and recipes in infra. See https://goto.google.com/jfswe for corresponding internal change.

Let me know what you think about above changes.

I was planning to work on remaining recipes and waterfalls (e.g. chromium.infra.cron), but I'd actually be happy to get some help with that. Ideally I'd re-focus on the chromium cases - please share your input on https://groups.google.com/a/chromium.org/d/msg/infra-dev/1o2Vr-C3q3Y/BJzioXyQBQAJ .

Paweł

[Stephen Martinis]

I believe the infra-continuous-win-64 failure is also because the path is too long. I saw a Traceback [1] which had a path > 260 chars in it.

Other CLs look ok. I think we should move configs from at the root level in builders.pyl to each builder config. Change path_config (https://codereview.chromium.org/2044423002) to just be inline with each builder, since you can specify properties per builder, and the repository for remote_run (https://codereview.chromium.org/2050683002) to be configurable per builder; these are more longer term changes I can do tomorrow; was trooper today, so fairly busy.

I've replied on the other thread about plans for what to do going forward.

[1]

Traceback (most recent call last):

  File "E:\b\build\slave\infra-continuous\build\.remote_run\tmpxrgo0g\remote_run_workdir\checkout\scripts\slave\.recipe_deps\depot_tools\gsutil.py", line 160, in <module>

    sys.exit(main())

  File "E:\b\build\slave\infra-continuous\build\.remote_run\tmpxrgo0g\remote_run_workdir\checkout\scripts\slave\.recipe_deps\depot_tools\gsutil.py", line 157, in main

    clean=args.clean)

  File "E:\b\build\slave\infra-continuous\build\.remote_run\tmpxrgo0g\remote_run_workdir\checkout\scripts\slave\.recipe_deps\depot_tools\gsutil.py", line 125, in run_gsutil

    gsutil_bin = ensure_gsutil(force_version, target, clean)

  File "E:\b\build\slave\infra-continuous\build\.remote_run\tmpxrgo0g\remote_run_workdir\checkout\scripts\slave\.recipe_deps\depot_tools\gsutil.py", line 109, in ensure_gsutil

    target_zip.extractall(download_dir)

  File "E:\b\depot_tools\python276_bin\lib\zipfile.py", line 1036, in extractall

    self.extract(zipinfo, path, pwd)

  File "E:\b\depot_tools\python276_bin\lib\zipfile.py", line 1024, in extract

    return self._extract_member(member, path, pwd)

  File "E:\b\depot_tools\python276_bin\lib\zipfile.py", line 1079, in _extract_member

    file(targetpath, "wb") as target:

IOError: [Errno 2] No such file or directory: 'E:\\b\\build\\slave\\infra-continuous\\build\\.remote_run\\tmpxrgo0g\\remote_run_workdir\\checkout\\scripts\\slave\\.recipe_deps\\depot_tools\\external_bin\\gsutil\\gsutil_pyfgpwuf\\download\\gsutil\\third_party\\gcs-oauth2-boto-plugin\\gcs_oauth2_boto_plugin\\test_oauth2_client.py'

[Paweł Hajdan, Jr.]

On Thu, Jun 9, 2016 at 3:03 AM, Stephen Martinis <mart...@chromium.org> wrote:

I believe the infra-continuous-win-64 failure is also because the path is too long. I saw a Traceback [1] which had a path > 260 chars in it.

Good point. I landed https://codereview.chromium.org/2052843002 to address this.

 

Other CLs look ok. I think we should move configs from at the root level in builders.pyl to each builder config. Change path_config (https://codereview.chromium.org/2044423002) to just be inline with each builder, since you can specify properties per builder, and the repository for remote_run (https://codereview.chromium.org/2050683002) to be configurable per builder; these are more longer term changes I can do tomorrow; was trooper today, so fairly busy.

Both suggestions are very reasonable, and I believe they're already implemented. Since currently all the builders use the same repo and all use path_config=kitchen, IMO it makes sense to set them only once, rather than duplicate for each builder.

I've also deleted infra_continuous from build (https://codereview.chromium.org/2051063002) since everything is moved to infra now.

I plan to be working more on remote_run and chromium parts, so consider rest of infra waterfalls (e.g. chromium.infra.cron) and recipes up for grabs - just let me know what you're working on.

Paweł