RFC: top-down approach to moving recipes to chromium repo

[Paweł Hajdan, Jr.]

I've been looking into next steps needed to start moving recipes to chromium repo.

Please consider this a call for relatively early feedback. Likely timeframe is about a few weeks; certainly not before resolving any outstanding questions/concerns, and probably starting with smaller repos that affect less people, e.g. infra.

I think I'd start with moving the relevant recipes first. kitchen_run/remote_run allows us to do this, even with buildbot. We'd need to add relevant repos to the recipe autoroller (trivial) and change buildbot master configs to use kitchen_run/remote_run factory. We'd also need to modify the factory so that it'd use the same recipe revision as requested build revision, if they refer to the same repo (e.g. chromium).

As a general rule, the other repos like chromium or infra would depend on build (via recipes.cfg), but not the other way around. Unfortunately there's still chromium-related code in build recipes that uses some build scripts there (e.g. compile.py) which makes it harder to migrate. As other recipes such as chromium would depend on build, the build repo can't depend back on them because it'd introduce a circular dependency.

Next step would likely be moving some recipe modules, starting from the top of the dependency tree, i.e. modules nothing else depends on. This requires that all recipes which depend on them move to relevant repos.

To give you a specific example, some starting ones could be auto_bisect, perf_try, ios, webrtc, chromium_tests. See attached dependency graph for reference. You can also generate one yourself by running "./recipes.py --package ~/infra_internal/build/infra/config/recipes.cfg depgraph --ignore-package recipe_engine | tred | dot -Tpdf > graph.pdf" from a recipes-py checkout.

If any internal recipes or modules depend on any of the moved modules, the relevant internal repos would start depending on the project repos (e.g. chromium) using a gitiles-based fetch strategy to avoid expensive pull of these large repos into .recipe_deps. I'm still working on supporting this in the recipe engine, including the autoroller.

WDYT?

Paweł

[Dirk Pranke]

This seems plausible to me but I'd like to resolve how we're going to reference the chromium recipe from other repos that depend on chromium

(i.e., how do changes to the chromium recipe get rolled into the other repos) before we can move it out of build.

Technically you can probably get away with not resolving this (by making recipe changes be compatible, which is roughly what we do today in

the build repo), but I fear that this will be harder and less intuitive once the code moves into the src repo. More importantly, it seems like a 

mistake to not resolve this before moving the recipe, because that defeats the point of recipes-in-repos.

-- Dirk

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAATLsPb_EVTpcmSxLsWQ0Go9OYm_9h_8-E57cm%3Do64Y%2Bh5ScEA%40mail.gmail.com.

[Erik Staab]

a few comments inline

On Wed, Jun 1, 2016 at 6:15 AM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

I've been looking into next steps needed to start moving recipes to chromium repo.

Please consider this a call for relatively early feedback. Likely timeframe is about a few weeks; certainly not before resolving any outstanding questions/concerns, and probably starting with smaller repos that affect less people, e.g. infra.

I think I'd start with moving the relevant recipes first. kitchen_run/remote_run allows us to do this, even with buildbot. We'd need to add relevant repos to the recipe autoroller (trivial) and change buildbot master configs to use kitchen_run/remote_run factory. We'd also need to modify the factory so that it'd use the same recipe revision as requested build revision, if they refer to the same repo (e.g. chromium).

Agree that we want to start with recipes, e.g. the leaf nodes. I'd like to include them in the graph that's generated from depgraph (probably merging nearly-identical ones) so we can see what can be picked off first.

As I've mentioned before we should definitely do this with the infra recipes first since we might discover something that affects our strategy with chromium. chromium == production and infra == staging.

 

As a general rule, the other repos like chromium or infra would depend on build (via recipes.cfg), but not the other way around. Unfortunately there's still chromium-related code in build recipes that uses some build scripts there (e.g. compile.py) which makes it harder to migrate. As other recipes such as chromium would depend on build, the build repo can't depend back on them because it'd introduce a circular dependency.

Next step would likely be moving some recipe modules, starting from the top of the dependency tree, i.e. modules nothing else depends on. This requires that all recipes which depend on them move to relevant repos.

It would be really good if we could edit the dot file produced by depgraph to look like the ideal end state so we can figure out if we have any cycles to break and if it looks sane. This is something I'd be happy to experiment with when I have some time if no one else is interested.

Erik

 

To give you a specific example, some starting ones could be auto_bisect, perf_try, ios, webrtc, chromium_tests. See attached dependency graph for reference. You can also generate one yourself by running "./recipes.py --package ~/infra_internal/build/infra/config/recipes.cfg depgraph --ignore-package recipe_engine | tred | dot -Tpdf > graph.pdf" from a recipes-py checkout.

If any internal recipes or modules depend on any of the moved modules, the relevant internal repos would start depending on the project repos (e.g. chromium) using a gitiles-based fetch strategy to avoid expensive pull of these large repos into .recipe_deps. I'm still working on supporting this in the recipe engine, including the autoroller.

WDYT?

Paweł

--

[Paweł Hajdan, Jr.]

Please note rolling internal recipes is discussed in an internal thread. FWIW, https://goto.google.com/lsrrg .

On Thu, Jun 2, 2016 at 7:15 AM, Erik Staab <est...@chromium.org> wrote:

a few comments inline

On Wed, Jun 1, 2016 at 6:15 AM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

I've been looking into next steps needed to start moving recipes to chromium repo.

Please consider this a call for relatively early feedback. Likely timeframe is about a few weeks; certainly not before resolving any outstanding questions/concerns, and probably starting with smaller repos that affect less people, e.g. infra.

I think I'd start with moving the relevant recipes first. kitchen_run/remote_run allows us to do this, even with buildbot. We'd need to add relevant repos to the recipe autoroller (trivial) and change buildbot master configs to use kitchen_run/remote_run factory. We'd also need to modify the factory so that it'd use the same recipe revision as requested build revision, if they refer to the same repo (e.g. chromium).

Agree that we want to start with recipes, e.g. the leaf nodes. I'd like to include them in the graph that's generated from depgraph (probably merging nearly-identical ones) so we can see what can be picked off first.

The graph becomes quite noisy with recipes included, but you can already get that with --recipe-filter switch for recipes.py depgraph. I'm also considering a command/option to just list recipes depending on a specific module.

At a high level, I believe we do know what can be picked off first, and I've mentioned that earlier: auto_bisect, perf_try, ios, webrtc, chromium_tests. I'm interested in your thoughts on this.

 

As I've mentioned before we should definitely do this with the infra recipes first since we might discover something that affects our strategy with chromium. chromium == production and infra == staging.

Yup, we're on the same page here.

 

As a general rule, the other repos like chromium or infra would depend on build (via recipes.cfg), but not the other way around. Unfortunately there's still chromium-related code in build recipes that uses some build scripts there (e.g. compile.py) which makes it harder to migrate. As other recipes such as chromium would depend on build, the build repo can't depend back on them because it'd introduce a circular dependency.

Next step would likely be moving some recipe modules, starting from the top of the dependency tree, i.e. modules nothing else depends on. This requires that all recipes which depend on them move to relevant repos.

It would be really good if we could edit the dot file produced by depgraph to look like the ideal end state so we can figure out if we have any cycles to break and if it looks sane. This is something I'd be happy to experiment with when I have some time if no one else is interested.

Sure. What do you think about the attached graph? It shows what would it take to extract chromium_tests. All the recipes depending on moved out modules would need to move as well.

--- graph.dot.orig 2016-06-06 19:15:02.816249401 +0200

+++ graph.dot 2016-06-06 19:16:17.989756254 +0200

@@ -36,10 +36,7 @@

  cipd;

  skia_swarming;

  commit_position;

- auto_bisect;

  bisect_tester;

- perf_try;

- chromium_tests;

  trigger;

  halt;

  crrev;

@@ -50,7 +47,6 @@

  findit;

  test_results;

  amp;

- webrtc;

  recipe_tryjob;

  luci_config;

  gatekeeper;

@@ -62,7 +58,6 @@

  ct;

  git_clone_bundler;

  ndk;

- ios;

  swarming_heartbeat;

  legion;

  libyuv;

@@ -75,6 +70,13 @@

  buildbucket;

  service_account;

  }

+ subgraph cluster_chromium {

+ auto_bisect;

+ perf_try;

+ ios;

+ webrtc;

+ chromium_tests;

+ }

  skia -> gsutil;

  skia -> adb;

  skia -> swarming;

Paweł

[Stephen]

There are two main things we should do next to verify that recipes in repos work correctly. The main issues I'm concerned about strange python imports and PATH dependencies.

One is to move all infra recipes (everything in this directory) into the infra repo. We might also want to move all the internal recipes into the corresponding internal repo, to test that functionality out, but maybe that could wait for later. This would probably test more edge cases and give us higher confidence that most everything works. We could also move some of the recipe modules in build (the recipe_autoroller module looks like a decent candidate, as it's self contained and does some fairly complicated logic.)

The other is to move recipes into chromium itself, along the lines of what Pawel has suggested with the 5 recipe modules he picked. The attached graph LGTM, although we might want to go through those files and see if there's any weird python imports which would mess with this; in particular, https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/client_skia.py?l=18 is problematic, and there is probably other stuff like that. This would make sure that the gitiles fetch stuff works correctly.

I think these could be pursued in parallel; I would probably work more on the infra side, and pawel would work on the chromium side. How does that sound?

Erik

 

--

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.

[Ben Henry]

Could you make a list of things we need to do to check/validate? I'd like some record of this so that there's no confusion in the future. It could just be a more succinct list of what you just put together, but we should be able to check things off. Email isn't a great medium for this.

Ben

Erik

 

--

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

To post to this group, send email to infr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAATLsPb_EVTpcmSxLsWQ0Go9OYm_9h_8-E57cm%3Do64Y%2Bh5ScEA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/6993610d-03cc-49f5-b675-4b9839c681c0%40chromium.org.

[Erik Staab]

Paweł will be working on a doc to outline next steps that we can review and open up to a wider group in the next week. machenbach brought some good points to our attention in the last week that apply to v8 and might apply elsewhere so we'll get signoff from chromium folks before any rollouts.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CA%2Bju%2B_0Cjnu40-q9MVdPUMeELD4EWstfpqth3V136WEmE2WP-g%40mail.gmail.com.

[Paweł Hajdan, Jr.]

To close the loop (I referenced this thread in some places), the doc is https://goto.google.com/recipes-in-repos (Google-internal at least for now).

Paweł

Erik

 

--

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.

To post to this group, send email to infr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAATLsPb_EVTpcmSxLsWQ0Go9OYm_9h_8-E57cm%3Do64Y%2Bh5ScEA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.

To post to this group, send email to infr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/6993610d-03cc-49f5-b675-4b9839c681c0%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.

To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CA%2Bju%2B_0Cjnu40-q9MVdPUMeELD4EWstfpqth3V136WEmE2WP-g%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.

To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAN60yaK8XoGNxQL2KenO-qJ6%2B1VjDsvMz2z%3D%3Dm9-5izB4OM0RQ%40mail.gmail.com.