PSA: `repo` no longer available inside the SDK chroot

[Brian Norris]

Within the next week, I plan to drop the `repo` tool from $PATH inside the CrOS SDK. You should always use `repo` from outside the SDK/chroot. For more details, see: https://issuetracker.google.com/270994985

Motivation:

We don't give clear advice on where to run `repo` (especially: `repo upload`, which runs pre-upload source code checks), which can lead to inconsistent advice (e.g., "didn't work? try running it (in|out)side the chroot") and behavior. All pre-upload checks should work outside the SDK and enter the SDK as needed.

If you find that pre-upload checks aren't working outside the SDK within a certain project, feel free to file a bug for the repository OWNERS and CC me if you need help routing.

Usage note:

To fully make the PATH changes take effect, you need to exit and re-enter any SDK/chroot shells you have open. Relevant PATH changes landed around March 8, so if you've restarted your SDK since then, no need for further action.

Feel free to reach out to me if you have other questions.

Thanks,

Brian

[Dmitry Torokhov]

Hmm, this is not a welcome change... I have always been running "repo sync" from inside the chroot (which I enter in a few tmuxed sessions after rebooting my workstation and don't exit until the next reboot), and would like to continue doing so...

--
--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
https://groups.google.com/a/chromium.org/group/chromium-os-dev

[Mike Frysinger]

the roadmap for the SDK is explicitly breaking this flow.  there has never been a requirement to run `repo sync` inside the SDK.

-mike

[Dmitry Torokhov]

While this may not been a requirement it was working for at least 8 years. Is there a completing reason for breaking something that is very convenient? 

[Jack Rosenthal]

On Mon, Mar 27, 2023 at 4:07 PM Mike Frysinger <vap...@chromium.org> wrote:

the roadmap for the SDK is explicitly breaking this flow.  there has never been a requirement to run `repo sync` inside the SDK.

-mike

On Mon, Mar 27, 2023 at 6:03 PM Dmitry Torokhov <dt...@chromium.org> wrote:

Hmm, this is not a welcome change... I have always been running "repo sync" from inside the chroot (which I enter in a few tmuxed sessions after rebooting my workstation and don't exit until the next reboot), and would like to continue doing so...

The end state we hope to be at is that entering the SDK is something you don't (or rarely) need to do, which I think mitigates your concern.  Tools such as build_packages will automatically enter the SDK as required.

See go/cros-sdk-revamp:vision for a high level overview of where we plan to be.  (this is a public list, so sorry to non-googlers who won't be able to see this doc 🙁)

[Dmitry Torokhov]

On Mon, Mar 27, 2023, 3:27 PM Jack Rosenthal <jros...@chromium.org> wrote:

On Mon, Mar 27, 2023 at 4:07 PM Mike Frysinger <vap...@chromium.org> wrote:

the roadmap for the SDK is explicitly breaking this flow.  there has never been a requirement to run `repo sync` inside the SDK.

-mike

On Mon, Mar 27, 2023 at 6:03 PM Dmitry Torokhov <dt...@chromium.org> wrote:

Hmm, this is not a welcome change... I have always been running "repo sync" from inside the chroot (which I enter in a few tmuxed sessions after rebooting my workstation and don't exit until the next reboot), and would like to continue doing so...

The end state we hope to be at is that entering the SDK is something you don't (or rarely) need to do, which I think mitigates your concern.  Tools such as build_packages will automatically enter the SDK as required.

I see. In this case cros_sdk --enter needs to be much faster and also rely less on sudo so I don't have to reenter my password multiple times a day. 

See go/cros-sdk-revamp:vision for a high level overview of where we plan to be.  (this is a public list, so sorry to non-googlers who won't be able to see this doc 🙁)

Could we rearrange the work order then? Make everything work transparently outside chroot and then break working flows inside chroot?

[Matthew DeVore]

On Mon, Mar 27, 2023 at 5:32 PM Dmitry Torokhov <dt...@chromium.org> wrote:

On Mon, Mar 27, 2023, 3:27 PM Jack Rosenthal <jros...@chromium.org> wrote:

On Mon, Mar 27, 2023 at 4:07 PM Mike Frysinger <vap...@chromium.org> wrote:

the roadmap for the SDK is explicitly breaking this flow.  there has never been a requirement to run `repo sync` inside the SDK.

-mike

On Mon, Mar 27, 2023 at 6:03 PM Dmitry Torokhov <dt...@chromium.org> wrote:

Hmm, this is not a welcome change... I have always been running "repo sync" from inside the chroot (which I enter in a few tmuxed sessions after rebooting my workstation and don't exit until the next reboot), and would like to continue doing so...

The end state we hope to be at is that entering the SDK is something you don't (or rarely) need to do, which I think mitigates your concern.  Tools such as build_packages will automatically enter the SDK as required.

I see. In this case cros_sdk --enter needs to be much faster and also rely less on sudo so I don't have to reenter my password multiple times a day. 

See go/cros-sdk-revamp:vision for a high level overview of where we plan to be.  (this is a public list, so sorry to non-googlers who won't be able to see this doc 🙁)

Could we rearrange the work order then? Make everything work transparently outside chroot and then break working flows inside chroot?

As someone not involved in the effort, I appreciate the direction to reduce fragmentation in workflows. This should reduce support burden in the short term and I would prefer they don't put it off.

I would recommend running repo sync and entering the chroot in a single wrapper script. Then you know that if you're in the chroot, you ran sync already.

Just my two cents.

 

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-os-d...@chromium.org.

[Dmitry Torokhov]

On Mon, Mar 27, 2023 at 3:40 PM Matthew DeVore <mat...@google.com> wrote:

On Mon, Mar 27, 2023 at 5:32 PM Dmitry Torokhov <dt...@chromium.org> wrote:

On Mon, Mar 27, 2023, 3:27 PM Jack Rosenthal <jros...@chromium.org> wrote:

On Mon, Mar 27, 2023 at 4:07 PM Mike Frysinger <vap...@chromium.org> wrote:

the roadmap for the SDK is explicitly breaking this flow.  there has never been a requirement to run `repo sync` inside the SDK.

-mike

On Mon, Mar 27, 2023 at 6:03 PM Dmitry Torokhov <dt...@chromium.org> wrote:

Hmm, this is not a welcome change... I have always been running "repo sync" from inside the chroot (which I enter in a few tmuxed sessions after rebooting my workstation and don't exit until the next reboot), and would like to continue doing so...

The end state we hope to be at is that entering the SDK is something you don't (or rarely) need to do, which I think mitigates your concern.  Tools such as build_packages will automatically enter the SDK as required.

I see. In this case cros_sdk --enter needs to be much faster and also rely less on sudo so I don't have to reenter my password multiple times a day. 

See go/cros-sdk-revamp:vision for a high level overview of where we plan to be.  (this is a public list, so sorry to non-googlers who won't be able to see this doc 🙁)

Could we rearrange the work order then? Make everything work transparently outside chroot and then break working flows inside chroot?

As someone not involved in the effort, I appreciate the direction to reduce fragmentation in workflows. This should reduce support burden in the short term and I would prefer they don't put it off.

I would say that the team should support customers' needs first. Now, If I am the only one running repo sync/upload/etc inside chroot I'll shut up, but I am pretty sure I am not alone.

 

I would recommend running repo sync and entering the chroot in a single wrapper script. Then you know that if you're in the chroot, you ran sync already.

Once in ~20 days only works if you are hacking on the kernel (oh, how I wish...) but not often enough for other cases. This also does not solve other repo usages.

[Junichi Uekawa (上川純一)]

Hi 

I have a script that runs repo upload inside or outside depending on current project.

It seems like some projects depend on being inside for running presubmit.

Keeping presubmit hermetic hasn't been done well outside of chroot on cros and depends on some external tools outside of codebase. Or was this already resolved?

2023年3月28日(火) 6:45 Brian Norris <brian...@chromium.org>:

--

[Brian Norris]

Hi Junichi,

Do you have a list of such projects? Private email, bug report, etc., welcome.

We don't have exhaustive test coverage on presubmits to know what is broken or not, but I'm not aware of anything that is currently broken outside the chroot. I personally haven't run `repo upload` inside the chroot for years, and I'm aware of others that do the same.

[Manoj Gupta]

For a data point, I do run repo start/abandon/upload etc. inside the chroot. One reason is paths inside ebuilds always refer to internal chroot paths when identifying files to modify and it will be a hassle to (tab) switch in/out just for upload.

[Junichi Uekawa (上川純一)]

I checked my script, seems like except for infra directories which seems to need outside of chroot, 

I ended up running preupload hooks inside the chroot because they didn't reliably pass.

From my records, I switched my script to running repo upload inside chroot back in March 2022.

function main() {
  case $PWD in
    */infra/*|*/config-internal/*)
      # some repositories need outside of chroot.
      record repo upload --yes --cbr .
      ;;
    *cros-work2/*)
      # CrOS usually works better with inside chroot.
      record cros_sdk --working-dir . repo upload --yes --cbr .
      ;;
    *)
      # And then the rest (Android)
      record repo upload --yes --cbr .
      ;;
  esac
}

main

2023年3月28日(火) 8:10 Brian Norris <brian...@chromium.org>:

--

Junichi Uekawa
Google

[Mike Frysinger]

it's been inconsistent for at least 8 years.  we're finally fixin' to make it consistent.

further, we've proved more than once that decisions made at the start of the project, while fine for the time, don't really scale.  the doc Jack linked to (which was announced & shared widely internally) provides quite a bit more detail.

"fix all the things first before making changes" doesn't really work.  there have been too many divergent workflows that we've enabled over the years, and there is literally no list of which ones are still actively used, and many of them are maintained by individual teams, not the build/infra teams.  we've identified & fixed many common issues already.

so as Brian pointed out, if you're aware of breakage where running a command outside the SDK doesn't work, please file a go/cros-infra-bug for triaging & routing.  i can confidently say that `repo sync` has never been such an issue -- it has always worked outside the SDK since the start of the project.  i've only ever run it outside.  same goes for every other repo command other than `repo upload` which executes a lot of project-specific hooks that individual teams have authored.
-mike

[Dmitry Torokhov]

"Fixin' to make it consistent" by removing repo from the SDK might benefit the SDK team as it makes their life easier, but it does absolutely nothing to your users. What would benefit users is making it work well from both places, or at least not removing it until you make cros_sdk truly transparent. For now you are simply breaking users workflows: the same way as you have "only ever run it outside" I ran it outside only once when I set up my chroot, and since then I ran it only from the inside.

Thanks,

Dmitry

[Matthew Blecker]

Thank you, this is good cleanup.  It's always been my understanding that `repo` should be run outside the chroot.  While I've worked on CrOS I've not encountered any situation that needed `repo` inside the chroot.  Its presence in chroot $PATH there was always surprising, and occasionally I would run it inside the chroot by mistake (wrong window focus).

On Mon, Mar 27, 2023 at 2:45 PM Brian Norris <brian...@chromium.org> wrote:

--

[Miriam Zimmerman]

This is strange to me. Until very recently I have been able to run `repo upload` from outside of the chroot in platform2, but recently I've started having to run it within the chroot.  If I try to run it from outside the chroot, it prompts for a password, which I assume is due to entering the chroot requiring sudo. But I shouldn't have to authenticate as root to run presubmits / upload a patch set, and it seems bad to encourage people to blindly authenticate as root multiple times a day.

[Brian Norris]

That might be https://issuetracker.google.com/272368241, which is resolved now. If you have further issues that aren't resolved, feel free to file a bug.

[Mike Frysinger]

i don't think "make the inside & outside behave the same" is realistic.  we've heard constant feedback from developers that having 2 execution environments is confusing.  the set of programs available is always inconsistent, both in what's available, and their versions, and this is completely irreconcilable when we, by design, don't control the host distro.  further, this trajectory aligns us with how basically every other project, inside & outside of Google, behaves.  which is also why we're driving bazel usage this year -- that will be the default development tool for people moving forward.

yes, we're aware that this has a non-zero impact on some/many developer's workflows.  but the status quo is already not great for many devs, as they have well told us, so the status quo needs to change.

many workflows are already transparent with the SDK, but people haven't been aware.  for example, it's always been possible to execute update_chroot & setup_board & build_packages & build_image without entering the chroot, but our docs & muscle memory have trained them differently.  if you're aware of something that doesn't work, please file a go/cros-build-bug to make sure it's on our radar.

-mike

[ggg]

On Tuesday, March 28, 2023 at 9:34:59 AM UTC-7 Dmitry Torokhov wrote:

"Fixin' to make it consistent" by removing repo from the SDK might benefit the SDK team as it makes their life easier, but it does absolutely nothing to your users. What would benefit users is making it work well from both places, or at least not removing it until you make cros_sdk truly transparent. For now you are simply breaking users workflows: the same way as you have "only ever run it outside" I ran it outside only once when I set up my chroot, and since then I ran it only from the inside.

I'm in the same situation as Dmitry: Other than "repo init" outside the chroot, I run everything inside thinking Chrome OS team can make sure the tools invoked are secure and working.

Can I do all of the following outside the chroot?

  repo start (create a working branch)

  cros_workon

  build_packages

  build_image

  cros_deploy (or update_kernel.sh)

  servod

  dut_control

  test_that (autotest)

  repo upload

  gerrit (change labels on groups of CLs)

  servod

  dut_control

Offhand, I think the only other command I regularly use outside the chroot is minicom and cscope/tag (I'm using cs.corp instead more often now).

cheers,

grant

[Jack Rosenthal]

On Tue, Mar 28, 2023 at 1:26 PM ggg <grun...@chromium.org> wrote:

On Tuesday, March 28, 2023 at 9:34:59 AM UTC-7 Dmitry Torokhov wrote:

"Fixin' to make it consistent" by removing repo from the SDK might benefit the SDK team as it makes their life easier, but it does absolutely nothing to your users. What would benefit users is making it work well from both places, or at least not removing it until you make cros_sdk truly transparent. For now you are simply breaking users workflows: the same way as you have "only ever run it outside" I ran it outside only once when I set up my chroot, and since then I ran it only from the inside.

I'm in the same situation as Dmitry: Other than "repo init" outside the chroot, I run everything inside thinking Chrome OS team can make sure the tools invoked are secure and working.

Can I do all of the following outside the chroot?

  repo start (create a working branch)

yes

 

  cros_workon

  build_packages

  build_image

  cros_deploy (or update_kernel.sh)

the plan is to have these tools auto-enter the chroot as required

 

  servod

  dut_control

the firmware sdk project intends to add launchers for these to depot tools so they can be run outside the chroot.  that's coming later in Q2

 

  test_that (autotest)

TBD on this one ... it likely requires coordination with the tast team

 

  repo upload

yup

 

  gerrit (change labels on groups of CLs)

yup, works fine outside the chroot.  you'll want to either manually specify the path to ${CHECKOUT}/chromite/bin/gerrit, symlink it into your PATH, or just put ${CHECKOUT}/chromite/bin in your PATH.

one option is we may add a launcher for this command in depot_tools if the demand is high enough

 

  servod

  dut_control

Dup of listed above.

[ggg]

On Tuesday, March 28, 2023 at 12:33:04 PM UTC-7 Jack Rosenthal wrote:

On Tue, Mar 28, 2023 at 1:26 PM ggg <grun...@chromium.org> wrote:

On Tuesday, March 28, 2023 at 9:34:59 AM UTC-7 Dmitry Torokhov wrote:

"Fixin' to make it consistent" by removing repo from the SDK might benefit the SDK team as it makes their life easier, but it does absolutely nothing to your users. What would benefit users is making it work well from both places, or at least not removing it until you make cros_sdk truly transparent. For now you are simply breaking users workflows: the same way as you have "only ever run it outside" I ran it outside only once when I set up my chroot, and since then I ran it only from the inside.

I'm in the same situation as Dmitry: Other than "repo init" outside the chroot, I run everything inside thinking Chrome OS team can make sure the tools invoked are secure and working.

Can I do all of the following outside the chroot?

  repo start (create a working branch)

yes

 

  cros_workon

  build_packages

  build_image

  cros_deploy (or update_kernel.sh)

the plan is to have these tools auto-enter the chroot as required

 

  servod

  dut_control

the firmware sdk project intends to add launchers for these to depot tools so they can be run outside the chroot.  that's coming later in Q2

 

  test_that (autotest)

TBD on this one ... it likely requires coordination with the tast team

 

  repo upload

yup

 

  gerrit (change labels on groups of CLs)

yup, works fine outside the chroot.  you'll want to either manually specify the path to ${CHECKOUT}/chromite/bin/gerrit, symlink it into your PATH, or just put ${CHECKOUT}/chromite/bin in your PATH.

one option is we may add a launcher for this command in depot_tools if the demand is high enough

Jack, thanks. That's what I needed to know.

Summary: only repo and gerrit (with some additional setup) currently work outside the chroot. Everything else is still a plan.

Sorry, a meme using "deprecated2" template somehow seems appropriate.

Dmitry is spot on: "What would benefit users is making it work well from both places, or at least not removing it until you make cros_sdk truly transparent."

But this applies to all the current workflows we use.

I saw Mike's response and understand why this needs to change. But please make the transition period short - like a day or two, not by the quarter.

cheers,

grant

[Jack Rosenthal]

Just an update: gerrit is now in depot_tools (crrev.com/c/4377791), so no additional setup should be required anymore for this one =)

 

Sorry, a meme using "deprecated2" template somehow seems appropriate.

Dmitry is spot on: "What would benefit users is making it work well from both places, or at least not removing it until you make cros_sdk truly transparent."

But this applies to all the current workflows we use.

I saw Mike's response and understand why this needs to change. But please make the transition period short - like a day or two, not by the quarter.

Thanks, I'll discuss with our team at our next meeting what we can do to minimize this transitional pain.

[Mike Frysinger]

On Tue, Mar 28, 2023 at 3:26 PM ggg <grun...@chromium.org> wrote:

On Tuesday, March 28, 2023 at 9:34:59 AM UTC-7 Dmitry Torokhov wrote:

"Fixin' to make it consistent" by removing repo from the SDK might benefit the SDK team as it makes their life easier, but it does absolutely nothing to your users. What would benefit users is making it work well from both places, or at least not removing it until you make cros_sdk truly transparent. For now you are simply breaking users workflows: the same way as you have "only ever run it outside" I ran it outside only once when I set up my chroot, and since then I ran it only from the inside.

I'm in the same situation as Dmitry: Other than "repo init" outside the chroot, I run everything inside thinking Chrome OS team can make sure the tools invoked are secure and working.

there are many tools that devs use that have been unavailable, or unreliable/outdated.  emacs & vscode & similar editors readily come to mind.  so i'll reiterate that, while there are some/many devs who have been fine with living inside the SDK, there are plenty who have not, and others who have always been confused by it.  we want to converge development flows rather than support multiple parallel modes because it means we end up poorly supporting all of them as we spread too thin.

we also run into the problem of things inside the SDK touching the network needing to be updated with the outside world otherwise it falls apart.  off the top of my head, we've had to backport git, gsutil, curl, and root CA certs to branches to keep them working.

-mike

[Mike Frysinger]

On Tue, Mar 28, 2023 at 3:32 PM Jack Rosenthal <jros...@chromium.org> wrote:

On Tue, Mar 28, 2023 at 1:26 PM ggg <grun...@chromium.org> wrote:

  cros_workon

  build_packages

  build_image

  cros_deploy (or update_kernel.sh)

the plan is to have these tools auto-enter the chroot as required

these have "always" worked inside & outside the SDK.  i didn't fully dive the history here, but i think you'll agree with my assessment.

build_packages since Dec 2011 (prob even earlier)

build_image since May 2010

cros_workon since Jun 2010 (when the script was introduced)

cros deploy since Feb 2015

update_kernel.sh since Jan 2011 (and actually, this only worked outside the SDK before this date)

-mike

[Brian Norris]

I've landed a warning within the tool (when run inside the SDK)
instead of complete removal for now. Please consider adapting your
flows if this is going to be a problem for you, and please continue to
provide feedback (preferably in the form of a bug report) if
pre-upload checks do not work or are providing inconsistent results
outside the SDK.

I don't plan to send a future PSA for the eventual removal, but I've
taken feedback into account on things to work on before then.

[Allen Webb]

I hate to revive an older thread, but this seemed like the best place
to bring up the issue. I think one thing we could do to make
dev-workflows more visible outside the sdk would be to make setting up
bash completion outside the SDK automated and part of the typical dev
setup.

[Vadim Bendebury (вб)]

I just ran into this discussion, catching up with email after a break.

One onther thing which comes to mind is that chroot us a sealed environment, and Chrome OS development is supposed to be possible by non-google contributors, which run various versions of UNIX.  Running inside chroot eliminates a lot of ambiguity.

For the record in my workflow I also run 'repo' inside chroot pretty much exclusively.

-vb

[Jack Rosenthal]

As an update to this thread, repo is now removed from CrOS SDK.  Please run it outside.