depot_tools and src/third_party/depot_tools

[Igor Bukanov]

Hi,

Chromium build instructions require first to checkout depot_tools into some directory, add it to PATH and use it as the source for Chromium tools to fetch and compile the rest.

But then Chromium src also contains a copy of depot_tools under src/third_party/depot_tools. What is the purpose for this second copy?

Regards, Igor

[Dirk Pranke]

The second copy is used to provide a reliable place for tools in src/ to import code from depot_tools. Without this copy one had to hunt around in $PATH trying to find it. Also, this pins depot_tools at a specific revision so that our builds are more hermetic and reliable.

It is awkward and unfortunate that we have both copies, but I have yet to come up with a way to avoid this and keep the properties I just mentioned. My vague hope is that someday we'll be able to check out the tree with a regular `git clone` and then we won't need depot_tools to be installed first. I don't know that that's a realistic hope, though.

If anyone else has a better idea of how to solve this problem I'd be happy to hear it :).

-- Dirk

Regards, Igor

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/9cbed0bb-41b0-4e29-9573-2bc313196a80n%40chromium.org.

[Igor Bukanov]

Hi,

Just to clarify, src/third_party/depot_tools can be used just as depot_tools outside Chromium tree, right? 

As for the initial bootstrap why not to have a script in Chromium src that clones depot_tools into src/third_party/depot_tools and then calls fetch/gclient there?

Regards, Igor

[Dirk Pranke]

On Thu, Mar 9, 2023 at 1:21 PM Igor Bukanov <ibuk...@zscaler.com> wrote:

Hi,

Just to clarify, src/third_party/depot_tools can be used just as depot_tools outside Chromium tree, right? 

Yes, except that the one in third_party is set to not auto-update, because we want the version to be pinned so scripts in src/ and other places can count on modules being where we expect them to be. And, we don't want you to use a pinned version of the tools in your $PATH, we want the commands that you do run to auto-update. 

As for the initial bootstrap why not to have a script in Chromium src that clones depot_tools into src/third_party/depot_tools and then calls fetch/gclient there?

You could do something like that, though there are a bunch of smaller issues you'd have to fix. There's also questions that arise only in that situation, like how to handle multiple checkouts, and whether you run into issues where something like fetch/gclient wants to update to a new version of itself (which is usually hard to do cleanly on Windows, at least).

One could also imagine a world where we split depot_tools up into the stuff that is used as third_party code vs the stuff that is used to bootstrap repos, though I'm not sure if that's actually a clean division and it'd be worth it. Or we could do things like not fetching all the history for the repos, which would make the overall size smaller, at least.

-- Dirk

[Igor Bukanov]

On Thu, Mar 9, 2023 at 10:44 PM Dirk Pranke <dpr...@google.com> wrote:

Yes, except that the one in third_party is set to not auto-update, because we want the version to be pinned so scripts in src/ and other places can count on modules being where we expect them to be. And, we don't want you to use a pinned version of the tools in your $PATH, we want the commands that you do run to auto-update.

Hm, but ninja, probably the most often used command, is already moved to Chromium src with ninja in depot_tools becoming just an invocation script to find ninja in Chromium/src. So why the reasoning about auto-updates does not apply for ninja?

You could do something like that, though there are a bunch of smaller issues you'd have to fix. There's also questions that arise only in that situation, like how to handle multiple checkouts, and whether you run into issues where something like fetch/gclient wants to update to a new version of itself (which is usually hard to do cleanly on Windows, at least).

At least on Linux the following already works to avoid duplicated depot_tools:

# No chromium at this point

mkdir -p chromium/src/third_party

cd chromium

git -C src/third_party clone https://chromium.googlesource.com/chromium/tools/depot_tools.git

set PATH to point to depot_tools under src/third_party

# fecth chromium does not work as it does not see that src is empty, so create .gclient manually

gclient config --spec 'solutions = [
  {
    "name": "src",
    "url": "https://chromium.googlesource.com/chromium/src.git",
    "managed": False,
    "custom_deps": {},
    "custom_vars": {},
  },
]

'

gclient sync

The sync updates depot_tools to the appropriate version and populates src with a full Chromium. So already on Linux and presumably on Mac one does not need an extra depot_tools. And if the fetch command can be fixed to allow the above usage, then the instructions for the checkout will be just by some characters longer.

Regards, Igor

[Dirk Pranke]

On Thu, Mar 9, 2023 at 3:27 PM Igor Bukanov <ibuk...@zscaler.com> wrote:

On Thu, Mar 9, 2023 at 10:44 PM Dirk Pranke <dpr...@google.com> wrote:

Yes, except that the one in third_party is set to not auto-update, because we want the version to be pinned so scripts in src/ and other places can count on modules being where we expect them to be. And, we don't want you to use a pinned version of the tools in your $PATH, we want the commands that you do run to auto-update.

Hm, but ninja, probably the most often used command, is already moved to Chromium src with ninja in depot_tools becoming just an invocation script to find ninja in Chromium/src. So why the reasoning about auto-updates does not apply for ninja?

We don't actually want ninja to auto-update, we want to be able to build using the exact version of the build tool that we used at that revision. That's exactly why we were able to move it into Chromium src once we had CIPD packages for it.

The things that we do want to update are the tools that go talk to other machines, like git-cl, because their API might change over time and we'd need to keep things up to date.

You could do something like that, though there are a bunch of smaller issues you'd have to fix. There's also questions that arise only in that situation, like how to handle multiple checkouts, and whether you run into issues where something like fetch/gclient wants to update to a new version of itself (which is usually hard to do cleanly on Windows, at least).

At least on Linux the following already works to avoid duplicated depot_tools:

# No chromium at this point

mkdir -p chromium/src/third_party

cd chromium

git -C src/third_party clone https://chromium.googlesource.com/chromium/tools/depot_tools.git

set PATH to point to depot_tools under src/third_party

# fecth chromium does not work as it does not see that src is empty, so create .gclient manually

gclient config --spec 'solutions = [
  {
    "name": "src",
    "url": "https://chromium.googlesource.com/chromium/src.git",
    "managed": False,
    "custom_deps": {},
    "custom_vars": {},
  },
]

'

gclient sync

The sync updates depot_tools to the appropriate version and populates src with a full Chromium. So already on Linux and presumably on Mac one does not need an extra depot_tools. And if the fetch command can be fixed to allow the above usage, then the instructions for the checkout will be just by some characters longer.

Yes, you can do this sort of thing. That doesn't mean you won't hit other issues.

-- Dirk

Regards, Igor

[Mike Frysinger]

On Thu, Mar 9, 2023 at 7:09 PM 'Dirk Pranke' via Chromium-dev <chromi...@chromium.org> wrote:

On Thu, Mar 9, 2023 at 3:27 PM Igor Bukanov <ibuk...@zscaler.com> wrote:

On Thu, Mar 9, 2023 at 10:44 PM Dirk Pranke <dpr...@google.com> wrote:

Yes, except that the one in third_party is set to not auto-update, because we want the version to be pinned so scripts in src/ and other places can count on modules being where we expect them to be. And, we don't want you to use a pinned version of the tools in your $PATH, we want the commands that you do run to auto-update.

Hm, but ninja, probably the most often used command, is already moved to Chromium src with ninja in depot_tools becoming just an invocation script to find ninja in Chromium/src. So why the reasoning about auto-updates does not apply for ninja?

We don't actually want ninja to auto-update, we want to be able to build using the exact version of the build tool that we used at that revision. That's exactly why we were able to move it into Chromium src once we had CIPD packages for it.

The things that we do want to update are the tools that go talk to other machines, like git-cl, because their API might change over time and we'd need to keep things up to date.

maybe a hypothetical might help explain what Dirk has been saying.  say you wanted to build Chromium R60 because you have some embedded system that you want to tweak and that's the version it uses.  R60 was released in 2017.  it is currently 2023.  the way we download source and talk to live network infrastructure has changed.  the way the Chromium source code is built has not.  build tools from 2023 are likely to have changed significantly from 2017, and in ways that are incompatible.  so we have two copies of depot_tools: one that tracks the world as it exists now so it can get the code, and one that tracks the world as it existed years ago so it can build the code from years ago.

or maybe an even more extreme example.  install Windows 11 and current web browser and source tools.  download MS-DOS using git.  try to compile it using Visual Studio 2022.  it's not going to work.  or, boot MS-DOS, and try to download anything using git.  that also won't work.  you need modern tools to talk to the modern world (the depot_tools you clone into your $PATH), but you need old tools to compile old software (the depot_tools pinned in the source checkout).

the only reason syncing the two depot_tools works for you right now is because you're trying to build current versions of Chromium ... currently.

-mike

[Bruce Dawson]

Unfortunately that ideal world of building M60 doesn't work. One of my teammates recently tried to build Chrome from two years ago and it failed because we pull ninja from the non-synched (tip-of-tree) depot_tools, and with a two-year-old Chromium repo that doesn't work. The specific message that they got was:

"C:\src\depot_tools\bootstrap-2@3_8_10_chromium_26_bin\python3\bin\python3.exe" C:\src\depot_tools\ninja.py -C out/rel chrome -j 1000 depot_tools/ninja.py: Could not find Ninja in the third_party of the current project, nor in your PATH. Please take a following action to install Ninja. - If your project has DEPS, Add a CIPD Ninja dependency to DEPS. - Otherwise, add Ninja to your PATH *after* depot_tools.

That is, the present version of depot_tools doesn't contain ninja.exe, and the old version of Chromium assumes that it does.

The solution was to sync c:\src\depot_tools back to two years ago, and that worked.

I guess this is to say that full backwards and forwards compatibility is really hard. Since we don't have regression tests to ensure that we don't break two-year-old Chromium we inevitably do break two-year-old Chromium. But, hopefully we don't break it in a way that can't be repaired with modest effort.

[Mike Frysinger]

i would characterize that as a bug/mistake that was corrected, not as an argument against the principle of having two copies of depot_tools

-mike

[Bruce Dawson]

I guess I'm not sure that the bug has been corrected. ninja is still being retrieved from the non-pinned depot_tools, a mechanism which has been fragile and may be in the future (from a backwards compatibility point of view). We won't know for sure whether the problem has been corrected until a few years from now when we try to build Chrome from 2023.

But, I do agree that pinning dependencies in general is good, that there are some exceptions, and perfect forwards/backwards compatibility is really hard.

Maybe we need to make more use of the pinned depot_tools. For instance, we're trying to turn off support for Python 2 in the non-pinned depot_tools. When we eventually remove support entirely that will, presumably, mean that old versions of Chrome will not build, old presubmits will not run, etc. I guess the question is whether pulling Python 2 from the pinned depot_tools would be worth the effort, especially given that the benefits take years to accrue and are modest.

--

Bruce Dawson, he/him

[Dirk Pranke]

What Mike wrote is a better way of explaining what I was trying to explain :).

On Thu, Mar 9, 2023 at 11:59 PM Bruce Dawson <bruce...@google.com> wrote:

I guess I'm not sure that the bug has been corrected. ninja is still being retrieved from the non-pinned depot_tools, a mechanism which has been fragile and may be in the future (from a backwards compatibility point of view).

You are correct that by default we pick up ninja through the depot_tools wrapper, but the binary itself comes from the pinned CIPD entry in //DEPS. So, even if depot_tools gets borked, we should still be okay, you'd just have to adjust your workflow a bit (e.g., put the third_party directory in front of depot_tools in your PATH). For what it's worth, this is the same indirection model in depot_tools that we've always used for GN (though the mechanism for downloading the GN binary itself has changed over time).

[ We of course have the dependency on CIPD, but that should be equivalent to having the dependency on git itself. Of course, only time will tell if that ends up being the case. ]

It is true that it's hard to build releases from very far in the past. These are all usually due to bugs of one form or another, but the build is not fully hermetic yet, we're just 

getting closer all the time (hopefully). We have spent less time on fixing this than I would like.

-- Dirk