depot_tools ships some pre compiled dynamically linked binaries

[Natanael Copa]

Hi!

We have been building chromium for Alpine Linux, which uses musl libc, for a while and I thought I should at least start to send patches upstream.

First issue was some build errors in breakpad, so i checked out breakpad. But I could not even start building it without first checking out even more third_party libraries, so I needed depot_tools for this.

But depot_tools does not run on my Alpine Linux system because there are 3 precompiled binares which are dynamically linked to GNU libc, which does not exist on Alpine Linux.

The exact list is:

ET_EXEC libpthread.so.0,libc.so.6 ./.cipd_client 

ET_EXEC libstdc++.so.6,libm.so.6,libgcc_s.so.1,libc.so.6 ./ninja-linux64 

ET_EXEC libstdc++.so.6,libm.so.6,libgcc_s.so.1,libc.so.6 ./ninja-linux32 

I suppose the ninja binary could be replaced or symlinked with the system ninja binary (we have a package for ninja). But I dont know where I find the sources for this .cipd_client?

Would it be an idea to at least build those binaries with static linking? It would not solve the problem when building on ppc64le or similar machines (which are quite nice for compiling huge projects btw...) but  that way there will be no dependency on any system libc.

I'd be happy to help getting the build tools working on Alpine Linux so I have a chance to contribute back patches we use.

-nc

[Mike Frysinger]

breakpad can be checked out using `repo` instead and then built using normal configure+make

-mike

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/3aa08c50-bf82-4f8e-8dde-349c368b0d36%40chromium.org.

[Lei Zhang]

For Ninja, why not just build your own or use a system provided copy,
and put it in front of depot_tools in your PATH?

[Natanael Copa]

But there are still precompiled binaries in depot_tools which was the issue I tried to report and fix with this thread.

That said, running `repo` told me i should run `repo init` which I did, but gave me the following error:

   fatal: manifest url (-u) is required.

repo --help tells me i can paginate something and print version but nothing about what the purpose of the `repo` tool is.

Googling "repo" geave me something about repurchment agreements and this: https://source.android.com/setup/using-repo

But does not say anything about the purpose of the `repo` tool. I guess its so fundamental obvious to most of the world that it is not even worth mention what `repo` is, so I am probably being stupid here.

-nc

[Natanael Copa]

That's what I planned to do, whenever I got there.

I still found it "weird" or unusual to find precompiled binaries instead of sources in a git repo.

[Mike Frysinger]

breakpad's usage of repo is like all the other projects that use repo (like Android).  but if you've never used any of those before, that doesn't help.

mkdir breakpad

cd breakpad

repo init -u https://chromium.googlesource.com/breakpad/breakpad

repo sync

-mike

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/ff07b47a-ff5b-4a10-b14c-4934a7766e59%40chromium.org.

[Daniel Jacques]

Unfortunately, for legacy reasons, "ninja" is actually checked into "depot_tools". This is actually a carry-over from when "depot_tools" was a SVN repository and part of Chromium's full source tree. This is something that Chrome Operations has a long-term goal of fixing, but is probably not going to get better anytime soon.

The ".cipd_client" binary is not checked in; instead, it is dynamically bootstrapped from a precompiled Go binary by the cipd script. However, as you noted, those Go binaries seem to be linked against GNU libraries. If you wanted to rebuild them, you can check them out by syncing the "infra/infra" repository (or the "luci-go" repository) and building the CIPD client binary:

A fast way to check out "infra" is:

$ fetch infra

Once checked out, you can bootstrap the Go toolchain and vendoring and build the binary using "env.py":

$ ./infra/go/env.py go build go.chromium.org/luci/cipd/client/cmd/cipd

I'm not sure if this will work under Alpine Linux; if it doesn't, you would have to install your own Go toolchain and manually build the CIPD client:

$ export GOPATH=/path/to/gopath

$ go get -u go.chromium.org/luci/cipd/client/cmd/cipd

$ go build go.chromium.org/luci/cipd/client/cmd/cipd

Note that unlike the infra build, this will build from HEAD of all of "cipd"'s dependencies, which may or may not be problematic. The "infra" repository specifically vendors in some versions.

Please follow up with your results. The Chrome Operations team has generally viewed 64-bit x86 Linux Go binaries as globally compatible, and if Alpine Linux is an exception, it would be good to know in case there is a desire to formally support it sometime in the future.

Thanks!

[Primiano Tucci]

cipd_client docs are here, but I don't think you need them to build chrome.

Going back to your concern, it's unpragmatic for a gigantic project like chrome to build everything from scratch.

Ultimately, we should bootstrap everything by starting building clang (the compiler) in the first place, because most of the code in our repository requires newer C++ features that aren't available in the compiler/c++ libs that are shipped in most linux distributions unless they are bleeding edge (e.g., debian testing).

Pragmatically, this would increase the development cycle time by hours and the size of the checkout and output folders by hundreds of GBs.

Unfortunately you chose to use a distribution that is not officially supported.

And while everything you are trying to do is legit, and I am very confident there is a way to rebuild all those packages from scratch, you are mostly on your own because nobody builds in the configuration you are currently using.

Realistically there are way too many distributions out there and we can't possibly support all of them, nor we can bootstrap every single binary we use.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAAbOSc%3D9DxionwyVAfDr5UjK0VYWzD2hejrsfsbv1C1QTP_Kbg%40mail.gmail.com.

[Natanael Copa]

This was very helpful and helped me forward. Thank you.

[Natanael Copa]

Thank you! This was very helpful.

`fetch infra` did not work since it failed with an error: ..../.cipd_client: Not a valid dynamic program

Which was sort of the problem I was trying to solve in the first place.

Building with my system go did produce a binary that runs on alpine.

I replaced the depot_tools/cipd with the homebuilt binary and tried to `fetch infra` again, but got this error:

  depot_tools update failed. Conflict in /home/ncopa/src/breakpad/../depot_tools

  Cannot rebase: You have unstaged changes.

  Please commit or stash them.

  Your current directory appears to already contain, or be part of, 

  a checkout. "fetch" is used only to get new checkouts. Use 

  "gclient sync" to update existing checkouts.

So it looks like it does a git pull --rebase which fails due to my hack and there I got stuck

One way to remove the libc dependency would be to build the official cipd binary static:

   go build -a -ldflags '-extldflags "-static"' go.chromium.org/luci/cipd/client/cmd/cipd

That way depot_tools would work regardless of the system libc. Alternatively, the cipd wrapper script could try build the cipd binary using system go as a fallback.

Thank you very much for you help!

-nc

[Christian Biesinger]

You can disable the auto update with the instructions in
https://www.chromium.org/developers/how-tos/depottools#TOC-Disabling-auto-update

Christian

> --
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
> ---
> You received this message because you are subscribed to the Google Groups
> "Chromium-dev" group.
> To view this discussion on the web visit

> https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/e1ce634b-b441-486c-8976-6b1826bb047a%40chromium.org.

[Natanael Copa]

Yes, unfortunately I started to build a distro at a time when google chrome did not even exist,and unfortunately I tried and succeeded to build chromium for this distro and sort of like it better than firefox too.

I perfectly understand that you can not support every distro that is out there and I don't expect you to support Alpine Linux either. I still believe it would be nice to be able to send patches upstream that improves the general portability as it will make it easier (or even possible) to port stuff for future architectures and systems. At least for some of those third_party components which may be used other places too. But of course, I have better things to spend my time on if such patches are not welcome.

-nc 

[d...@chromium.org]

Thanks for the follow-up. I've posted crbug.com/796320 to look into your recommendation for Go binary building.

[sv...@d34dl0ck.me]

Hi Natanael,

I'm also currently working on building something similar on Alpine Linux, not chromium but dart. However I also ran into issues, not so much related to cipd client itself, but related to the python wheel packages that are fetch from gsutil also one of the depot_tools. 

bash-4.4# gsutil.py --version
[E2019-02-17T18:16:03.075419Z 1895 0 venv.go:942] Command (cwd=): [/root/.vpython-root/e7d824/bin/python -B -E -s -m pip install --isolated --compile --no-index --find-links /tmp/vpython_bootstrap282292112/packages --requirement /tmp/vpython_bootstrap282292112/requirements.txt --no-binary=:none: --only-binary=:all: --use-wheel]
Process output:
Collecting appdirs==1.4.3 (from -r /tmp/vpython_bootstrap282292112/requirements.txt (line 1))
Collecting asn1crypto==0.22.0 (from -r /tmp/vpython_bootstrap282292112/requirements.txt (line 2))
Collecting boto==2.48.0 (from -r /tmp/vpython_bootstrap282292112/requirements.txt (line 3))
Collecting cffi==1.10.0 (from -r /tmp/vpython_bootstrap282292112/requirements.txt (line 4))
  Could not find a version that satisfies the requirement cffi==1.10.0 (from -r /tmp/vpython_bootstrap282292112/requirements.txt (line 4)) (from versions: )
No matching distribution found for cffi==1.10.0 (from -r /tmp/vpython_bootstrap282292112/requirements.txt (line 4))

I tried out couple of approaches to build those wheel packages from source. That workout out, but unfortunately is gsutil executed with vpython and a spec file that downloads the wheels and installs them in a virutalEnv everytime it is launched. E.g. the cffi package comes from here https://chrome-infra-packages.appspot.com/p/infra/python/wheels/cffi/linux-amd64_cp27_cp27mu/+/d38ff8aa26ead615db8c1beae2e921f89aa8fb27

So I was not able to "inject" my pre build wheels somehow. (I tried to manipulate the .vpython_cipd_cache/ and make up own files). Did you also came across this issue? Or found even a solution to sort it out?