Running v8 build inside containers, `gclient sync` throws on a `cipd ensure ...` with `invalid cross-device link`

[Joshua Hemphill]

Attempting to run the chromium tool chain to build v8 inside a container results in it failing at the `gclient sync` stage; cipd throws errors that it can't move or remove because `invalid cross-device link`.

I've done my best to eliminate anything else as the culprit; I've run it on 3 different hosts with 2 different host file systems (so with and without native overlay diffs) and tried with three different storage drivers, all resulting in the same error. (also some old bug reports on this error in other tools say to disable metacopy, which is already disabled by default in all the container backends I  tried)

The remaining info I can find on similar errors point to it being that the program throwing the error incorrectly handles `mv` throwing `XDEV` errors, where other posix tools handle it by falling back to `cp` then `rm`.

I'm currently trying to contribute to a repo that builds v8 as part of its build chain, and the maintainer wants to just add scripts to search and replace in gclient/cipd to intercept/change the functionality, but I wanted to see if it's possible to get the functionality changed in the tool itself so I can eliminate some brittle `sed` commands.

[Vadim Shtayura]

Can you identify what files exactly it fails to move? i.e. what device boundary move crosses?

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/9cd0a2e9-1883-4421-9f20-ed9102d179ean%40chromium.org.

[Joshua Hemphill]

After posting I realized I should have explained why I'm posting here and not opening a bug report.

At the time I thought it would be filed under the chromium project, and couldn't find any templates/tags that I could create a bug with that would have been appropriate for this; though now I've found there's a separate `gn` project that looks like it might be the right place, though I could still use confirmation on that. 

I also can't imagine I'm the only one running the build tools inside a container, so I want to make sure this isn't something known that I just haven't been able to find an exact bug for or if my conclusion is wrong.

The files involved, exact error is:

________ running 'cipd ensure -log-level error -root /google -ensure-file /tmp/tmpq0p1bgkv.ensure' in '.'
[P717 06:13:56.876 client.go:1901 E] [cleanup] Failed to remove infra/3pp/tools/ninja/linux-amd64 in "v8/third_party/ninja": removing the deployed package directory: rename /google/.cipd/pkgs/1 /google/.cipd/pkgs/1KdFTau6oBml: invalid cross-device link
Errors:
  failed to remove infra/3pp/tools/ninja/linux-amd64 in "v8/third_party/ninja": removing the deployed package directory: rename /google/.cipd/pkgs/1 /google/.cipd/pkgs/1KdFTau6oBml: invalid cross-device link
Error: Command 'cipd ensure -log-level error -root /google -ensure-file /tmp/tmpq0p1bgkv.ensure' returned non-zero exit status 1
[P717 06:13:56.876 client.go:1901 E] [cleanup] Failed to remove infra/3pp/tools/ninja/linux-amd64 in "v8/third_party/ninja": removing the deployed package directory: rename /google/.cipd/pkgs/1 /google/.cipd/pkgs/1KdFTau6oBml: invalid cross-device link
Errors:
  failed to remove infra/3pp/tools/ninja/linux-amd64 in "v8/third_party/ninja": removing the deployed package directory: rename /google/.cipd/pkgs/1 /google/.cipd/pkgs/1KdFTau6oBml: invalid cross-device link

Portions of my dockerfile are probably relevant as well for the versions being pulled:

# Prepare V8
RUN mkdir google
WORKDIR /google
RUN git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
WORKDIR /google/depot_tools
RUN git checkout remotes/origin/main
ENV PATH=/google/depot_tools:$PATH
WORKDIR /google
RUN fetch v8
WORKDIR /google/v8
RUN git checkout 10.6.194.14
RUN sed -i 's/snapcraft/nosnapcraft/g' ./build/install-build-deps.sh
RUN ./build/install-build-deps.sh
RUN sed -i 's/nosnapcraft/snapcraft/g' ./build/install-build-deps.sh
WORKDIR /google
RUN gclient sync
RUN echo V8 preparation is completed.

# Build V8
WORKDIR /google/v8
RUN python3 tools/dev/v8gen.py x64.release -- v8_monolithic=true v8_use_external_startup_data=false is_component_build=false v8_enable_i18n_support=false v8_enable_pointer_compression=false v8_static_library=true symbol_level=0 use_custom_libcxx=false v8_enable_sandbox=false
COPY ./scripts/python/patch_v8_build.py .
RUN ninja -C out.gn/x64.release v8_monolith || python3 patch_v8_build.py -p ./
RUN ninja -C out.gn/x64.release v8_monolith
RUN rm patch_v8_build.py
RUN echo V8 build is completed.

[Vadim Shtayura]

I think it happens because "/google/.cipd/pkgs/1" is created in RUN fetch v8 step, but moved in RUN gclient sync step, which is using a different file system layer already. A simple workaround on your side might be doing all this stuff in a single RUN step e.g. by moving all checkout logic in a script and running this script as a single RUN (or chaining a series of commands in RUN using &&). Another potential workaround would be to rm -rf "/google/.cipd" before calling "gclient sync", though I'm not sure it will work. It will likely delay the error a bit.

The problem is that CIPD relies on the atomicity of rename(...) in a lot of places. It assumes files in the same directory are on the same device. Looks like Docker layers violate this assumption. Replacing atomic rename with cp+rm will likely introduce subtle bugs.

If you want to have a bug on file, use https://crbug.com and "Infra>LUCI>Artifacts>Packages" component.

[Joshua Hemphill]

Thanks for your help. That makes a lot of sense; I've been trying to add arm builds to the repo I'm working on, so just making a separate script is making a lot of sense now.

Though I did some more digging, and it looks like `rename()` has a bitmap option parameter specifically for overlay filesystems:  RENAME_WHITEOUT

https://man7.org/linux/man-pages/man2/rename.2.html#:~:text=vfat%2C%20and%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20bpf.-,RENAME_WHITEOUT,-(since%20Linux%203.18

Would you happen to know enough about the specifics to know if that might be what I'm looking for? It mentions that tmpfs has explicit support for RENAME_WHITEOUT, so maybe RUN commands with explicit tmpfs mount type would do the trick?