Missing llvm binaries with reclient on Linux

[Marshall Greenblatt]

Hello!

I'm trying to build Chromium using reclient on Linux. I'm following the guide here (which mostly uses the default Chromium config + docker image). Upon execution the remote build is failing with "executable file `../../third_party/llvm-build/Release+Asserts/bin/clang++` not found in $PATH". Is there something more that I need to configure?

Thanks,

Marshall

My args.gn is:

clang_use_chrome_plugins=false
dcheck_always_on=true
disable_fieldtrial_testing_config=true
enable_background_mode=false
enable_backup_ref_ptr_support=false
enable_downgrade_processing=false
enable_linux_installer=false
enable_nacl=false
enable_resource_allowlist_generation=false
enable_widevine=true
is_cfi=false
is_component_build=false
is_debug=false
is_official_build=true
optimize_webui=true
symbol_level=1
target_cpu="x64"
use_partition_alloc_as_malloc=false
use_qt5=false
use_qt6=false
use_remoteexec=true
use_sysroot=true

[Marshall Greenblatt]

On Tue, Apr 15, 2025 at 12:21 PM Marshall Greenblatt <magree...@gmail.com> wrote:

Hello!

I'm trying to build Chromium using reclient on Linux. I'm following the guide here (which mostly uses the default Chromium config + docker image). Upon execution the remote build is failing with "executable file `../../third_party/llvm-build/Release+Asserts/bin/clang++` not found in $PATH". Is there something more that I need to configure?

Complete error message (example):

FAILED: obj/third_party/protobuf/protobuf_full/dynamic_message.o
../../buildtools/reclient/rewrapper -cfg=../../buildtools/reclient_cfgs/chromium-browser-clang/rewrapper_linux.cfg -inputs=build/config/warning_suppression.txt -exec_root=/home/marshall/code/chromium_git/chromium/src/ ../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF obj/third_party/protobuf/protobuf_full/dynamic_message.o.d -DDCHECK_ALWAYS_ON=1 -DUSE_UDEV -DUSE_AURA=1 -DUSE_GLIB=1 -DUSE_OZONE=1 -DOFFICIAL_BUILD -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -DNO_UNWIND_TABLES -D_GNU_SOURCE -DCR_CLANG_REVISION=\"llvmorg-21-init-5118-g52cd27e6-5\" -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_EXTENSIVE -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCXXABI_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCPP_INSTRUMENTED_WITH_ASAN=0 -DCR_LIBCXX_REVISION=449310fe2e37834a7e62972d2a690cade2ef596b -DTMP_REBUILD_HACK -DCR_SYSROOT_KEY=20250129T203412Z-1 -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DGOOGLE_PROTOBUF_NO_RTTI -DHAVE_PTHREAD -I../.. -Igen -I../../buildtools/third_party/libc++ -I../../third_party/protobuf/third_party/utf8_range -I../../third_party/protobuf/src -I../../third_party/abseil-cpp -fno-delete-null-pointer-checks -fno-strict-overflow -fno-ident -fno-strict-aliasing -fstack-protector -fno-unwind-tables -fno-asynchronous-unwind-tables -fPIC -pthread -fcolor-diagnostics -fmerge-all-constants -fno-sized-deallocation -fcrash-diagnostics-dir=../../tools/clang/crashreports -mllvm -instcombine-lower-dbg-declare=0 -mllvm -split-threshold-for-reg-with-hint=0 -ffp-contract=off -Wa,--crel,--allow-experimental-crel -flto=thin -fsplit-lto-unit -mllvm -inlinehint-threshold=360 -fwhole-program-vtables -fcomplete-member-pointers -m64 -msse3 -ffile-compilation-dir=. -no-canonical-prefixes -Xclang --warning-suppression-mappings=../../build/config/warning_suppression.txt -ftrivial-auto-var-init=pattern -O2 -fdata-sections -ffunction-sections -fno-unique-section-names -fno-math-errno -fno-omit-frame-pointer -gdwarf-4 -g1 -gdwarf-aranges -fvisibility=hidden -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -Wall -Wno-unused-variable -Wno-c++11-narrowing -Wno-unused-but-set-variable -Wno-misleading-indentation -Wno-missing-field-initializers -Wno-unused-parameter -Wno-psabi -Wloop-analysis -Wno-unneeded-internal-declaration -Wno-cast-function-type -Wno-thread-safety-reference-return -Wno-nontrivial-memcall -Werror -Wno-deprecated-this-capture -std=c++20 -Wno-trigraphs -gsimple-template-names -fno-exceptions -fno-rtti -nostdinc++ -isystem../../third_party/libc++/src/include -isystem../../third_party/libc++abi/src/include --sysroot=../../build/linux/debian_bullseye_amd64-sysroot -fvisibility-inlines-hidden -Wno-invalid-offsetof -Wenum-compare-conditional -Wno-nullability-completeness  -c ../../third_party/protobuf/src/google/protobuf/dynamic_message.cc -o obj/third_party/protobuf/protobuf_full/dynamic_message.o
{"msg":"executable file `../../third_party/llvm-build/Release+Asserts/bin/clang++` not found in $PATH: No such file or directory","level":"error","time":"2025-04-15T18:57:50.993914Z"}
ninja: build stopped: subcommand failed.
RBE Stats: down 197 B, up 0 B, 1 non zero exit

Contents of rewrapper_linux.cfg:

canonicalize_working_dir=true
dial_timeout=10m
exec_strategy=remote
exec_timeout=10m
labels=type=compile,compiler=clang,lang=cpp
platform=container-image=docker://gcr.io/chops-public-images-prod/rbe/siso-chromium/linux@sha256:26de99218a1a8b527d4840490bcbf1690ee0b55c84316300b60776e6b3a03fe1,OSFamily=linux
reclient_timeout=10m

server_address=unix:///tmp/reproxy.sock 

--
You received this message because you are subscribed to the Google Groups "reclient-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reclient-user...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/reclient-users/e318fc64-8b11-41e1-94e4-4cad77c12af7n%40chromium.org.

[Marshall Greenblatt]

On Tue, Apr 15, 2025 at 3:11 PM Marshall Greenblatt <magree...@gmail.com> wrote:

On Tue, Apr 15, 2025 at 12:21 PM Marshall Greenblatt <magree...@gmail.com> wrote:

Hello!

I'm trying to build Chromium using reclient on Linux. I'm following the guide here (which mostly uses the default Chromium config + docker image). Upon execution the remote build is failing with "executable file `../../third_party/llvm-build/Release+Asserts/bin/clang++` not found in $PATH". Is there something more that I need to configure?

I should also mention that "third_party/llvm-build/Release+Asserts/bin/clang++" exists locally, and local fallback builds work when enabled.

[Michael Savigny]

Is that error happening on most/every action using rewrapper, or just that one?

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/reclient-users/CAFW9UJ991OT01j01VhVhy9jFr1a7BtsSLdXQB%2BBVVhOg5JjywA%40mail.gmail.com.

[Marshall Greenblatt]

On Apr 15, 2025, at 15:33, Michael Savigny <msav...@google.com> wrote:



Is that error happening on most/every action using rewrapper, or just that one?

Every clang compile action. I disabled local fallback to make the error more visible. The reclient setup does pass a simple “echo” test (like this).

[Yngve N. Pettersen]

Hi,

Make sure your rewrapper config file "input" field lists all the necessary clang executables and dependent files

My entry for Linux is:

inputs=third_party/llvm-build/Release+Asserts/bin/clang,third_party/llvm-build/Release+Asserts/bin/clang++,third_party/llvm-build/Release+Asserts/bin/clang-cl,third_party/llvm-build/Release+Asserts/lib/libstdc++.so.6

You may also have to check your remote clang wrapper.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/reclient-users/0C2A52DD-F94F-4FB6-80F2-F91F9BF42D2A%40gmail.com.

-- 
Sincerely,
Yngve N. Pettersen
Vivaldi Technologies AS

[Marshall Greenblatt]

On Tue, Apr 15, 2025 at 4:08 PM Yngve N. Pettersen <yn...@vivaldi.com> wrote:

Hi,

Make sure your rewrapper config file "input" field lists all the necessary clang executables and dependent files

My entry for Linux is:

inputs=third_party/llvm-build/Release+Asserts/bin/clang,third_party/llvm-build/Release+Asserts/bin/clang++,third_party/llvm-build/Release+Asserts/bin/clang-cl,third_party/llvm-build/Release+Asserts/lib/libstdc++.so.6

You may also have to check your remote clang wrapper.

Thanks! Are you building with the Chromium default docker image (e.g "gcr.io/chops-public-images-prod/rbe/siso-chromium/linux")? Is there a way to check/change the remote clang wrapper without creating a custom docker image?

[Yngve N. Pettersen]

Hi,

No, we have our own reclient backend configuration, installed directly on the hardware machines, the "cluster" is not big enough to even defend using containers of any kind.

[Marshall Greenblatt]

On Tue, Apr 15, 2025 at 4:22 PM Marshall Greenblatt <magree...@gmail.com> wrote:

On Tue, Apr 15, 2025 at 4:08 PM Yngve N. Pettersen <yn...@vivaldi.com> wrote:

Hi,

Make sure your rewrapper config file "input" field lists all the necessary clang executables and dependent files

My entry for Linux is:

inputs=third_party/llvm-build/Release+Asserts/bin/clang,third_party/llvm-build/Release+Asserts/bin/clang++,third_party/llvm-build/Release+Asserts/bin/clang-cl,third_party/llvm-build/Release+Asserts/lib/libstdc++.so.6

You may also have to check your remote clang wrapper.

Thanks! Are you building with the Chromium default docker image (e.g "gcr.io/chops-public-images-prod/rbe/siso-chromium/linux")? Is there a way to check/change the remote clang wrapper without creating a custom docker image?

To partially answer my own question, it looks like I can add "remote_wrapper=path/to/helper" in rewrapper_linux.cfg and it will execute "path/to/helper" remotely and pass it the complete clang command-line.

As a test, I set "remote_wrapper=third_party/llvm-build/Release+Asserts/bin/clang++" which successfully executed clang++, but then failed with:

clang++: error: no such file or directory: '../../third_party/llvm-build/Release+Asserts/bin/clang++'
clang++: error: no such file or directory: '../../third_party/protobuf/src/google/protobuf/dynamic_message.cc'

So, I suspect there may be some issue with the current working directory configuration (e.g. the use of "../../" in the clang command-line).

[Marshall Greenblatt]

On Tue, Apr 15, 2025 at 5:03 PM Marshall Greenblatt <magree...@gmail.com> wrote:

On Tue, Apr 15, 2025 at 4:22 PM Marshall Greenblatt <magree...@gmail.com> wrote:

On Tue, Apr 15, 2025 at 4:08 PM Yngve N. Pettersen <yn...@vivaldi.com> wrote:

Hi,

Make sure your rewrapper config file "input" field lists all the necessary clang executables and dependent files

My entry for Linux is:

inputs=third_party/llvm-build/Release+Asserts/bin/clang,third_party/llvm-build/Release+Asserts/bin/clang++,third_party/llvm-build/Release+Asserts/bin/clang-cl,third_party/llvm-build/Release+Asserts/lib/libstdc++.so.6

You may also have to check your remote clang wrapper.

Thanks! Are you building with the Chromium default docker image (e.g "gcr.io/chops-public-images-prod/rbe/siso-chromium/linux")? Is there a way to check/change the remote clang wrapper without creating a custom docker image?

To partially answer my own question, it looks like I can add "remote_wrapper=path/to/helper" in rewrapper_linux.cfg and it will execute "path/to/helper" remotely and pass it the complete clang command-line.

As a test, I set "remote_wrapper=third_party/llvm-build/Release+Asserts/bin/clang++" which successfully executed clang++, but then failed with:

clang++: error: no such file or directory: '../../third_party/llvm-build/Release+Asserts/bin/clang++'
clang++: error: no such file or directory: '../../third_party/protobuf/src/google/protobuf/dynamic_message.cc'

So, I suspect there may be some issue with the current working directory configuration (e.g. the use of "../../" in the clang command-line).

This fixes the issue for me:

1. Create a rewrite_wrapper binary with the attached (mostly) AI-generated code.

2. Add the following to your rewrapper_linux.cfg file:

# All paths relative to the src directory.

inputs=third_party/llvm-build/Release+Asserts/bin/clang,third_party/llvm-build/Release+Asserts/bin/clang++,third_party/llvm-build/Release+Asserts/bin/clang-cl,third_party/llvm-build/Release+Asserts/lib/libstdc++.so.6
toolchain_inputs=path/to/remote_wrapper
remote_wrapper=path/to/remote_wrapper

3. Build:

$ autoninja -C out/Debug_GN_x64 obj/third_party/protobuf/protobuf_full/dynamic_message.o
Proxy started successfully.
ninja: Entering directory `out/Debug_GN_x64'
[1/1] CXX obj/third_party/protobuf/protobuf_full/dynamic_message.o
RBE Stats: down 50.19 KB, up 19.18 KB, 1 remote execution

[Marshall Greenblatt]

On Tue, Apr 15, 2025 at 7:04 PM Marshall Greenblatt <magree...@gmail.com> wrote:

On Tue, Apr 15, 2025 at 5:03 PM Marshall Greenblatt <magree...@gmail.com> wrote:

On Tue, Apr 15, 2025 at 4:22 PM Marshall Greenblatt <magree...@gmail.com> wrote:

On Tue, Apr 15, 2025 at 4:08 PM Yngve N. Pettersen <yn...@vivaldi.com> wrote:

Hi,

Make sure your rewrapper config file "input" field lists all the necessary clang executables and dependent files

My entry for Linux is:

inputs=third_party/llvm-build/Release+Asserts/bin/clang,third_party/llvm-build/Release+Asserts/bin/clang++,third_party/llvm-build/Release+Asserts/bin/clang-cl,third_party/llvm-build/Release+Asserts/lib/libstdc++.so.6

You may also have to check your remote clang wrapper.

Thanks! Are you building with the Chromium default docker image (e.g "gcr.io/chops-public-images-prod/rbe/siso-chromium/linux")? Is there a way to check/change the remote clang wrapper without creating a custom docker image?

To partially answer my own question, it looks like I can add "remote_wrapper=path/to/helper" in rewrapper_linux.cfg and it will execute "path/to/helper" remotely and pass it the complete clang command-line.

As a test, I set "remote_wrapper=third_party/llvm-build/Release+Asserts/bin/clang++" which successfully executed clang++, but then failed with:

clang++: error: no such file or directory: '../../third_party/llvm-build/Release+Asserts/bin/clang++'
clang++: error: no such file or directory: '../../third_party/protobuf/src/google/protobuf/dynamic_message.cc'

So, I suspect there may be some issue with the current working directory configuration (e.g. the use of "../../" in the clang command-line).

This fixes the issue for me:

1. Create a rewrite_wrapper binary with the attached (mostly) AI-generated code.

s/rewrite_wrapper/remote_wrapper

[Gokul V G]

Hi Marshall,

Thank you so much for your assistance. it really helped me and i was able to proceed with the build. i am getting some issues with the build. 

Once the build is complete, the chrome browser is only showing in the loading screen. 

the commit id i used to build is 0f351bbd2617e0f775543271f34b7d241f8ebcdc

could you please help me resolve this issue?. it happened never before, usually chrome version build will be successful and working fine, we are facing this kind of issue for the first time. 

much  much appreciated for your response and help. if you need more information please let me know.

a big thanks in advance

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/reclient-users/CAFW9UJ9aiSnHYquRWC7TYn15J12GdjUasOu1KsN2E5e8Jz38sQ%40mail.gmail.com.

--

Gokul VG

Software Engineer III

Canada | USA | India

+91 8075481683

[Takuto Ikuta (生田 拓人)]

Hi Gokul,

It's great to see that the reclient setup is now working for you, Gokul. Regarding the issue you're encountering with the Chrome browser only showing a loading screen after the build is complete, it might be more appropriate to discuss behavior specific to Chrome itself on the chromi...@chromium.org mailing list, as this list, reclient-users, is primarily focused on build-related matters and issues concerning the reclient build system. This way, you'll likely reach a wider audience of developers who specialize in Chrome's runtime behavior and might be able to provide more targeted assistance.

Thanks,
Takuto

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/reclient-users/CAEGHyyLQVuDm0ZqZuFATUeQ7o-O0N-m3TgOx9JsYweamXNHsTA%40mail.gmail.com.

--

Takuto Ikuta

Software Engineer in Tokyo

Chrome Ops (chrome browser build team)