Various reclient issues

[Yngve N. Pettersen]

Hello all,

Here at Vivaldi we have just replaced our internal Goma system with
Reclient (Buildfarm backend, client side based on Engflow), but we are
unfortunately seeing some issues (current version is 0.134 from Chromium
124).

* Many devs report that the reproxy seems to fail, and all compilation
seems to start building exclusively local (default for us is local
fallback), resulting in a "too many fallbacks" error eventually. There are
some indications that the reproxy shut completely down, and I think I have
seen this happen after a build, not during a build. To some extent we can
work around it by forcing remote, but as there are target actions,
especially in angle, that are invalid (bug reported) one have to use
fallback to complete building. Debug builds seems particularly affected.

* The reproxy status does not seem to work, it does not detect the running
proxy, even when it is idle.

* I am also seeing a lot of errors when starting/stopping the reproxy, they
may be coming from the bootstrap, like:


W0507 11:15:03.276496 65296 rbeflag.go:58] Failed to set flag
compression_threshold to "0"

a full list of options

Auth is disabled, but:
W0507 11:15:03.276496 65296 auth.go:311] Failed to load credentials
cache file

* A webgui like Goma had would be nice to have

--
Sincerely,
Yngve N. Pettersen
Vivaldi Technologies AS

[Gokul V G]

Hi all,

We are also facing issues while building using Reclient, we are using reclient version  re_client_version:0.126.0.4aaef37-gomaip . 

FAILED: obj/components/page_content_annotations/core/features/page_content_annotations_switches.obj
..\..\buildtools\reclient\rewrapper -cfg=..\..\buildtools\reclient_cfgs\chromium-browser-clang\rewrapper_windows.cfg -inputs=build\config\unsafe_buffers_paths.txt -exec_root=C:\code\chromium\src\ -labels=type=compile,compiler=clang-cl,lang=cpp ..\..\third_party\llvm-build\Release+Asserts\bin\clang-cl.exe /c ../../components/page_content_annotations/core/page_content_annotations_switches.cc /Foobj/components/page_content_annotations/core/features/page_content_annotations_switches.obj /nologo /showIncludes:user /winsysroot../../third_party/depot_tools/win_toolchain/vs_files/e5d511ae93 -DIS_PAGE_CONTENT_ANNOTATIONS_FEATURES_IMPL -DUSE_AURA=1 -DOFFICIAL_BUILD -D_HAS_NODISCARD -D_CRT_NONSTDC_NO_WARNINGS -D_WINSOCK_DEPRECATED_NO_WARNINGS -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_EXTENSIVE "-DCR_CLANG_REVISION=\"llvmorg-19-init-2941-ga0b3dbaf-22\"" -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -DCR_LIBCXX_REVISION=80307e66e74bae927fb8709a549859e777e3bf0b -D__STD_C -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_DEPRECATE -D_ATL_NO_OPENGL -D_WINDOWS -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -DPSAPI_VERSION=2 -DWIN32 -D_SECURE_ATL -DWINAPI_FAMILY=WINAPI_FAMILY_DESKTOP_APP -DWIN32_LEAN_AND_MEAN -DNOMINMAX -D_UNICODE -DUNICODE -DNTDDI_VERSION=NTDDI_WIN10_NI -D_WIN32_WINNT=0x0A00 -DWINVER=0x0A00 -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DBASE_USE_PERFETTO_CLIENT_LIBRARY=1 -DSK_ENABLE_SKSL -DSK_UNTIL_CRBUG_1187654_IS_FIXED "-DSK_USER_CONFIG_HEADER=\"../../skia/config/SkUserConfig.h\"" -DSK_WIN_FONTMGR_NO_SIMULATIONS -DSK_DISABLE_LEGACY_INIT_DECODERS -DSK_SLUG_DISABLE_LEGACY_DESERIALIZE -DSK_DISABLE_LEGACY_VULKAN_BACKENDSEMAPHORE -DSK_DISABLE_LEGACY_CREATE_CHARACTERIZATION -DSK_DISABLE_LEGACY_VULKAN_MUTABLE_TEXTURE_STATE -DSK_CODEC_DECODES_JPEG -DSK_ENCODE_JPEG -DSK_ENCODE_PNG -DSK_ENCODE_WEBP -DGR_GL_FUNCTION_TYPE=__stdcall -DSK_GANESH "-DSK_GPU_WORKAROUNDS_HEADER=\"gpu/config/gpu_driver_bug_workaround_autogen.h\"" -DSK_GL -DSK_VULKAN=1 -DSK_GRAPHITE -DSK_DAWN -DVK_USE_PLATFORM_WIN32_KHR -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DUSE_CHROMIUM_ICU=1 -DU_ENABLE_TRACING=1 -DU_ENABLE_RESOURCE_TRACING=0 -DU_STATIC_IMPLEMENTATION -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -DCRASHPAD_ZLIB_SOURCE_EXTERNAL -I../.. -Igen -I../../buildtools/third_party/libc++ -I../../third_party/perfetto/include -Igen/third_party/perfetto/build_config -Igen/third_party/perfetto -I../../net/third_party/quiche/overrides -I../../net/third_party/quiche/src/quiche/common/platform/default -I../../net/third_party/quiche/src -I../../third_party/skia -Igen/third_party/skia -I../../third_party/wuffs/src/release/c -I../../third_party/vulkan/include -I../../third_party/vulkan-deps/vulkan-headers/src/include -Igen/third_party/dawn/include -I../../third_party/dawn/include -I../../base/allocator/partition_allocator/src -Igen/base/allocator/partition_allocator/src -I../../third_party/abseil-cpp -I../../third_party/boringssl/src/include -I../../third_party/protobuf/src -Igen/protoc_out -I../../third_party/ipcz/include -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -I../../third_party/ced/src -I../../third_party/crashpad/crashpad -I../../third_party/crashpad/crashpad/compat/win -I../../third_party/zlib /W4 -Wextra-semi -Wunreachable-code-aggressive -Wthread-safety -Wno-missing-field-initializers -Wno-unused-parameter -Wno-psabi -Wloop-analysis -Wno-unneeded-internal-declaration -Wno-unused-function -Wuninitialized -Wno-unused-variable -Wno-nonportable-include-path -Wno-ignored-pragma-optimize -Wno-deprecated-builtins -Wno-bitfield-constant-conversion -Wno-deprecated-this-capture -Wno-invalid-offsetof -Wno-vla-extension -Wno-thread-safety-reference-return -Wshadow /WX -fno-delete-null-pointer-checks -fno-ident -fcolor-diagnostics -fmerge-all-constants -fcrash-diagnostics-dir=../../tools/clang/crashreports -mllvm -instcombine-lower-dbg-declare=0 -mllvm -split-threshold-for-reg-with-hint=0 /clang:-ffp-contract=off -flto=thin -fsplit-lto-unit -fwhole-program-vtables -fcomplete-member-pointers /Gy /FS /bigobj /utf-8 /Zc:twoPhase -ffile-reproducible /Zc:sizedDealloc- /D__WRL_ENABLE_FUNCTION_STATICS__ -fmsc-version=1934 -m32 -msse3 -ffile-compilation-dir=. -no-canonical-prefixes -ftrivial-auto-var-init=pattern /O2 /Oy- /Zc:inline /Gw /clang:-fno-math-errno /Z7 -gno-codeview-command-line -gcodeview-ghash -Xclang -fuse-ctor-homing /guard:cf /MT -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ref-template-as-trivial-member -Xclang -plugin-arg-find-bad-constructs -Xclang check-stack-allocated -Xclang -plugin-arg-find-bad-constructs -Xclang check-allow-auto-typedefs-better-nested -Xclang -plugin-arg-find-bad-constructs -Xclang check-raw-ptr-to-stack-allocated -Xclang -plugin-arg-find-bad-constructs -Xclang disable-check-raw-ptr-to-stack-allocated-error -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=/third_party/dawn/ -Xclang -add-plugin -Xclang unsafe-buffers -Xclang -plugin-arg-unsafe-buffers -Xclang ../../build/config/unsafe_buffers_paths.txt -Wno-redundant-parens -DPROTOBUF_ALLOW_DEPRECATED=1 -Wno-redundant-parens -Wenum-compare-conditional -Wno-c++11-narrowing-const-reference /std:c++20 -Wno-trigraphs /TP /GR- -I../../third_party/libc++/src/include /Fd"obj/components/page_content_annotations/core/features_cc.pdb"
reclient[040432f7-e36f-49f9-9c9a-cf2b20e555b8]: LocalErrorResultStatus: this build has encountered too many local fallbacks. This means that the ratio of actions that failed remotely is equal to or above the preconfigured threshold of 50.0%[5292/36649] CXX obj/components/viz/common/common/solid_color_draw_quad.obj
Any idea why this is happening? Requesting dev team for help,any help would be appreciated, Thanks in adavance

[Philipp Wollermann]

Hi Gokul,

You should see error messages explaining why these remote executions fail in the reproxy.INFO log file, which should be in the ".reproxy_tmp/logs" sub-directory of your output directory. So, for example, if my output directory is called "debug", I could find the logs in "out/debug/.reproxy_tmp/logs".

Hope that helps!

Philipp

--
You received this message because you are subscribed to the Google Groups "reclient-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reclient-user...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/reclient-users/f0ecab6d-43ea-4fad-8ead-1541bf71e7a7n%40chromium.org.

[Gokul V G]

Hi Philipp,

Thank you so much for your response, i have checked the log file in the mentioned directory, but I couldn't find the reason from logs, the logs are the same as the error message I got from the command prompt
providing the logs for your response.It will be a great help if you can provide me any insights why this is happening, I tried using latest re_client_version:0.146.0.0c7ca4be-gomaip' instead of 126, But issue still persists.

LOG details from C:\code\chromium\src\out\Release\.reproxy_tmp\logs\20240528T094338.022053_2d57313c-ed05-4046-995c-e3f54933bd57\reproxy_outerr.log

E0528 03:10:08.805575    6356 server.go:545] 1e72c103-28e5-473a-97ca-76755d32331c: Execution failed with &{ExitCode:35 Status:LocalErrorResultStatus Err:this build has encountered too many local fallbacks. This means that the ratio of actions that failed remotely is equal to or above the preconfigured threshold of 50.0%}
E0528 03:10:08.805641    6356 server.go:545] 322ac85a-71d6-4001-962e-c938cbc5a252: Execution failed with &{ExitCode:35 Status:LocalErrorResultStatus Err:this build has encountered too many local fallbacks. This means that the ratio of actions that failed remotely is equal to or above the preconfigured threshold of 50.0%}
E0528 03:10:08.805641    6356 server.go:545] 91e531bf-2c2b-4f88-ad6a-c7263533b939: Execution failed with &{ExitCode:35 Status:LocalErrorResultStatus Err:this build has encountered too many local fallbacks. This means that the ratio of actions that failed remotely is equal to or above the preconfigured threshold of 50.0%}
E0528 03:10:08.805641    6356 server.go:545] 33f7da59-b20d-4f66-ae0b-2469c0c07f24: Execution failed with &{ExitCode:35 Status:LocalErrorResultStatus Err:this build has encountered too many local fallbacks. This means that the ratio of actions that failed remotely is equal to or above the preconfigured threshold of 50.0%}
E0528 03:10:08.805641    6356 server.go:545] 2aad61cc-240e-4fa0-a59e-648ba7a56554: Execution failed with &{ExitCode:35 Status:LocalErrorResultStatus Err:this build has encountered too many local fallbacks. This means that the ratio of actions that failed remotely is equal to or above the preconfigured threshold of 50.0%}
E0528 03:10:08.805641    6356 server.go:545] c7ba0f79-3156-4959-8a22-4aef645f08f6: Execution failed with &{ExitCode:35 Status:LocalErrorResultStatus Err:this build has encountered too many local fallbacks. This means that the ratio of actions that failed remotely is equal to or above the preconfigured threshold of 50.0%}
E0528 03:10:08.806254    6356 server.go:545] d47fcf25-ca01-4843-b9a0-a12fffbc449a: Execution failed with &{ExitCode:35 Status:LocalErrorResultStatus Err:this build has encountered too many local fallbacks. This means that the ratio of actions that failed remotely is equal to or above the preconfigured threshold of 50.0%}
E0528 03:10:08.806254    6356 server.go:545] 9d37110f-465c-4aa9-b2f3-984b1d1cd488: Execution failed with &{ExitCode:35 Status:LocalErrorResultStatus Err:this build has encountered too many local fallbacks. This means that the ratio of actions that failed remotely is equal to or above the preconfigured threshold of 50.0%}
E0528 03:10:08.806254    6356 server.go:545] 740c326f-4978-4998-8297-5a21cfb95315: Execution failed with &{ExitCode:35 Status:LocalErrorResultStatus Err:this build has encountered too many local fallbacks. This means that the ratio of actions that failed remotely is equal to or above the preconfigured threshold of 50.0%}
E0528 03:10:08.806254    6356 server.go:545] 0f07eaa7-6e4a-4e93-98fb-ae4e40127024: Execution failed with &{ExitCode:35 Status:LocalErrorResultStatus Err:this build has encountered too many local fallbacks. This means that the ratio of actions that failed remotely is equal to or above the preconfigured threshold of 50.0%}
E0528 03:10:08.806254    6356 server.go:545] 7ae78257-74ad-4e58-a1ca-41d2c0f14ea9: Execution failed with &{ExitCode:1 Status:NonZeroExitResultStatus Err:<nil>}
E0528 03:10:08.806877    6356 server.go:545] f2aeaa13-14a3-4080-90a5-a929cbe82e69: Execution failed with &{ExitCode:1 Status:NonZeroExitResultStatus Err:<nil>}
E0528 03:10:08.806877    6356 server.go:545] 139e6fd2-2e4e-4160-8bdb-ef7d8e35690c: Execution failed with &{ExitCode:1 Status:NonZeroExitResultStatus Err:<nil>}
E0528 03:10:18.600810    6356 depsscannerclient.go:63] depsscannerclient stderr: 





Error in CMD

FAILED: obj/third_party/libjpeg_turbo/libjpeg/jdpostct.obj
..\..\buildtools\reclient\rewrapper -cfg=..\..\buildtools\reclient_cfgs\chromium-browser-clang\rewrapper_windows.cfg -inputs=build\config\unsafe_buffers_paths.txt -exec_root=C:\code\chromium\src\ -labels=type=compile,compiler=clang-cl,lang=cpp ..\..\third_party\llvm-build\Release+Asserts\bin\clang-cl.exe /c ../../third_party/libjpeg_turbo/jdpostct.c /nologo /showIncludes:user "-imsvc../../../../../Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.39.33519/include" "-imsvc../../../../../Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.39.33519/ATLMFC/include" "-imsvc../../../../../Program Files/Microsoft Visual Studio/2022/Community/VC/Auxiliary/VS/include" "-imsvc../../../../../Program Files (x86)/Windows Kits/10/include/10.0.22621.0/ucrt" "-imsvc../../../../../Program Files (x86)/Windows Kits/10/include/10.0.22621.0/um" "-imsvc../../../../../Program Files (x86)/Windows Kits/10/include/10.0.22621.0/shared" "-imsvc../../../../../Program Files (x86)/Windows Kits/10/include/10.0.22621.0/winrt" "-imsvc../../../../../Program Files (x86)/Windows Kits/10/include/10.0.22621.0/cppwinrt" "-imsvc../../../../../Program Files (x86)/Windows Kits/NETFXSDK/4.8/include/um" -DWITH_SIMD -DNO_GETENV -DNO_PUTENV -DUSE_AURA=1 -DOFFICIAL_BUILD -D_HAS_NODISCARD -D_CRT_NONSTDC_NO_WARNINGS -D_WINSOCK_DEPRECATED_NO_WARNINGS -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_EXTENSIVE "-DCR_CLANG_REVISION=\"llvmorg-19-init-2941-ga0b3dbaf-22\"" -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -DCR_LIBCXX_REVISION=80307e66e74bae927fb8709a549859e777e3bf0b -D__STD_C -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_DEPRECATE -D_ATL_NO_OPENGL -D_WINDOWS -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -DPSAPI_VERSION=2 -DWIN32 -D_SECURE_ATL -DWINAPI_FAMILY=WINAPI_FAMILY_DESKTOP_APP -DWIN32_LEAN_AND_MEAN -DNOMINMAX -D_UNICODE -DUNICODE -DNTDDI_VERSION=NTDDI_WIN10_NI -D_WIN32_WINNT=0x0A00 -DWINVER=0x0A00 -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DMANGLE_JPEG_NAMES -I../.. -Igen -I../../buildtools/third_party/libc++ -I../../third_party/libjpeg_turbo /W4 -Wextra-semi -Wunreachable-code-aggressive -Wthread-safety -Wno-missing-field-initializers -Wno-unused-parameter -Wno-psabi -Wloop-analysis -Wno-unneeded-internal-declaration -Wno-unused-function -Wuninitialized -Wno-unused-variable -Wno-nonportable-include-path -Wno-ignored-pragma-optimize -Wno-deprecated-builtins -Wno-bitfield-constant-conversion -Wno-deprecated-this-capture -Wno-invalid-offsetof -Wno-vla-extension -Wno-thread-safety-reference-return -Wshadow /WX -fno-delete-null-pointer-checks -fno-ident -fcolor-diagnostics -fmerge-all-constants -fcrash-diagnostics-dir=../../tools/clang/crashreports -mllvm -instcombine-lower-dbg-declare=0 -mllvm -split-threshold-for-reg-with-hint=0 /clang:-ffp-contract=off -flto=thin -fsplit-lto-unit -fwhole-program-vtables -fcomplete-member-pointers /Gy /FS /bigobj /utf-8 /Zc:twoPhase -ffile-reproducible /Zc:sizedDealloc- /D__WRL_ENABLE_FUNCTION_STATICS__ -fmsc-version=1934 -m32 -msse3 -ffile-compilation-dir=. -no-canonical-prefixes -ftrivial-auto-var-init=pattern /O2 /Oy- /Zc:inline /Gw /clang:-fno-math-errno /Z7 -gno-codeview-command-line -gcodeview-ghash -Xclang -fuse-ctor-homing /guard:cf /MT -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ref-template-as-trivial-member -Xclang -plugin-arg-find-bad-constructs -Xclang check-stack-allocated -Xclang -plugin-arg-find-bad-constructs -Xclang check-allow-auto-typedefs-better-nested -Xclang -plugin-arg-find-bad-constructs -Xclang check-raw-ptr-to-stack-allocated -Xclang -plugin-arg-find-bad-constructs -Xclang disable-check-raw-ptr-to-stack-allocated-error -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=/third_party/dawn/ -Xclang -add-plugin -Xclang unsafe-buffers -Xclang -plugin-arg-unsafe-buffers -Xclang ../../build/config/unsafe_buffers_paths.txt /std:c11 /TC /Foobj/third_party/libjpeg_turbo/libjpeg/jdpostct.obj /Fd"obj/third_party/libjpeg_turbo/libjpeg_c.pdb"
reclient[9b1fcf93-b14e-40b1-830e-f3a775c82f34]: LocalErrorResultStatus: this build has encountered too many local fallbacks. This means that the ratio of actions that failed remotely is equal to or above the preconfigured threshold of 50.0%[11520/60480] CC obj/third_party/libaom/libaom_intrinsics_avx2/highbd_quantize_intrin_avx2.obj

--

Your Sincerely,

Gokul VG

Software Developer

AOT Technologies Pvt. Ltd.

3rd Floor, Amstor Building, Technopark,

Thiruvananthapuram, Kerala, India - 695581

http://www.aot-technologies.com 

"Learn something new everyday to ignite your inner self"

[Ben Segall]

Is there a C:\code\chromium\src\out\Release\.reproxy_tmp\logs\20240528T094338.022053_2d57313c-ed05-4046-995c-e3f54933bd57\reproxy.INFO file?

This should have more information. You will want to look for messages before the too many fallback messages.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/reclient-users/CAEGHyy%2Bcv%3DKybtyS_xkOt4VgmEGw4LJAr9NSLR27hSCeOupZsQ%40mail.gmail.com.