[llvm-dev] LLD issue on a massively parallel build machine
Itaru Kitayama via llvm-dev
FAILED: bin/clang-11
: && /home/usr4/c74014i/opt/clang/current/bin/clang++ -stdlib=libc++ -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3 -DNDEBUG -stdlib=libc++ -fuse-ld=lld -Wl,--color-diagnostics -Wl,-allow-shlib-undefined -Wl,--export-dynamic -Wl,-O3 tools/clang/tools/driver/CMakeFiles/clang.dir/driver.cpp.o tools/clang/tools/driver/CMakeFiles/clang.dir/cc1_main.cpp.o tools/clang/tools/driver/CMakeFiles/clang.dir/cc1as_main.cpp.o tools/clang/tools/driver/CMakeFiles/clang.dir/cc1gen_reproducer_main.cpp.o -o bin/clang-11 -Wl,-rpath,"\$ORIGIN/../lib" lib/libLLVMAArch64CodeGen.a lib/libLLVMAArch64AsmParser.a lib/libLLVMAArch64Desc.a lib/libLLVMAArch64Disassembler.a lib/libLLVMAArch64Info.a lib/libLLVMAArch64Utils.a lib/libLLVMAMDGPUCodeGen.a lib/libLLVMAMDGPUAsmParser.a lib/libLLVMAMDGPUDesc.a lib/libLLVMAMDGPUDisassembler.a lib/libLLVMAMDGPUInfo.a lib/libLLVMAMDGPUUtils.a lib/libLLVMARMCodeGen.a lib/libLLVMARMAsmParser.a lib/libLLVMARMDesc.a lib/libLLVMARMDisassembler.a lib/libLLVMARMInfo.a lib/libLLVMARMUtils.a lib/libLLVMAVRCodeGen.a lib/libLLVMAVRAsmParser.a lib/libLLVMAVRDesc.a lib/libLLVMAVRDisassembler.a lib/libLLVMAVRInfo.a lib/libLLVMBPFCodeGen.a lib/libLLVMBPFAsmParser.a lib/libLLVMBPFDesc.a lib/libLLVMBPFDisassembler.a lib/libLLVMBPFInfo.a lib/libLLVMHexagonCodeGen.a lib/libLLVMHexagonAsmParser.a lib/libLLVMHexagonDesc.a lib/libLLVMHexagonDisassembler.a lib/libLLVMHexagonInfo.a lib/libLLVMLanaiCodeGen.a lib/libLLVMLanaiAsmParser.a lib/libLLVMLanaiDesc.a lib/libLLVMLanaiDisassembler.a lib/libLLVMLanaiInfo.a lib/libLLVMMipsCodeGen.a lib/libLLVMMipsAsmParser.a lib/libLLVMMipsDesc.a lib/libLLVMMipsDisassembler.a lib/libLLVMMipsInfo.a lib/libLLVMMSP430CodeGen.a lib/libLLVMMSP430AsmParser.a lib/libLLVMMSP430Desc.a lib/libLLVMMSP430Disassembler.a lib/libLLVMMSP430Info.a lib/libLLVMNVPTXCodeGen.a lib/libLLVMNVPTXDesc.a lib/libLLVMNVPTXInfo.a lib/libLLVMPowerPCCodeGen.a lib/libLLVMPowerPCAsmParser.a lib/libLLVMPowerPCDesc.a lib/libLLVMPowerPCDisassembler.a lib/libLLVMPowerPCInfo.a lib/libLLVMRISCVCodeGen.a lib/libLLVMRISCVAsmParser.a lib/libLLVMRISCVDesc.a lib/libLLVMRISCVDisassembler.a lib/libLLVMRISCVInfo.a lib/libLLVMRISCVUtils.a lib/libLLVMSparcCodeGen.a lib/libLLVMSparcAsmParser.a lib/libLLVMSparcDesc.a lib/libLLVMSparcDisassembler.a lib/libLLVMSparcInfo.a lib/libLLVMSystemZCodeGen.a lib/libLLVMSystemZAsmParser.a lib/libLLVMSystemZDesc.a lib/libLLVMSystemZDisassembler.a lib/libLLVMSystemZInfo.a lib/libLLVMWebAssemblyCodeGen.a lib/libLLVMWebAssemblyAsmParser.a lib/libLLVMWebAssemblyDesc.a lib/libLLVMWebAssemblyDisassembler.a lib/libLLVMWebAssemblyInfo.a lib/libLLVMX86CodeGen.a lib/libLLVMX86AsmParser.a lib/libLLVMX86Desc.a lib/libLLVMX86Disassembler.a lib/libLLVMX86Info.a lib/libLLVMX86Utils.a lib/libLLVMXCoreCodeGen.a lib/libLLVMXCoreDesc.a lib/libLLVMXCoreDisassembler.a lib/libLLVMXCoreInfo.a lib/libLLVMAnalysis.a lib/libLLVMCodeGen.a lib/libLLVMCore.a lib/libLLVMipo.a lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a lib/libLLVMInstrumentation.a lib/libLLVMMC.a lib/libLLVMMCParser.a lib/libLLVMObjCARCOpts.a lib/libLLVMOption.a lib/libLLVMScalarOpts.a lib/libLLVMSupport.a lib/libLLVMTransformUtils.a lib/libLLVMVectorize.a -lpthread lib/libclangBasic.a lib/libclangCodeGen.a lib/libclangDriver.a lib/libclangFrontend.a lib/libclangFrontendTool.a lib/libclangSerialization.a lib/libLLVMAArch64Desc.a lib/libLLVMAArch64Info.a lib/libLLVMAArch64Utils.a lib/libLLVMMIRParser.a lib/libLLVMAMDGPUDesc.a lib/libLLVMAMDGPUInfo.a lib/libLLVMAMDGPUUtils.a lib/libLLVMARMDesc.a lib/libLLVMARMInfo.a lib/libLLVMARMUtils.a lib/libLLVMHexagonDesc.a lib/libLLVMHexagonInfo.a lib/libLLVMLanaiDesc.a lib/libLLVMLanaiInfo.a lib/libLLVMSystemZDesc.a lib/libLLVMSystemZInfo.a lib/libLLVMWebAssemblyDesc.a lib/libLLVMWebAssemblyInfo.a lib/libLLVMCFGuard.a lib/libLLVMGlobalISel.a lib/libLLVMAsmPrinter.a lib/libLLVMDebugInfoDWARF.a lib/libLLVMSelectionDAG.a lib/libLLVMMCDisassembler.a lib/libclangCodeGen.a lib/libLLVMCoverage.a lib/libLLVMLTO.a lib/libLLVMObjCARCOpts.a lib/libLLVMPasses.a lib/libLLVMCodeGen.a lib/libLLVMTarget.a lib/libLLVMCoroutines.a lib/libLLVMipo.a lib/libLLVMInstrumentation.a lib/libLLVMVectorize.a lib/libLLVMBitWriter.a lib/libLLVMIRReader.a lib/libLLVMAsmParser.a lib/libLLVMLinker.a lib/libLLVMScalarOpts.a lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a lib/libclangRewriteFrontend.a lib/libclangARCMigrate.a lib/libclangStaticAnalyzerFrontend.a lib/libclangStaticAnalyzerCheckers.a lib/libclangStaticAnalyzerCore.a lib/libclangCrossTU.a lib/libclangIndex.a lib/libclangFrontend.a lib/libclangDriver.a lib/libLLVMOption.a lib/libclangParse.a lib/libclangSerialization.a lib/libclangSema.a lib/libclangAnalysis.a lib/libclangASTMatchers.a lib/libclangEdit.a lib/libclangFormat.a lib/libclangToolingInclusions.a lib/libclangToolingCore.a lib/libclangAST.a lib/libLLVMFrontendOpenMP.a lib/libLLVMTransformUtils.a lib/libLLVMAnalysis.a lib/libLLVMProfileData.a lib/libLLVMObject.a lib/libLLVMMCParser.a lib/libLLVMBitReader.a lib/libLLVMTextAPI.a lib/libclangRewrite.a lib/libclangLex.a lib/libclangBasic.a lib/libLLVMCore.a lib/libLLVMRemarks.a lib/libLLVMBitstreamReader.a lib/libLLVMMC.a lib/libLLVMBinaryFormat.a lib/libLLVMDebugInfoCodeView.a lib/libLLVMDebugInfoMSF.a lib/libLLVMSupport.a -lz -lrt -ldl -ltinfo -lpthread -lm lib/libLLVMDemangle.a && :
ld.lld: error: failed to open bin/clang-11: Cannot allocate memory
Alex Brachet-Mialot via llvm-dev
_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Alexandre Ganea via llvm-dev
Alex :
Can you please try `numactl` or `taskset` after https://github.com/llvm/llvm-project/commit/09158252f777c2e2f06a86b154c44abcbcf9bb74 ?
There was a tiny bug in how sched_getaffinity() was used, see: https://reviews.llvm.org/D75153#1942336
De : Alex Brachet-Mialot <alexbrac...@gmail.com>
Envoyé : March 28, 2020 12:11 PM
À : Itaru Kitayama <itaru.k...@gmail.com>
Cc : LLVM Dev <llvm...@lists.llvm.org>; Alexandre Ganea <alexand...@ubisoft.com>; Fāng-ruì Sòng <mas...@google.com>
Objet : Re: [llvm-dev] LLD issue on a massively parallel build machine
Nemanja Ivanovic via llvm-dev
Alex Brachet-Mialot via llvm-dev
> Can you please try `numactl` or `taskset` after https://github.com/llvm/llvm-project/commit/09158252f777c2e2f06a86b154c44abcbcf9bb74 ?
It works! I can finally use lld on that machine :)
Itaru Kitayama via llvm-dev
FAILED: bin/lld
: && /home/usr4/c74014i/opt/clang/current/bin/clang++ -stdlib=libc++ -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG -stdlib=libc++ -fuse-ld=lld -Wl,--color-diagnostics -Wl,-allow-shlib-undefined -Wl,--export-dynamic -Wl,-O3 tools/lld/tools/lld/CMakeFiles/lld.dir/lld.cpp.o -o bin/lld -Wl,-rpath,"\$ORIGIN/../lib" lib/libLLVMSupport.a -lpthread lib/liblldCommon.a lib/liblldCOFF.a lib/liblldDriver.a lib/liblldELF.a lib/liblldMinGW.a lib/liblldWasm.a lib/liblldMachO.a lib/liblldReaderWriter.a lib/liblldYAML.a lib/liblldCore.a lib/liblldCOFF.a lib/libLLVMDebugInfoPDB.a lib/libLLVMLibDriver.a lib/libLLVMWindowsManifest.a /usr/lib64/libxml2.so lib/liblldCommon.a lib/libLLVMOption.a lib/libLLVMAArch64CodeGen.a lib/libLLVMAArch64AsmParser.a lib/libLLVMAArch64Disassembler.a lib/libLLVMAArch64Desc.a lib/libLLVMAArch64Info.a lib/libLLVMAArch64Utils.a lib/libLLVMAMDGPUCodeGen.a lib/libLLVMMIRParser.a lib/libLLVMAMDGPUAsmParser.a lib/libLLVMAMDGPUDisassembler.a lib/libLLVMAMDGPUDesc.a lib/libLLVMAMDGPUInfo.a lib/libLLVMAMDGPUUtils.a lib/libLLVMARMCodeGen.a lib/libLLVMARMAsmParser.a lib/libLLVMARMDisassembler.a lib/libLLVMARMDesc.a lib/libLLVMARMInfo.a lib/libLLVMARMUtils.a lib/libLLVMAVRCodeGen.a lib/libLLVMAVRAsmParser.a lib/libLLVMAVRDesc.a lib/libLLVMAVRDisassembler.a lib/libLLVMAVRInfo.a lib/libLLVMBPFCodeGen.a lib/libLLVMBPFAsmParser.a lib/libLLVMBPFDesc.a lib/libLLVMBPFDisassembler.a lib/libLLVMBPFInfo.a lib/libLLVMHexagonCodeGen.a lib/libLLVMHexagonAsmParser.a lib/libLLVMHexagonDisassembler.a lib/libLLVMHexagonDesc.a lib/libLLVMHexagonInfo.a lib/libLLVMLanaiCodeGen.a lib/libLLVMLanaiAsmParser.a lib/libLLVMLanaiDisassembler.a lib/libLLVMLanaiDesc.a lib/libLLVMLanaiInfo.a lib/libLLVMMipsCodeGen.a lib/libLLVMMipsAsmParser.a lib/libLLVMMipsDesc.a lib/libLLVMMipsDisassembler.a lib/libLLVMMipsInfo.a lib/libLLVMMSP430CodeGen.a lib/libLLVMMSP430AsmParser.a lib/libLLVMMSP430Desc.a lib/libLLVMMSP430Disassembler.a lib/libLLVMMSP430Info.a lib/libLLVMNVPTXCodeGen.a lib/libLLVMNVPTXDesc.a lib/libLLVMNVPTXInfo.a lib/libLLVMPowerPCCodeGen.a lib/libLLVMPowerPCAsmParser.a lib/libLLVMPowerPCDesc.a lib/libLLVMPowerPCDisassembler.a lib/libLLVMPowerPCInfo.a lib/libLLVMRISCVCodeGen.a lib/libLLVMRISCVAsmParser.a lib/libLLVMRISCVDesc.a lib/libLLVMRISCVDisassembler.a lib/libLLVMRISCVInfo.a lib/libLLVMRISCVUtils.a lib/libLLVMSparcCodeGen.a lib/libLLVMSparcAsmParser.a lib/libLLVMSparcDesc.a lib/libLLVMSparcDisassembler.a lib/libLLVMSparcInfo.a lib/libLLVMSystemZCodeGen.a lib/libLLVMSystemZAsmParser.a lib/libLLVMSystemZDisassembler.a lib/libLLVMSystemZDesc.a lib/libLLVMSystemZInfo.a lib/libLLVMWebAssemblyCodeGen.a lib/libLLVMWebAssemblyAsmParser.a lib/libLLVMWebAssemblyDisassembler.a lib/libLLVMWebAssemblyDesc.a lib/libLLVMWebAssemblyInfo.a lib/libLLVMX86CodeGen.a lib/libLLVMCFGuard.a lib/libLLVMGlobalISel.a lib/libLLVMX86AsmParser.a lib/libLLVMX86Desc.a lib/libLLVMX86Disassembler.a lib/libLLVMX86Info.a lib/libLLVMX86Utils.a lib/libLLVMXCoreCodeGen.a lib/libLLVMAsmPrinter.a lib/libLLVMDebugInfoDWARF.a lib/libLLVMSelectionDAG.a lib/libLLVMXCoreDesc.a lib/libLLVMXCoreDisassembler.a lib/libLLVMMCDisassembler.a lib/libLLVMXCoreInfo.a lib/libLLVMLTO.a lib/libLLVMObjCARCOpts.a lib/libLLVMPasses.a lib/libLLVMCodeGen.a lib/libLLVMTarget.a lib/libLLVMCoroutines.a lib/libLLVMipo.a lib/libLLVMBitWriter.a lib/libLLVMScalarOpts.a lib/libLLVMVectorize.a lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a lib/libLLVMLinker.a lib/libLLVMFrontendOpenMP.a lib/libLLVMIRReader.a lib/libLLVMAsmParser.a lib/libLLVMInstrumentation.a lib/libLLVMTransformUtils.a lib/libLLVMAnalysis.a lib/libLLVMObject.a lib/libLLVMBitReader.a lib/libLLVMMCParser.a lib/libLLVMMC.a lib/libLLVMDebugInfoCodeView.a lib/libLLVMDebugInfoMSF.a lib/libLLVMTextAPI.a lib/libLLVMProfileData.a lib/libLLVMCore.a lib/libLLVMBinaryFormat.a lib/libLLVMRemarks.a lib/libLLVMBitstreamReader.a lib/libLLVMSupport.a -lz -lrt -ldl -ltinfo -lpthread -lm lib/libLLVMDemangle.a && cd /tmp/build/202003290405/tools/lld/tools/lld && /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E create_symlink lld /tmp/build/202003290405/./bin/lld-link && cd /tmp/build/202003290405/tools/lld/tools/lld && /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E create_symlink lld /tmp/build/202003290405/./bin/ld.lld && cd /tmp/build/202003290405/tools/lld/tools/lld && /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E create_symlink lld /tmp/build/202003290405/./bin/ld64.lld && cd /tmp/build/202003290405/tools/lld/tools/lld && /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E create_symlink lld /tmp/build/202003290405/./bin/wasm-ld
terminating with uncaught exception of type std::__1::system_error: thread constructor failed: Resource temporarily unavailable
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
#0 0x00000000007924f4 PrintStackTraceSignalHandler(void*) (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x7924f4)
#1 0x00000000007902ce llvm::sys::RunSignalHandlers() (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x7902ce)
#2 0x0000000000792bd5 SignalHandler(int) (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x792bd5)
#3 0x00007fddd54bb630 __restore_rt (/lib64/libpthread.so.0+0xf630)
#4 0x00007fddd3b40377 raise /usr/src/debug/glibc-2.17-c758a686/signal/../nptl/sysdeps/unix/sysv/linux/raise.c:55:0
#5 0x00007fddd3b41a68 abort /usr/src/debug/glibc-2.17-c758a686/stdlib/abort.c:92:0
#6 0x00007fddd411ae8b (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2be8b)
#7 0x00007fddd40fdb9f demangling_terminate_handler() (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0xeb9f)
#8 0x00007fddd411a0a3 std::__terminate(void (*)()) (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2b0a3)
#9 0x00007fddd411ce06 (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2de06)
#10 0x00007fddd411cd9f (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2dd9f)
#11 0x00007fddd43bded4 std::__1::__throw_system_error(int, char const*) (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++.so.1+0x90ed4)
#12 0x0000000002a7a316 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, llvm::parallel::detail::(anonymous namespace)::ThreadPoolExecutor::ThreadPoolExecutor(llvm::ThreadPoolStrategy)::'lambda'()> >(void*) (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x2a7a316)
#13 0x00007fddd54b3ea5 start_thread /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:307:0
#14 0x00007fddd3c088cd clone /usr/src/debug////////glibc-2.17-c758a686/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:113:0
clang-11: error: unable to execute command: Aborted
clang-11: error: linker command failed due to signal (use -v to see invocation)
Alex Brachet-Mialot via llvm-dev
Itaru Kitayama via llvm-dev
total used free shared buff/cache available
Mem: 376 149 20 1 207 225
Swap: 3 0 3
Itaru Kitayama via llvm-dev
Alexandre Ganea via llvm-dev
Would `taskset -c 0-3 ninja check-all -j4` work?
De : Itaru Kitayama <itaru.k...@gmail.com>
Envoyé : March 28, 2020 3:37 PM
À : Alex Brachet-Mialot <alexbrac...@gmail.com>
Cc : Alexandre Ganea <alexand...@ubisoft.com>; LLVM Dev <llvm...@lists.llvm.org>; Nemanja Ivanovic <nemanj...@gmail.com>
Itaru Kitayama via llvm-dev
Alex Brachet-Mialot via llvm-dev
Itaru Kitayama via llvm-dev
Nemanja Ivanovic via llvm-dev
Fangrui Song via llvm-dev
>Glad you got it working.
>My suggestion about LLVM_ENABLE_THREADS didn't work because you didn't
>apply it when building the build linker.
>
>When you don't have the ability to rebuild the build compiler, this doesn't
>apply. In those cases, I end up doing a dirty hack where I use a wrapper
>script with something like:
>
><path-to-build-compiler> "$@" -Wl,--no-threads
>
>To invoke the build compiler. That way you still get full parallelism while
>compiling and linking, but you prevent LLD from using threads.
https://reviews.llvm.org/D76885 (committed yesterday) changed
--no-threads to --threads={1,2,...} (--no-threads is used rarely (GNU ld
does not support it) so I'd rather not keep the option for
compatibility).
You may try -Wl,--threads=1
>On Sat., Mar. 28, 2020, 4:19 p.m. Itaru Kitayama, <itaru.k...@gmail.com>
>wrote:
>
>> Good news, I was able to use up to 37 cores for LLVM build with LLD.
>> The build speed, did not measure precisely though, is comparable to the
>> build with GNU ld case.
>>
>> Thank you all for your help!
>>
>> On Sun, Mar 29, 2020 at 5:04 AM Alex Brachet-Mialot <
>> alexbrac...@gmail.com> wrote:
>>
>>> Yes unfortunately that would limit you to 4 cores.
>>>
>>> There’s no easy way to use lld with —no-threads with our build system at
>>> the moment unfortunately.
>>>
>>> I also had just been using ld.gold, but will switch now that numactl
>>> works for me.
>>>
>>> On Sat, Mar 28, 2020 at 3:58 PM Itaru Kitayama <itaru.k...@gmail.com>
>>> wrote:
>>>
>>>> That is slowing down the build visibly, for the speed I should stick
>>>> with ld at the moment.
>>>>
>>>>
>>>> On Sun, Mar 29, 2020 at 4:42 AM Alexandre Ganea <
>>>> alexand...@ubisoft.com> wrote:
>>>>
>>>>> Would `taskset -c 0-3 ninja check-all -j4` work?
>>>>>
>>>>>
>>>>>
>>>>> *De :* Itaru Kitayama <itaru.k...@gmail.com>
>>>>> *Envoyé :* March 28, 2020 3:37 PM
>>>>> *À :* Alex Brachet-Mialot <alexbrac...@gmail.com>
>>>>> *Cc :* Alexandre Ganea <alexand...@ubisoft.com>; LLVM Dev <
>>>>> llvm...@lists.llvm.org>; Nemanja Ivanovic <nemanj...@gmail.com>
>>>>> *Objet :* Re: [llvm-dev] LLD issue on a massively parallel build
>>>>> There was a tiny bug in how *sched_getaffinity*() was used, see:
>>>>> https://reviews.llvm.org/D75153#1942336
>>>>>
>>>>>
>>>>>
>>>>> *De :* Alex Brachet-Mialot <alexbrac...@gmail.com>
>>>>> *Envoyé :* March 28, 2020 12:11 PM
>>>>> *À :* Itaru Kitayama <itaru.k...@gmail.com>
>>>>> *Cc :* LLVM Dev <llvm...@lists.llvm.org>; Alexandre Ganea <
>>>>> alexand...@ubisoft.com>; Fāng-ruì Sòng <mas...@google.com>
>>>>> *Objet :* Re: [llvm-dev] LLD issue on a massively parallel build
Itaru Kitayama via llvm-dev
Tom Stellard via llvm-dev
> Thanks for the heads up the supercomputer
> Is down for maintenance this week.
> I’ll give it a try when it gets back.
>
This doesn't look to me like it's necessarily an lld issue. Trying
to build all the sub-projects without limiting the number of ninja jobs
is almost guaranteed to run out of memory on a machine like this with such
a low memory:core ratio.
-Tom
> On Thu, Apr 2, 2020 at 1:49 Fangrui Song <mas...@google.com <mailto:mas...@google.com>> wrote:
>
> On 2020-03-29, Nemanja Ivanovic via llvm-dev wrote:
> >Glad you got it working.
> >My suggestion about LLVM_ENABLE_THREADS didn't work because you didn't
> >apply it when building the build linker.
> >
> >When you don't have the ability to rebuild the build compiler, this doesn't
> >apply. In those cases, I end up doing a dirty hack where I use a wrapper
> >script with something like:
> >
> ><path-to-build-compiler> "$@" -Wl,--no-threads
> >
> >To invoke the build compiler. That way you still get full parallelism while
> >compiling and linking, but you prevent LLD from using threads.
>
>
> https://reviews.llvm.org/D76885 (committed yesterday) changed
> --no-threads to --threads={1,2,...} (--no-threads is used rarely (GNU ld
> does not support it) so I'd rather not keep the option for
> compatibility).
>
> You may try -Wl,--threads=1
>
> >On Sat., Mar. 28, 2020, 4:19 p.m. Itaru Kitayama, <itaru.k...@gmail.com <mailto:itaru.k...@gmail.com>>
> >wrote:
> >
> >> Good news, I was able to use up to 37 cores for LLVM build with LLD.
> >> The build speed, did not measure precisely though, is comparable to the
> >> build with GNU ld case.
> >>
> >> Thank you all for your help!
> >>
> >> On Sun, Mar 29, 2020 at 5:04 AM Alex Brachet-Mialot <
> >> alexbrac...@gmail.com <mailto:alexbrac...@gmail.com>> wrote:
> >>
> >>> Yes unfortunately that would limit you to 4 cores.
> >>>
> >>> There’s no easy way to use lld with —no-threads with our build system at
> >>> the moment unfortunately.
> >>>
> >>> I also had just been using ld.gold, but will switch now that numactl
> >>> works for me.
> >>>
> >>> On Sat, Mar 28, 2020 at 3:58 PM Itaru Kitayama <itaru.k...@gmail.com <mailto:itaru.k...@gmail.com>>
> >>> wrote:
> >>>
> >>>> That is slowing down the build visibly, for the speed I should stick
> >>>> with ld at the moment.
> >>>>
> >>>>
> >>>> On Sun, Mar 29, 2020 at 4:42 AM Alexandre Ganea <
> >>>> alexand...@ubisoft.com <mailto:alexand...@ubisoft.com>> wrote:
> >>>>
> >>>>> Would `taskset -c 0-3 ninja check-all -j4` work?
> >>>>>
> >>>>>
> >>>>>
> >>>>> *De :* Itaru Kitayama <itaru.k...@gmail.com <mailto:itaru.k...@gmail.com>>
> >>>>> *Envoyé :* March 28, 2020 3:37 PM
> >>>>> *À :* Alex Brachet-Mialot <alexbrac...@gmail.com <mailto:alexbrac...@gmail.com>>
> >>>>> *Cc :* Alexandre Ganea <alexand...@ubisoft.com <mailto:alexand...@ubisoft.com>>; LLVM Dev <
> >>>>> llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org>>; Nemanja Ivanovic <nemanj...@gmail.com <mailto:nemanj...@gmail.com>>
> >>>>> *De :* Alex Brachet-Mialot <alexbrac...@gmail.com <mailto:alexbrac...@gmail.com>>
> >>>>> *Envoyé :* March 28, 2020 12:11 PM
> >>>>> *À :* Itaru Kitayama <itaru.k...@gmail.com <mailto:itaru.k...@gmail.com>>
> >>>>> *Cc :* LLVM Dev <llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org>>; Alexandre Ganea <
> >>>>> alexand...@ubisoft.com <mailto:alexand...@ubisoft.com>>; Fāng-ruì Sòng <mas...@google.com <mailto:mas...@google.com>>
> >>>>> llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org>
> >>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >>>>>
> >>>>> _______________________________________________
> >>>>> LLVM Developers mailing list
> >>>>> llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org>
> >>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >>>>>
> >>>>>
>
> >_______________________________________________
> >LLVM Developers mailing list
> >llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org>
Itaru Kitayama via llvm-dev
Itaru Kitayama via llvm-dev
Tom Stellard via llvm-dev
> On another login node which is 256 (GB)/48 (nodes) JURECA at JSC, I never had an LLD issue without setting -j when executing ninja
> in the past few weeks.
>
>
> Tom,
> Then what ratio do you think it’s minimal?
>
It really depends on your configuration, but I usually try to have at least 2 GB
of memory per core. However, I usually do Release builds, so Debug builds might
need more. If you aren't using LLVM_PARALLEL_LINK_JOBS it's pretty easy to
run out of memory once ninja starts linking the tools and unittests.
-Tom
> On Thu, Apr 2, 2020 at 6:11 Tom Stellard <tste...@redhat.com <mailto:tste...@redhat.com>> wrote:
>
> On 04/01/2020 01:47 PM, Itaru Kitayama via llvm-dev wrote:
> > Thanks for the heads up the supercomputer
> > Is down for maintenance this week.
> > I’ll give it a try when it gets back.
> >
>
> This doesn't look to me like it's necessarily an lld issue. Trying
> to build all the sub-projects without limiting the number of ninja jobs
> is almost guaranteed to run out of memory on a machine like this with such
> a low memory:core ratio.
>
> -Tom
>
> > On Thu, Apr 2, 2020 at 1:49 Fangrui Song <mas...@google.com <mailto:mas...@google.com> <mailto:mas...@google.com <mailto:mas...@google.com>>> wrote:
> >
> > On 2020-03-29, Nemanja Ivanovic via llvm-dev wrote:
> > >Glad you got it working.
> > >My suggestion about LLVM_ENABLE_THREADS didn't work because you didn't
> > >apply it when building the build linker.
> > >
> > >When you don't have the ability to rebuild the build compiler, this doesn't
> > >apply. In those cases, I end up doing a dirty hack where I use a wrapper
> > >script with something like:
> > >
> > ><path-to-build-compiler> "$@" -Wl,--no-threads
> > >
> > >To invoke the build compiler. That way you still get full parallelism while
> > >compiling and linking, but you prevent LLD from using threads.
> >
> >
> > https://reviews.llvm.org/D76885 (committed yesterday) changed
> > --no-threads to --threads={1,2,...} (--no-threads is used rarely (GNU ld
> > does not support it) so I'd rather not keep the option for
> > compatibility).
> >
> > You may try -Wl,--threads=1
> >
> > >On Sat., Mar. 28, 2020, 4:19 p.m. Itaru Kitayama, <itaru.k...@gmail.com <mailto:itaru.k...@gmail.com> <mailto:itaru.k...@gmail.com <mailto:itaru.k...@gmail.com>>>
> > >wrote:
> > >
> > >> Good news, I was able to use up to 37 cores for LLVM build with LLD.
> > >> The build speed, did not measure precisely though, is comparable to the
> > >> build with GNU ld case.
> > >>
> > >> Thank you all for your help!
> > >>
> > >> On Sun, Mar 29, 2020 at 5:04 AM Alex Brachet-Mialot <
> > >> alexbrac...@gmail.com <mailto:alexbrac...@gmail.com> <mailto:alexbrac...@gmail.com <mailto:alexbrac...@gmail.com>>> wrote:
> > >>
> > >>> Yes unfortunately that would limit you to 4 cores.
> > >>>
> > >>> There’s no easy way to use lld with —no-threads with our build system at
> > >>> the moment unfortunately.
> > >>>
> > >>> I also had just been using ld.gold, but will switch now that numactl
> > >>> works for me.
> > >>>
> > >>> On Sat, Mar 28, 2020 at 3:58 PM Itaru Kitayama <itaru.k...@gmail.com <mailto:itaru.k...@gmail.com> <mailto:itaru.k...@gmail.com <mailto:itaru.k...@gmail.com>>>
> > >>> wrote:
> > >>>
> > >>>> That is slowing down the build visibly, for the speed I should stick
> > >>>> with ld at the moment.
> > >>>>
> > >>>>
> > >>>> On Sun, Mar 29, 2020 at 4:42 AM Alexandre Ganea <
> > >>>> alexand...@ubisoft.com <mailto:alexand...@ubisoft.com> <mailto:alexand...@ubisoft.com <mailto:alexand...@ubisoft.com>>> wrote:
> > >>>>
> > >>>>> Would `taskset -c 0-3 ninja check-all -j4` work?
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> *De :* Itaru Kitayama <itaru.k...@gmail.com <mailto:itaru.k...@gmail.com> <mailto:itaru.k...@gmail.com <mailto:itaru.k...@gmail.com>>>
> > >>>>> *Envoyé :* March 28, 2020 3:37 PM
> > >>>>> *À :* Alex Brachet-Mialot <alexbrac...@gmail.com <mailto:alexbrac...@gmail.com> <mailto:alexbrac...@gmail.com <mailto:alexbrac...@gmail.com>>>
> > >>>>> *Cc :* Alexandre Ganea <alexand...@ubisoft.com <mailto:alexand...@ubisoft.com> <mailto:alexand...@ubisoft.com <mailto:alexand...@ubisoft.com>>>; LLVM Dev <
> > >>>>> llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org> <mailto:llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org>>>; Nemanja Ivanovic <nemanj...@gmail.com <mailto:nemanj...@gmail.com> <mailto:nemanj...@gmail.com <mailto:nemanj...@gmail.com>>>
> > >>>>> *De :* Alex Brachet-Mialot <alexbrac...@gmail.com <mailto:alexbrac...@gmail.com> <mailto:alexbrac...@gmail.com <mailto:alexbrac...@gmail.com>>>
> > >>>>> *Envoyé :* March 28, 2020 12:11 PM
> > >>>>> *À :* Itaru Kitayama <itaru.k...@gmail.com <mailto:itaru.k...@gmail.com> <mailto:itaru.k...@gmail.com <mailto:itaru.k...@gmail.com>>>
> > >>>>> *Cc :* LLVM Dev <llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org> <mailto:llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org>>>; Alexandre Ganea <
> > >>>>> alexand...@ubisoft.com <mailto:alexand...@ubisoft.com> <mailto:alexand...@ubisoft.com <mailto:alexand...@ubisoft.com>>>; Fāng-ruì Sòng <mas...@google.com <mailto:mas...@google.com> <mailto:mas...@google.com <mailto:mas...@google.com>>>
> > >>>>> llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org> <mailto:llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org>>
> > >>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > >>>>>
> > >>>>> _______________________________________________
> > >>>>> LLVM Developers mailing list
> > >>>>> llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org> <mailto:llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org>>
> > >>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > >>>>>
> > >>>>>
> >
> > >_______________________________________________
> > >LLVM Developers mailing list
> > >llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org> <mailto:llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org>>
Robinson, Paul via llvm-dev
> -----Original Message-----
> From: llvm-dev <llvm-dev...@lists.llvm.org> On Behalf Of Tom Stellard
> via llvm-dev
> Sent: Wednesday, April 1, 2020 7:49 PM
> To: Itaru Kitayama <itaru.k...@gmail.com>
> Cc: Nemanja Ivanovic via llvm-dev <llvm...@lists.llvm.org>
> Subject: Re: [llvm-dev] LLD issue on a massively parallel build machine
>
> On 04/01/2020 04:12 PM, Itaru Kitayama wrote:
> > On another login node which is 256 (GB)/48 (nodes) JURECA at JSC, I
> never had an LLD issue without setting -j when executing ninja
> > in the past few weeks.
> >
> > On Thu, Apr 2, 2020 at 7:17 AM Itaru Kitayama <itaru.k...@gmail.com
> <mailto:itaru.k...@gmail.com>> wrote:
> >
> > Tom,
> > Then what ratio do you think it’s minimal?
> >
>
> It really depends on your configuration, but I usually try to have at
> least 2 GB
> of memory per core. However, I usually do Release builds, so Debug builds
> might
> need more. If you aren't using LLVM_PARALLEL_LINK_JOBS it's pretty easy
> to
> run out of memory once ninja starts linking the tools and unittests.
>
> -Tom
to avoid swapping. Compiling is never an issue, it's the linking phase
that uses memory. LLVM_PARALLEL_LINK_JOBS works well for ninja builds,
it has been a real help.
--paulr
Itaru Kitayama via llvm-dev
Itaru Kitayama via llvm-dev
CMake Error at /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/share/cmake-3.16/Modules/CMakeTestCXXCompiler.cmake:53 (message):
The C++ compiler
"/home/usr4/c74014i/opt/clang/current/bin/clang++"
is not able to compile a simple test program.
It fails with the following output:
Change Dir: /tmp/202004040902/CMakeFiles/CMakeTmp
Run Build Command(s):/home/usr4/c74014i/opt/ninja/bin/ninja cmTC_9430a && [1/2] Building CXX object CMakeFiles/cmTC_9430a.dir/testCXXCompiler.cxx.o
clang-11: warning: -Wl,--threads=1: 'linker' input unused [-Wunused-command-line-argument]
[2/2] Linking CXX executable cmTC_9430a
FAILED: cmTC_9430a
: && /home/usr4/c74014i/opt/clang/current/bin/clang++ -Wl,--threads=1 CMakeFiles/cmTC_9430a.dir/testCXXCompiler.cxx.o -o cmTC_9430a && :
/usr/bin/ld: unrecognized option '--threads=1'
/usr/bin/ld: use the --help option for usage information
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
Fāng-ruì Sòng via llvm-dev
Passing -Wl,--threads=1 via -DCMAKE_CXX_FLAGS causes a config error.-- Check for working CXX compiler: /home/usr4/c74014i/opt/clang/current/bin/clang++ -- broken
CMake Error at /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/share/cmake-3.16/Modules/CMakeTestCXXCompiler.cmake:53 (message):
The C++ compiler
"/home/usr4/c74014i/opt/clang/current/bin/clang++"
is not able to compile a simple test program.
It fails with the following output:
Change Dir: /tmp/202004040902/CMakeFiles/CMakeTmp
Run Build Command(s):/home/usr4/c74014i/opt/ninja/bin/ninja cmTC_9430a && [1/2] Building CXX object CMakeFiles/cmTC_9430a.dir/testCXXCompiler.cxx.o
clang-11: warning: -Wl,--threads=1: 'linker' input unused [-Wunused-command-line-argument]
[2/2] Linking CXX executable cmTC_9430a
FAILED: cmTC_9430a
: && /home/usr4/c74014i/opt/clang/current/bin/clang++ -Wl,--threads=1 CMakeFiles/cmTC_9430a.dir/testCXXCompiler.cxx.o -o cmTC_9430a && :
/usr/bin/ld: unrecognized option '--threads=1'
/usr/bin/ld: use the --help option for usage information
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
Itaru Kitayama via llvm-dev
Mehdi AMINI via llvm-dev
Setting LLVM_PARALLEL_LINK_JOBSdid not help a week or two weeks ago’s lld.But recent commits to lld might reflect the variable correctly.
_______________________________________________
Itaru Kitayama via llvm-dev
Mehdi AMINI via llvm-dev
Then I’d suggest this should be renamed.IMO it’s pretty much confusing.it’s so easy to set ninja parallelism with the direct -j option as all we know.