wasm-ld returns -11, is it out of memory?

[Shi Yan]

I'm compiling clang/llvm into wasm. It worked on my local machine with 32gb memory. but fails on a 16gb mem machine. I suspect the issue is out of memory. But it's not very obvious from the error message:

FAILED: bin/clang.js-10

: && /home/shiy/emsdk/upstream/emscripten/em++  -s WASM=1 -s NO_EXIT_RUNTIME=1 --bind -s ALLOW_MEMORY_GROWTH=1 -s FORCE_FILESYSTEM=1 -s TOTAL_MEMORY=149880832 -s ERROR_ON_UNDEFINED_SYMBOLS=0 --profiling -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -fno-common -Woverloaded-virtual -Wno-nested-anon-types -g  -Wl,--color-diagnostics -Wl,-allow-shlib-undefined tools/clang/tools/driver/CMakeFiles/clang.dir/driver.cpp.o tools/clang/tools/driver/CMakeFiles/clang.dir/cc1_main.cpp.o tools/clang/tools/driver/CMakeFiles/clang.dir/cc1as_main.cpp.o tools/clang/tools/driver/CMakeFiles/clang.dir/cc1gen_reproducer_main.cpp.o  -o bin/clang.js-10  lib/libLLVMWebAssemblyCodeGen.a lib/libLLVMWebAssemblyAsmParser.a lib/libLLVMWebAssemblyDesc.a lib/libLLVMWebAssemblyDisassembler.a lib/libLLVMWebAssemblyInfo.a lib/libLLVMAnalysis.a lib/libLLVMCodeGen.a lib/libLLVMCore.a lib/libLLVMipo.a lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a lib/libLLVMInstrumentation.a lib/libLLVMMC.a lib/libLLVMMCParser.a lib/libLLVMObjCARCOpts.a lib/libLLVMOption.a lib/libLLVMScalarOpts.a lib/libLLVMSupport.a lib/libLLVMTransformUtils.a lib/libLLVMVectorize.a lib/libclangBasic.a lib/libclangCodeGen.a lib/libclangDriver.a lib/libclangFrontend.a lib/libclangFrontendTool.a lib/libclangSerialization.a lib/libLLVMAsmPrinter.a lib/libLLVMDebugInfoDWARF.a lib/libLLVMSelectionDAG.a lib/libLLVMWebAssemblyDesc.a lib/libLLVMWebAssemblyInfo.a lib/libLLVMMCDisassembler.a lib/libclangCodeGen.a lib/libLLVMCoroutines.a lib/libLLVMCoverage.a lib/libLLVMLTO.a lib/libLLVMObjCARCOpts.a lib/libLLVMPasses.a lib/libLLVMCodeGen.a lib/libLLVMipo.a lib/libLLVMBitWriter.a lib/libLLVMIRReader.a lib/libLLVMAsmParser.a lib/libLLVMLinker.a lib/libLLVMInstrumentation.a lib/libLLVMScalarOpts.a lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a lib/libLLVMVectorize.a lib/libLLVMTransformUtils.a lib/libLLVMTarget.a lib/libLLVMAnalysis.a lib/libLLVMObject.a lib/libLLVMTextAPI.a lib/libclangRewriteFrontend.a lib/libclangARCMigrate.a lib/libclangStaticAnalyzerFrontend.a lib/libclangStaticAnalyzerCheckers.a lib/libclangStaticAnalyzerCore.a lib/libclangCrossTU.a lib/libclangIndex.a lib/libclangFrontend.a lib/libclangDriver.a lib/libLLVMOption.a lib/libLLVMProfileData.a lib/libclangParse.a lib/libLLVMMCParser.a lib/libclangSerialization.a lib/libLLVMBitReader.a lib/libclangSema.a lib/libclangAnalysis.a lib/libclangASTMatchers.a lib/libclangEdit.a lib/libclangFormat.a lib/libclangToolingInclusions.a lib/libclangToolingCore.a lib/libclangAST.a lib/libclangRewrite.a lib/libclangLex.a lib/libclangBasic.a lib/libLLVMCore.a lib/libLLVMRemarks.a lib/libLLVMBitstreamReader.a lib/libLLVMMC.a lib/libLLVMBinaryFormat.a lib/libLLVMDebugInfoCodeView.a lib/libLLVMDebugInfoMSF.a lib/libLLVMSupport.a -lrt -lm lib/libLLVMDemangle.a && :

shared:WARNING: Assuming object file output in the absence of `-c`, based on output filename. Please add with `-c` or `-r` to avoid this warning

shared:WARNING: ignoring unsupported linker flag: `--color-diagnostics`

shared:WARNING: ignoring unsupported linker flag: `-allow-shlib-undefined`

 #0 0x00007f71714d6134 PrintStackTraceSignalHandler(void*) (/home/shiy/emsdk/upstream/bin/../lib/libLLVM-10git.so+0x719134)

 #1 0x00007f71714d3e0e llvm::sys::RunSignalHandlers() (/home/shiy/emsdk/upstream/bin/../lib/libLLVM-10git.so+0x716e0e)

 #2 0x00007f71714d63e8 SignalHandler(int) (/home/shiy/emsdk/upstream/bin/../lib/libLLVM-10git.so+0x7193e8)

 #3 0x00007f717452c890 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x12890)

 #4 0x00007f717021bdd2 (/lib/x86_64-linux-gnu/libc.so.6+0x18edd2)

 #5 0x00000000006c1ef9 lld::wasm::InputChunk::writeTo(unsigned char*) const (/home/shiy/emsdk/upstream/bin/wasm-ld+0x6c1ef9)

 #6 0x00000000006f6a0d lld::wasm::CustomSection::writeTo(unsigned char*) (/home/shiy/emsdk/upstream/bin/wasm-ld+0x6f6a0d)

 #7 0x00000000006f205d std::_Function_handler<void (), void llvm::parallel::detail::parallel_for_each<__gnu_cxx::__normal_iterator<lld::wasm::OutputSection**, std::vector<lld::wasm::OutputSection*, std::allocator<lld::wasm::OutputSection*> > >, lld::wasm::(anonymous namespace)::Writer::writeSections()::$_0>(__gnu_cxx::__normal_iterator<lld::wasm::OutputSection**, std::vector<lld::wasm::OutputSection*, std::allocator<lld::wasm::OutputSection*> > >, __gnu_cxx::__normal_iterator<lld::wasm::OutputSection**, std::vector<lld::wasm::OutputSection*, std::allocator<lld::wasm::OutputSection*> > >, lld::wasm::(anonymous namespace)::Writer::writeSections()::$_0)::'lambda'()>::_M_invoke(std::_Any_data const&) (/home/shiy/emsdk/upstream/bin/wasm-ld+0x6f205d)

 #8 0x00007f717146b5e5 std::_Function_handler<void (), llvm::parallel::detail::TaskGroup::spawn(std::function<void ()>)::$_0>::_M_invoke(std::_Any_data const&) (/home/shiy/emsdk/upstream/bin/../lib/libLLVM-10git.so+0x6ae5e5)

 #9 0x00007f717146b1c7 llvm::parallel::detail::(anonymous namespace)::ThreadPoolExecutor::work() (/home/shiy/emsdk/upstream/bin/../lib/libLLVM-10git.so+0x6ae1c7)

#10 0x00007f7170af19e0 (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbd9e0)

#11 0x00007f71745216db start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76db)

#12 0x00007f71701ae88f clone (/lib/x86_64-linux-gnu/libc.so.6+0x12188f)

shared:ERROR: '/home/shiy/emsdk/upstream/bin/wasm-ld -o bin/clang.js-10 --allow-undefined --lto-O0 -L/home/shiy/emsdk/upstream/emscripten/system/local/lib tools/clang/tools/driver/CMakeFiles/clang.dir/driver.cpp.o -L/home/shiy/emsdk/upstream/emscripten/system/lib tools/clang/tools/driver/CMakeFiles/clang.dir/cc1_main.cpp.o -L/home/shiy/.emscripten_cache/wasm-obj tools/clang/tools/driver/CMakeFiles/clang.dir/cc1as_main.cpp.o tools/clang/tools/driver/CMakeFiles/clang.dir/cc1gen_reproducer_main.cpp.o lib/libLLVMWebAssemblyCodeGen.a lib/libLLVMWebAssemblyAsmParser.a lib/libLLVMWebAssemblyDesc.a lib/libLLVMWebAssemblyDisassembler.a lib/libLLVMWebAssemblyInfo.a lib/libLLVMAnalysis.a lib/libLLVMCodeGen.a lib/libLLVMCore.a lib/libLLVMipo.a lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a lib/libLLVMInstrumentation.a lib/libLLVMMC.a lib/libLLVMMCParser.a lib/libLLVMObjCARCOpts.a lib/libLLVMOption.a lib/libLLVMScalarOpts.a lib/libLLVMSupport.a lib/libLLVMTransformUtils.a lib/libLLVMVectorize.a lib/libclangBasic.a lib/libclangCodeGen.a lib/libclangDriver.a lib/libclangFrontend.a lib/libclangFrontendTool.a lib/libclangSerialization.a lib/libLLVMAsmPrinter.a lib/libLLVMDebugInfoDWARF.a lib/libLLVMSelectionDAG.a lib/libLLVMWebAssemblyDesc.a lib/libLLVMWebAssemblyInfo.a lib/libLLVMMCDisassembler.a lib/libclangCodeGen.a lib/libLLVMCoroutines.a lib/libLLVMCoverage.a lib/libLLVMLTO.a lib/libLLVMObjCARCOpts.a lib/libLLVMPasses.a lib/libLLVMCodeGen.a lib/libLLVMipo.a lib/libLLVMBitWriter.a lib/libLLVMIRReader.a lib/libLLVMAsmParser.a lib/libLLVMLinker.a lib/libLLVMInstrumentation.a lib/libLLVMScalarOpts.a lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a lib/libLLVMVectorize.a lib/libLLVMTransformUtils.a lib/libLLVMTarget.a lib/libLLVMAnalysis.a lib/libLLVMObject.a lib/libLLVMTextAPI.a lib/libclangRewriteFrontend.a lib/libclangARCMigrate.a lib/libclangStaticAnalyzerFrontend.a lib/libclangStaticAnalyzerCheckers.a lib/libclangStaticAnalyzerCore.a lib/libclangCrossTU.a lib/libclangIndex.a lib/libclangFrontend.a lib/libclangDriver.a lib/libLLVMOption.a lib/libLLVMProfileData.a lib/libclangParse.a lib/libLLVMMCParser.a lib/libclangSerialization.a lib/libLLVMBitReader.a lib/libclangSema.a lib/libclangAnalysis.a lib/libclangASTMatchers.a lib/libclangEdit.a lib/libclangFormat.a lib/libclangToolingInclusions.a lib/libclangToolingCore.a lib/libclangAST.a lib/libclangRewrite.a lib/libclangLex.a lib/libclangBasic.a lib/libLLVMCore.a lib/libLLVMRemarks.a lib/libLLVMBitstreamReader.a lib/libLLVMMC.a lib/libLLVMBinaryFormat.a lib/libLLVMDebugInfoCodeView.a lib/libLLVMDebugInfoMSF.a lib/libLLVMSupport.a lib/libLLVMDemangle.a --import-memory --import-table -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --export __wasm_call_ctors --export __data_end --export main --export malloc --export free --export setThrew --export __errno_location --export fflush -z stack-size=5242880 --initial-memory=149880832 --no-entry --global-base=1024 --relocatable' failed (-11)

 

Am I right?

Also, I can't apply -g4. I want to debug llvm. But if I add -g4 to enable debug symbols, the linker would fail. I don't know why? And ideas?

Thanks!

[Alon Zakai]

That could be the linker running out of memory, yes. DWARF info can be extremely verbose. We actually recently made emscripten not emit all the DWARF by default, so perhaps this would be fixed by updating? If it's still on the latest 1.39.4, then maybe there's something else to look at there, if you can provide a testcase.

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/86a0daa1-846b-4881-8626-278b75a911be%40googlegroups.com.