memory usage running emscripten

[goo...@luketitley.com]

Hey,

I'm really impressed with imscripten, it's such a cool project. 

I was wondering if anyone had any tips for reducing the memory usage when building with emcc.

I have a c++ project that was originally written to run natively, but ports nicely to wasm and webgl. It's really cool to see it running in a web browser. But I've noticed that emcc aquires about ~3gb of memory during the final linking stage. I get that it's doing link time optimisation and that all my object files are essentially still code (llvm bytecode), so I can maybe understand why it needs this much memory, but I was wondering if anyone has advice on how I can get this down ? so that I could build my project on a less powerful (and therefore cheaper) build server.

Really appreciate any insites you guys have about this.

Thanks in advance.

Luke

[Alon Zakai]

Using the LLVM wasm backend, emscripten will by default use wasm object files - so it doesn't compile all the bitcode at the link step, and should use a lot less memory. On linux you can get it using the emsdk with latest-upstream (more builds coming soon), or you can build it manually from source.

You may also be able to reduce some memory usage with the EMCC_CORES and BINARYEN_CORES environment variables, as they tell parts of the compiler/optimizer how many cores to use (but then compilation might take longer).

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[goo...@luketitley.com]

Thanks for this. Will investigate.

[goo...@luketitley.com]

Hey so I tried this. I ran with latest-upstream but and ran 'env EMCC_WASM_BACKEND=1=1' before my usual build instructions.

Unfortunetly, I get a build error, that I don't get on 'latest'.

I don't supose you have any thoughts on this ?

The error is:

wasm-ld: /b/s/w/ir/cache/builder/emscripten-releases/llvm-project/llvm/include/llvm/ADT/Optional.h:177: const T &llvm::optional_detail::OptionalStorage<unsigned int, true>::getValue() const & [T = unsigned int]: Assertion `hasVal' failed.

Seems to be failing building a cache of a standard library...

The rest of the error is this...

/emsdk/upstream/bin/../lib/libLLVM-9svn.so(+0x6a4924)[0x7fa3bc6d9924]
/emsdk/upstream/bin/../lib/libLLVM-9svn.so(_ZN4llvm3sys17RunSignalHandlersEv+0xee)[0x7fa3bc6d761e]
/emsdk/upstream/bin/../lib/libLLVM-9svn.so(+0x6a4bd8)[0x7fa3bc6d9bd8]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0)[0x7fa3bf3fc0e0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf)[0x7fa3bb42bfff]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fa3bb42d42a]
/lib/x86_64-linux-gnu/libc.so.6(+0x2be67)[0x7fa3bb424e67]
/lib/x86_64-linux-gnu/libc.so.6(+0x2bf12)[0x7fa3bb424f12]
/emsdk/upstream/bin/wasm-ld[0x66f912]
/emsdk/upstream/bin/wasm-ld[0x6597f5]
/emsdk/upstream/bin/wasm-ld[0x68571d]
/emsdk/upstream/bin/wasm-ld[0x68153d]
/emsdk/upstream/bin/../lib/libLLVM-9svn.so(+0x63b305)[0x7fa3bc670305]
/emsdk/upstream/bin/../lib/libLLVM-9svn.so(+0x63aed7)[0x7fa3bc66fed7]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xba200)[0x7fa3bbd6d200]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7fa3bf3f24a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fa3bb4e1d0f]

shared:ERROR: '/emsdk/upstream/bin/wasm-ld -o /tmp/emscripten_temp_9SLNtL/index.wasm --allow-undefined --import-memory --import-table --lto-O0

...

my object files

...

/root/.emscripten_cache/wasm-obj/libc++_noexcept.a --whole-archive /root/.emscripten_cache/wasm-obj/libc++abi.a --no-whole-archive /root/.emscripten_cache/wasm-obj/libc.a /root/.emscripten_cache/wasm-obj/libcompiler_rt.a /root/.emscripten_cache/wasm-obj/libc-wasm.a /root/.emscripten_cache/wasm-obj/libdlmalloc_debug.a /root/.emscripten_cache/wasm-obj/libgl-webgl2.a /root/.emscripten_cache/wasm-obj/libpthreads_stub.a /root/.emscripten_cache/wasm-obj/libcompiler_rt_wasm.a /root/.emscripten_cache/wasm-obj/libc_rt_wasm.a -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --export __wasm_call_ctors --export __data_end --export main --export __cxa_demangle --export malloc --export free --export setThrew --export __errno_location --export fflush --export __cxa_can_catch --export __cxa_is_pointer_type -z stack-size=5242880 --initial-memory=16777216 --no-entry --max-memory=16777216 --global-base=1024' failed (-6)

On Tuesday, May 7, 2019 at 1:02:28 PM UTC+2, goo...@luketitley.com wrote:

Thanks for this. Will investigate.

[Alon Zakai]

That sounds like a wasm-ld bug. Can you provide the input files so we can file a bug with a testcase?

If you can't provide the files, then a debug build of LLVM may provide a better stack trace, and there is some chance that would be enough to understand the problem.

- Alon

--

You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/2eee571b-3536-40f1-b620-39f5a5bab653%40googlegroups.com.