Multiple threads but one link thread

[Andy]

I recently build llvm (clang and lldb) by ninja-build.

Compiling .cpp to .o is time consuming but linking .o to a. or .so or executable is memory consuming. Especially if  is set debug mode.

One linker uses 10 GB memory, I have 20 GB in my computer, but 3 threads is too more and it uses swap memory file. (swapping destroys my SSD drive)

It will be well, if compiling will be threaded but if one thread link, other threads waits as option,

[Konstantin Tokarev]

09.09.2020, 21:03, "Andy" <borucki...@gmail.com>:

This can be solved by using pools feature: https://ninja-build.org/manual.html#ref_pool
What remains is to make proper use of them in higher-level build system. For cmake it seems to be https://cmake.org/cmake/help/latest/prop_gbl/JOB_POOLS.html

BTW, try using -gsplit-dwarf for debug builds, it should drastically decrease amount of memory needed for linking.


-- 
Regards,
Konstantin

[Nico Weber]

This is usually done by putting link steps in a pool: https://ninja-build.org/manual.html#ref_pool

For LLVM, ninja files are generated by cmake. I don't know how (or if) cmake sets up pools for link steps. From some quick searching, cmake seems to have JOB_POOL_LINK and the LLVM build seems to set that here http://llvm-cs.pcc.me.uk/cmake/modules/HandleLLVMOptions.cmake#45 so you can try passing -DLLVM_PARALLEL_LINK_JOBS=2 to your llvm cmake invocation and you should get at most 2 links at once.

(There's also a (completely unsupported) GN build for llvm (but not lldb, so likely not useful to you), where it'd be reasonably easy to add pool support.)

Nico

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/4638883a-c6c8-445f-9827-3b347634c970n%40googlegroups.com.

[Andy]

Where can I set -gsplit-dwarf options? In Cmakefiles.txt?

[Matthew Woehlke]

On 09/09/2020 16.31, Andy wrote:
> Where can I set -gsplit-dwarf options? In Cmakefiles.txt?

It looks like you could add this to CMAKE_CXX_FLAGS. (You might,
however, want to add them to CMAKE_CXX_FLAGS_DEBUG /
CMAKE_CXX_FLAGS_RELWITHDEBINFO to an already-configured build e.g. using
CCMake, as it seems otherwise debug info gets generated for release
builds also. Note: configure first because you'll want the other,
default flags in those variables.)

--
Matthew

[Matthew Woehlke]

Oh, and for LLVM specifically, n.b. https://reviews.llvm.org/D75328.

--
Matthew