[llvm-dev] debug build busts memory

[Bagel via llvm-dev]

The linking state of a Debug build seems to require a humongous amount of
memory. My poor little linux machine with 16G of ram swaps its brains out.
Waiting for it to finish (if it ever does) is like the old days when you
submitted your deck of cards and waited until the next day to see the results.

To debug a new backend, is there a way to just get the debug info for the
Target/Foo directory?

Is there a way to just build "llc"?

Any ideas to speed things up?

Thanks, brian

_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

[Tim Northover via llvm-dev]

Hi Bagel,

On Tue, 26 Nov 2019 at 18:15, Bagel via llvm-dev
<llvm...@lists.llvm.org> wrote:
> Is there a way to just build "llc"?

"make llc" should do the trick (though ninja tends to be even faster).

> Any ideas to speed things up?

As you can imagine, this is an issue lots of people hit. Knobs you can
twist to make things better:

1. Use lld, for example with the LLVM_ENABLE_LLD CMake option
(assuming it's installed).
2. Lower the LLVM_PARALLEL_LINK_JOBS CMake option to something your
system can support.
3. The LLVM_USE_SPLIT_DWARF essentially leaves the DWARF info in the
.o files and avoids linking it properly until it's used. This speeds
up linking, at the cost of slower gdb/lldb load, and you need a quite
new version of lldb to debug at all.
4. You can skip building unnecessary targets with
LLVM_TARGETS_TO_BUILD, though that's more significant as a
compile-time saver than link-memory.
5. Even more-so for LLVM_OPTIMIZED_TABLEGEN.
6. There's some option to build .so files instead of .a, which saves
linker memory. But it slows down test runs significantly and is also
annoying for debug so I don't think people tend to use it these days.

I tend to run with the first three on Linux.

Cheers.

Tim.

[Tim Northover via llvm-dev]

On Tue, 26 Nov 2019 at 18:27, Tim Northover <t.p.no...@gmail.com> wrote:
> 2. Lower the LLVM_PARALLEL_LINK_JOBS CMake option to something your
> system can support.

Oh, and this one only works with ninja. So another good reason to
switch if you haven't already.

[David Jones via llvm-dev]

Throw hardware at it.

I build LLVM just fine on my 32GB Ryzen 1700X system.

You don't indicate what your CPU is. At the very least, another 16GB should not be that expensive these days.  However, if you are serious on doing LLVM development, consider a modern system with at least 32GB and a high-thread-count processor. e.g. Ryzen 1700X is cheap these days and provides 16 threads. Newer AMD processors will be even better, and Intel has similar offerings.

[Reid Kleckner via llvm-dev]

On Tue, Nov 26, 2019 at 10:31 AM David Jones via llvm-dev <llvm...@lists.llvm.org> wrote:

Throw hardware at it.

While I agree that this is good advice with the current state of affairs, I don't think we should consider the current situation acceptable.

LLVM has a lot of knobs that Tim alluded to, but first time users shouldn't have to search for them to get a working debug build of LLVM out of the box. As a community, we need to raise the bar here, and spend some time making first time build setup easier. I seem to recall that there were discussions at the dev meeting about simplifying our CMake build, and I think this can be part of that effort.

[Keane, Erich via llvm-dev]

I believe it is pretty clear that the stock ld linker is basically unusable for our project without a sizable amount of hardware.  I’ve typically found that using gold or lld fixes this problem entirely.

 

Perhaps we need to start having CMake detect the presence of lld and use that if possible, else check for ‘gold’ and use it if possible, and only THEN fall back on ld.

 

Additionally, it is pretty rare for someone to need to debug TableGen (at least in my experience), and if they DO need it, they would know enough to know how to change the cmake variable.  I think we should have a serious discussion about making LLVM_OPTIMIZED_TABLEGEN=true be default.

[David Tellenbach via llvm-dev]

Hi Brian,

> The linking state of a Debug build seems to require a humongous amount of
> memory. My poor little linux machine with 16G of ram swaps its brains out.
> Waiting for it to finish (if it ever does) is like the old days when you
> submitted your deck of cards and waited until the next day to see the results.

This actually something I've experienced myself when trying to build LLVM in
debug mode on a machine with 16G of memory. It really is annoying.

> Any ideas to speed things up?

Beside the obvious remark that a machine with more memory for frequent debug
builds is desirable, let me give some remarks:

1. Usually compiling itself is not the problem but linking is. Using a single
threaded build `ninja -j1` should make the build work but causes really
annoyingly long build times.

2. Depending on your host target using a different linker can really help. The
BFD linker didn't work well for me but replacing it with gold (ELF only) or lld
really helped. You can use e.g. the gold linker by setting
`LLVM_USE_LINKER=gold` in your CMake configuration.

3. You can specify the number of parallel link jobs explicitly by setting the
CMake option `LLVM_PARALLEL_LINK_JOBS`. Use a higher number of jobs for
compiling and a lower number for linking.

4. This is a general remark and not limited to a debug build: Use ccache! This
drastically improved build times for me. Enable the CMake option
`LLVM_CCACHE_BUILD` and make sure that ccache is installed and the max. cache
size is big enough (check by running `ccache -s`), I personally use 50G.

> Is there a way to just build "llc"?

Something I've never tested myself: Build e.g. llc in debug mode (`ninja llc`
in a cmake configuration where the build type is `Debug`), build another
toolchain in non-debug mode and simply replace the non-debug version of llc with
the debug version.

> To debug a new backend, is there a way to just get the debug info for the
> Target/Foo directory?

You could try to modify the step above by building llc for your desired target
only by setting the CMake option LLVM_TARGETS_TO_BUILD="your desired target"`.

> Thanks, brian

Cheers,
David

[Reid Kleckner via llvm-dev]

On Tue, Nov 26, 2019 at 10:46 AM Keane, Erich via llvm-dev <llvm...@lists.llvm.org> wrote:

I believe it is pretty clear that the stock ld linker is basically unusable for our project without a sizable amount of hardware.  I’ve typically found that using gold or lld fixes this problem entirely.

 

Perhaps we need to start having CMake detect the presence of lld and use that if possible, else check for ‘gold’ and use it if possible, and only THEN fall back on ld.

+1, we should probably emit a warning if we have to fall back to ld.bfd. It's basically non-functional for large C++ apps.

  

Additionally, it is pretty rare for someone to need to debug TableGen (at least in my experience), and if they DO need it, they would know enough to know how to change the cmake variable.  I think we should have a serious discussion about making LLVM_OPTIMIZED_TABLEGEN=true be default.

If you are doing a release build already (my workflow), this represents a lot of additional overhead: second cmake config step, separate ninja subbuild. But if you are doing a debug build, it's clearly better, and there is no easy alternative. What do you think about enabling it for multi-config generators and single-config debug builds, so single-config release builds are not affected? I'd be in favor.

[Keane, Erich via llvm-dev]

• What do you think about enabling it for multi-config generators and single-config debug builds, so single-config release builds are not affected?

I’d be perfectly OK with that.  I only compile debug, so having my normal set of build flags be unnecessary would be wonderful.  In addition to the two below, I only add -G “Ninja” (plus ‘LLVM_ENABLE_PROJECTS=…

 

From: Reid Kleckner <r...@google.com>
Sent: Tuesday, November 26, 2019 11:04 AM
To: Keane, Erich <erich...@intel.com>
Cc: llvm...@lists.llvm.org
Subject: Re: [llvm-dev] debug build busts memory

 

On Tue, Nov 26, 2019 at 10:46 AM Keane, Erich via llvm-dev <llvm...@lists.llvm.org> wrote:

[Alex Brachet-Mialot via llvm-dev]

The suggestion to just upgrade hardware isn't a constructive one in my
opinion, not everyone can do this. As for hardware, I've built llvm
with with a tiny 2 core pentium chip and 8gb of memory, (granted I
needed to increase my swap space) and I left it over night. Once it's
built you generally don't need to build it in full ever again and the
future strain on your machine will always be less.

These questions come up often enough maybe we should add a new section
about building with not enough memory to llvm/docs/CMake.rst. If 16gb
isn't enough then most people will have a hard time building. There
has been a lot of talk recently about making llvm development more
accessible to newcomers, being unable to even build has to be by far
the most egregious problem. How many people had a passing interest in
llvm but when their build failed didn't want to ask the mailing list
what other options they had and gave up?

[Martin Storsjö via llvm-dev]

On Tue, 26 Nov 2019, Reid Kleckner via llvm-dev wrote:

> On Tue, Nov 26, 2019 at 10:46 AM Keane, Erich via llvm-dev
> <llvm...@lists.llvm.org> wrote:
>
> I believe it is pretty clear that the stock ld linker is
> basically unusable for our project without a sizable amount of
> hardware.  I’ve typically found that using gold or lld fixes
> this problem entirely.
>
>  
>
> Perhaps we need to start having CMake detect the presence of lld
> and use that if possible, else check for ‘gold’ and use it if
> possible, and only THEN fall back on ld.
>
>
> +1, we should probably emit a warning if we have to fall back to ld.bfd.
> It's basically non-functional for large C++ apps.

At least when building in debug mode. In release mode, it's obviously
slower than gold/lld, but not that much slower that I'd label it as
non-functional.

// Martin

[David Blaikie via llvm-dev]

I don't really think of LLVM as being an especially large project (is that inaccurate, compared to other open source software) - any ideas what other projects do?

But I'd at least be happy to see a warning about "hey, you're using the bfd linker - it's slow and uses lots of memory, consider installing gold or lld" - like there used to be a warning about building with asserts enabled. I'm not sure there's much else we can do if their system linker is bfd ld, right? I guess we could specifically look for non-default linkers & fallback to bfd only as a last resort (with a warning if possible)?
 

[Tom Stellard via llvm-dev]

On 11/26/2019 10:41 AM, Reid Kleckner via llvm-dev wrote:

Can we make LLVM_USE_SPLIT_DWARF=ON the default? This seems like the
easiest way to solve the memory usage issues, and we are already recommending
this whenever users hit this problem.

-Tom

[David Blaikie via llvm-dev]

(minor cleanup required: gdb's split dwarf support needs -gdb-index, which means you need a linker that can do that (gold or lld) & also I think the LLVM_USE_SPLIT_DWARF is incomplete (because it only enables -gsplit-dwarf but doesn't add the -Wl,-gdb-index flag to the linker action) & so before recommending it as a mainstream thing should probably be tidied up a bit/fleshed out to address thoes issues)