[llvm-dev] Disk space and RAM requirements in docs
Petr Penzin via llvm-dev
Hi,
Have anybody recently built LLVM in Debug mode within
space requirements from the Getting Started doc?
https://llvm.org/docs/GettingStarted.html#hardware
> An LLVM-only build will need about 1-3 GB of space. A full build of LLVM and Clang will need around 15-20 GB of disk space.
From my experience this numbers looks drastically low. On FreeBSD my recent builds consumed more than 70 GB and on Linux - more than 60. I am building LLVM, Clang, and LLD with all default targets and default settings. On FreeBSD I am also building Polly. Also, building `clang-check` target adds around to 20 GB to the build directory size.
Apologies for the long snippet, but here is top 20 lines from `du -h | sort -rh` on my FreeBSD directory (full log is attached):
91G build
40G build/bin
25G build/tools
25G build/lib
24G build/tools/clang
15G build/tools/clang/unittests
5.0G build/tools/clang/lib
3.9G build/lib/Target
3.1G build/tools/clang/unittests/Tooling
3.1G build/tools/clang/test
2.8G
build/tools/clang/test/Tooling/Output/clang-check-mac-libcxx-fixed-compilation-db.cpp.tmp/mock-libcxx/bin
2.8G
build/tools/clang/test/Tooling/Output/clang-check-mac-libcxx-fixed-compilation-db.cpp.tmp/mock-libcxx
2.8G
build/tools/clang/test/Tooling/Output/clang-check-mac-libcxx-fixed-compilation-db.cpp.tmp
2.8G build/tools/clang/test/Tooling/Output
2.8G build/tools/clang/test/Tooling
1.9G build/tools/clang/unittests/Frontend
1.6G build/tools/clang/unittests/Driver
1.5G build/tools/clang/unittests/ASTMatchers
1.4G build/tools/clang/lib/StaticAnalyzer
1.3G build/lib/CodeGen
Note that Clang's `unittest` and `test` directory together occupy
more than 18 GB (which is about the size manual assigns for the
entire build), with most of the space in `test` taken up by output
of tooling tests (is that for MacOS, by the way?). Also, looks
like five of the unittest directories for clang components are
taking more than a gigabyte each (with Tooling using more than 3
GB).
Is this expected or am I doing something wrong? If it is expected, should the manual be fixed?
There are no RAM requirements in the doc, from my observation linker process on Linux or FreeBSD needs about 10 GB at the moment (and this has been growing for a while). Should this be also reflected in hardware requirements as well?
And lastly is there a way to reduce the space consumption by Clang unittests and Tooling tests aside of disabling those?
Best,
Petr
Tim Northover via llvm-dev
<llvm...@lists.llvm.org> wrote:
> Is this expected or am I doing something wrong? If it is expected, should the manual be fixed?
It's been like this for a while, and IMO it's a bug in how those tests
were implemented. Disk space is cheap, but shouldn't be wasted that
badly.
Unfortunately the fix isn't simple. I tried modifying the tests to
remove their output when done a while back, but Windows has weird
non-determinism in when locks on just-closed files are released and
wouldn't let the script remove them sometimes; so it had to be
reverted.
I think someone suggested modifying llvm-lit to keep closer track of
temporaries and remove them, but I don't recall how that necessarily
helps things and never got around to implementing it.
Cheers.
Tim.
_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Alex Bradbury via llvm-dev
I think it would be worth strongly recommending the use of LLD or gold
to link. I've helped out a number of students/newcomers getting LLVM
to build for the first time and encountering issues due to binutils ld
OOMing is a common issue.
> And lastly is there a way to reduce the space consumption by Clang unittests and Tooling tests aside of disabling those?
Setting -DBUILD_SHARED_LIBS=True will reduce space requirements by
using shared libraries rather than static libraries. It will impact
the time taken to run LLVM and Clang tests though, due to greater
startup time.
Best,
Alex
Nicolai Hähnle via llvm-dev
>> And lastly is there a way to reduce the space consumption by Clang unittests and Tooling tests aside of disabling those?
>
> Setting -DBUILD_SHARED_LIBS=True will reduce space requirements by
> using shared libraries rather than static libraries. It will impact
> the time taken to run LLVM and Clang tests though, due to greater
> startup time.
Presumably it'll also increase runtime due to ELF interposing requirements.
LLVM_BUILD_LLVM_DYLIB, which builds the large monolithic shared library,
could be better on that front if symbol visibility is set correctly
(which I haven't double-checked).
Cheers,
Nicolai
>
> Best,
>
> Alex
> _______________________________________________
> LLVM Developers mailing list
> llvm...@lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
--
Lerne, wie die Welt wirklich ist,
Aber vergiss niemals, wie sie sein sollte.
David Chisnall via llvm-dev
> Hi,
>
>
> https://llvm.org/docs/GettingStarted.html#hardware
>
> > An LLVM-only build will need about 1-3 GB of space. A full build of
> LLVM and Clang will need around 15-20 GB of disk space.
>
> From my experience this numbers looks drastically low. On FreeBSD my
> recent builds consumed more than 70 GB and on Linux - more than 60. I am
> building LLVM, Clang, and LLD with all default targets and default
> settings. On FreeBSD I am also building Polly. Also, building
> `clang-check` target adds around to 20 GB to the build directory size.
It looks as if running the tests is adding a lot of space to your
builds. I have three LLVM trunk build directories on this machine
(FreeBSD):
Release, building clang, clang-tools-extras, and lld: 1.2GB
Debug, building clang, clang-tools-extras, and lld: 36GB
Debug + ASan, building clang, lldb, and lld: 40GB
1-3GB is still enough to do a release build and not run the tests. I
don't think I've ever done a 'ninja check' in any of these (I have other
build directories for things I'm actively working on), so you can get
down to under 40GB without running the tests for a Debug build.
I'd definitely recommend following Chandler's advice though and not
bothering with a debug-without-asan build. Debug builds are
sufficiently slow that the overhead of Asan on top is not very big -
working on a release build for fast testing and then switching to Asan +
Debug for debugging is a nice way of working.
David
Petr Penzin via llvm-dev
I deleted the build directory and re-ran CMake from scratch, still got
to 70 GB on FreeBSD. My Linux debug build is around 40 GB, which is what
is everyone seem to expect, so I have two questions:
- Would it be reasonable to update the Getting Started document to bump
up the size of statically linked debug build to 30-40 GB?
- I am also very curious about the size disparity between the build on
my machine and David's, any pointers would be greatly appreciated.
On 4/30/19 6:23 AM, David Chisnall wrote:
> On 30/04/2019 04:29, Petr Penzin via llvm-dev wrote:
>> Hi,
>>
>> Have anybody recently built LLVM in Debug mode /within/ space
>> requirements from the Getting Started doc?
>>
>> https://llvm.org/docs/GettingStarted.html#hardware
>>
>> > An LLVM-only build will need about 1-3 GB of space. A full build
>> of LLVM and Clang will need around 15-20 GB of disk space.
>>
>> From my experience this numbers looks drastically low. On FreeBSD my
>> recent builds consumed more than 70 GB and on Linux - more than 60. I
>> am building LLVM, Clang, and LLD with all default targets and default
>> settings. On FreeBSD I am also building Polly. Also, building
>> `clang-check` target adds around to 20 GB to the build directory size.
>
> It looks as if running the tests is adding a lot of space to your
> builds. I have three LLVM trunk build directories on this machine
> (FreeBSD):
>
> Release, building clang, clang-tools-extras, and lld: 1.2GB
> Debug, building clang, clang-tools-extras, and lld: 36GB
> Debug + ASan, building clang, lldb, and lld: 40GB
Can you please share your build settings for debug builds (any extra
CMake variables that you set)? Are you using the system toolchain? I'll
try to compare build output structure to the structure of Linux build
next week, may be there are some clues.
Best,
Petr
David Blaikie via llvm-dev
option produces significantly more debug info per object file (though
if you use DWARF type units, -fdebug-types-section, at least the
executables won't be any larger with/without -fstandalone-debug) so
that the resulting binaries can be successfully debugged with LLDB.
If you don't plan to use LLDB, I'd suggest disabling this feature by
adding -fno-standalone-debug somewhere in your compiler arguments.
I'd also suggest using Split DWARF (there's a CMake variable for that
- I forget the exact name, but searching your CMakeCache.txt for
"SPLIT" should find it quick enough) as that'll reduce the debug info
size impact significantly (with the tradeoff that now the executable
will have a dependency on other files with debug info (more like the
MachO/Apple model of debug info distribution) - so you can't take the
executable & delete your build tree and still debug the binary).
> cfe-dev mailing list
> cfe...@lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev