[llvm-dev] Building with LLVM_PARALLEL_XXX_JOBS

[Sedat Dilek via llvm-dev]

Hi,

I switched from "configure and make" to "cmake" build-system and
wanted to speedup my build.

In my build-script I use...

CMAKE_JOBS="1"
##CMAKE_JOBS=$(($(getconf _NPROCESSORS_ONLN)+1))
JOBS_CMAKE_OPTS="-DLLVM_PARALLEL_COMPILE_JOBS=$CMAKE_JOBS
-DLLVM_PARALLEL_LINK_JOBS=$CMAKE_JOBS"

[1] says in "LLVM-specific variables" section...

*** LLVM_PARALLEL_COMPILE_JOBS:STRING

Define the maximum number of concurrent compilation jobs.

*** LLVM_PARALLEL_LINK_JOBS:STRING

Define the maximum number of concurrent link jobs.

...whereas my configure-log says...

$ grep -i job -A1 logs/configure-log_llvm-toolchain-3.8.0rc3.txt
Job pooling is only available with Ninja generators and CMake 3.0 and
later.
--
Job pooling is only available with Ninja generators and CMake 3.0 and
later.

My cmake-version is...

$ cmake --version
cmake version 2.8.12.2

So, I need Ninja *and* CMake >= v3.0 (or is the right CMake version
sufficient) to use LLVM_PARALLEL_XXX_JOBS?
If this is a fact, can you please adjust the information in [1]?

Do I have other options to speedup my build?
( For testing RCs I tend to use slow build - one single (c)make job. )

My build-script and configure-log are attached.

Thanks.

Regards,
- Sedat -


[1] http://llvm.org/docs/CMake.html

[Mehdi Amini via llvm-dev]

Yes. When you run ninja without any argument you have parallelism by default. This options can control the default number of jobs for ninja.
Also, especially when doing LTO, you may want to limit the number of link jobs independently from the number of compile job, which is again a facility that ninja has.

> If this is a fact, can you please adjust the information in [1]?
>
> Do I have other options to speedup my build?

Yes: start to use ninja ;)
If for any reason (?) you are really stuck with "make", then run "make -j ~ncpus" (with ~ncpus "approximately the number of cores in your machine). It will build in parallel.

--
Mehdi

> ( For testing RCs I tend to use slow build - one single (c)make job. )
>
> My build-script and configure-log are attached.
>
> Thanks.
>
> Regards,
> - Sedat -
>
>
> [1] http://llvm.org/docs/CMake.html

> <configure-log_llvm-toolchain-3.8.0rc3.txt><build_llvm-toolchain.sh>_______________________________________________
> LLVM Developers mailing list
> llvm...@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

[Sedat Dilek via llvm-dev]

So, I like NinjaS but I have not used it yet as software.

From my build-script...

[ -d ${BUILD_DIR} ] || mkdir -p ${BUILD_DIR}
cd $BUILD_DIR

# BUILD-VARIANT #1: CONFIGURE AND MAKE (will be DEPRECATED with LLVM v3.9)
##../llvm/configure $CONFIGURE_OPTS 2>&1 | tee $LOGS_DIR/$CONFIGURE_LOG_FILE
##$MAKE $MAKE_OPTS 2>&1 | tee $LOGS_DIR/$BUILD_LOG_FILE
##sudo $MAKE install 2>&1 | tee $LOGS_DIR/$INSTALL_LOG_FILE

# BUILD-VARIANT #2: CMAKE
$CMAKE ../llvm $CMAKE_OPTS 2>&1 | tee $LOGS_DIR/$CONFIGURE_LOG_FILE
$CMAKE --build . 2>&1 | tee $LOGS_DIR/$BUILD_LOG_FILE
##sudo $CMAKE --build . --target install 2>&1 | tee $LOGS_DIR/$INSTALL_LOG_FILE

You mean configuring with cmake and build (and install) with make?

$CMAKE ../llvm $CMAKE_OPTS 2>&1 | tee $LOGS_DIR/$CONFIGURE_LOG_FILE

$MAKE $MAKE_OPTS 2>&1 | tee $LOGS_DIR/$BUILD_LOG_FILE

sudo $MAKE install 2>&1 | tee $LOGS_DIR/$INSTALL_LOG_FILE

Which combination of cmake/ninja versions are you using (latest are
v3.4.3 and v1.6.0)?

- Sedat -

[C Bergström]

On linux (not Windows) I doubt using Ninja vs make will make drastic
differences.. (Others with actual numbers please chime in to correct
me)

/*
I think the difference could be more beneficial if you're doing
incremental builds, but I don't think that is what you're doing..
*/

On Thu, Feb 25, 2016 at 1:51 PM, Sedat Dilek via llvm-dev

[Sedat Dilek via llvm-dev]

On Thu, Feb 25, 2016 at 7:55 AM, C Bergström <cberg...@pathscale.com> wrote:
> On linux (not Windows) I doubt using Ninja vs make will make drastic
> differences.. (Others with actual numbers please chime in to correct
> me)
>
> /*
> I think the difference could be more beneficial if you're doing
> incremental builds, but I don't think that is what you're doing..
> */
>

What do you mean by "incremental builds"?

/me wanted to test RCs of upcoming LLVM v3.8.0 - not doing daily builds.

I am building a "modern" Linux graphics driver stack (libdrm | mesa |
intel-ddx) and a Linux v4.4.y-LTS.

Things will be more modern when I switch from Ubuntu/precise
(12.04-LTS) to Ubuntu/xenial (upcoming 16.04-LTS, beta1 should be
available today according release-schedule).
( I am a bit sick of backporting software. )

- Sedat -

[1] https://wiki.ubuntu.com/XenialXerus/ReleaseSchedule

[C Bergström]

On Thu, Feb 25, 2016 at 2:04 PM, Sedat Dilek <sedat...@gmail.com> wrote:
> On Thu, Feb 25, 2016 at 7:55 AM, C Bergström <cberg...@pathscale.com> wrote:
>> On linux (not Windows) I doubt using Ninja vs make will make drastic
>> differences.. (Others with actual numbers please chime in to correct
>> me)
>>
>> /*
>> I think the difference could be more beneficial if you're doing
>> incremental builds, but I don't think that is what you're doing..
>> */
>>
>
> What do you mean by "incremental builds"?
>
> /me wanted to test RCs of upcoming LLVM v3.8.0 - not doing daily builds.

By incremental I mean..
build the compiler

change some source file
rebuild
...

[Sedat Dilek via llvm-dev]

> Which combination of cmake/ninja versions are you using (latest are
> v3.4.3 and v1.6.0)?
>

With this combination I could reduce build-time down from approx. 3h
down to 01h20m.

$ egrep -i 'jobs|ninja' llvm-build/CMakeCache.txt
//Program used to build from build.ninja files.
CMAKE_MAKE_PROGRAM:FILEPATH=/opt/cmake/bin/ninja
//Define the maximum number of concurrent compilation jobs.
LLVM_PARALLEL_COMPILE_JOBS:STRING=3
//Define the maximum number of concurrent link jobs.
LLVM_PARALLEL_LINK_JOBS:STRING=1
CMAKE_GENERATOR:INTERNAL=Ninja

$ LC_ALL=C ls -alt logs/3.8.0rc3_clang-3-8-0-rc3_cmake-3-4-3_ninja-1-6-0/
total 360
drwxr-xr-x 2 wearefam wearefam 4096 Feb 25 19:58 .
drwxr-xr-x 6 wearefam wearefam 4096 Feb 25 19:58 ..
-rw-r--r-- 1 wearefam wearefam 130196 Feb 25 19:54
install-log_llvm-toolchain-3.8.0rc3.txt
-rw-r--r-- 1 wearefam wearefam 205762 Feb 25 19:51
build-log_llvm-toolchain-3.8.0rc3.txt
-rw-r--r-- 1 wearefam wearefam 14331 Feb 25 18:30
configure-log_llvm-toolchain-3.8.0rc3.txt

$ LC_ALL=C du -s -m llvm* /opt/llvm-toolchain-3.8.0rc3
315 llvm
941 llvm-build
609 /opt/llvm-toolchain-3.8.0rc3

- Sedat -

[1] https://cmake.org/files/v3.5/cmake-3.5.0-rc3-Linux-x86_64.tar.gz

[Chris Bieneman via llvm-dev]

There are a few notes I'd like to add to this thread.

(1) we have a number of places throughout out CMake build where we use features from newer CMakes gated by version checks. Most of these features are performance or usability related. None of them are correctness. Using the latest CMake release will often result in faster builds, so I encourage it.

(2) CMake's "install" target will pretty much always be slower from clean than the old autoconf/make "install" target. This is because in CMake "install" depends on "all", and our CMake builds more stuff in "all" than autoconf did. To help with this or CMake system has lots of convenient "install-${name}" targets that support component-based installation. Not every component has one of these rules, but if one you need is missing let me know. I also recently (r261681) added a new option (LLVM_DISTRIBUTION_COMPONENTS) that allows you to specify a list of components that have custom install targets. It then creates a new "install-distribution" target that wraps just the components you want. For Apple this is almost a 40% speed up over "ninja install".

-Chris

[Mehdi Amini via llvm-dev]

> On Mar 1, 2016, at 9:57 AM, Chris Bieneman <cbie...@apple.com> wrote:
>
> There are a few notes I'd like to add to this thread.
>
> (1) we have a number of places throughout out CMake build where we use features from newer CMakes gated by version checks. Most of these features are performance or usability related. None of them are correctness. Using the latest CMake release will often result in faster builds, so I encourage it.
>
> (2) CMake's "install" target will pretty much always be slower from clean than the old autoconf/make "install" target. This is because in CMake "install" depends on "all", and our CMake builds more stuff in "all" than autoconf did. To help with this or CMake system has lots of convenient "install-${name}" targets that support component-based installation. Not every component has one of these rules, but if one you need is missing let me know. I also recently (r261681) added a new option (LLVM_DISTRIBUTION_COMPONENTS) that allows you to specify a list of components that have custom install targets. It then creates a new "install-distribution" target that wraps just the components you want. For Apple this is almost a 40% speed up over "ninja install".

That sounds great, I want to use it!
It would even be more awesome with an description/example in docs/CMake.rst :)

--
Mehdi

[ChrisBieneman via llvm-dev]

> On Mar 1, 2016, at 10:01 AM, Mehdi Amini <mehdi...@apple.com> wrote:
>
>
>> On Mar 1, 2016, at 9:57 AM, Chris Bieneman <cbie...@apple.com> wrote:
>>
>> There are a few notes I'd like to add to this thread.
>>
>> (1) we have a number of places throughout out CMake build where we use features from newer CMakes gated by version checks. Most of these features are performance or usability related. None of them are correctness. Using the latest CMake release will often result in faster builds, so I encourage it.
>>
>> (2) CMake's "install" target will pretty much always be slower from clean than the old autoconf/make "install" target. This is because in CMake "install" depends on "all", and our CMake builds more stuff in "all" than autoconf did. To help with this or CMake system has lots of convenient "install-${name}" targets that support component-based installation. Not every component has one of these rules, but if one you need is missing let me know. I also recently (r261681) added a new option (LLVM_DISTRIBUTION_COMPONENTS) that allows you to specify a list of components that have custom install targets. It then creates a new "install-distribution" target that wraps just the components you want. For Apple this is almost a 40% speed up over "ninja install".
>
> That sounds great, I want to use it!
> It would even be more awesome with an description/example in docs/CMake.rst :)

Once I get the last of the kinks worked out for our internal adoption I'm going to open source our config files that use it.

I've also made a note to remind myself to document it in docs/CMake.rst. I need to do a pass updating that with a bunch of the cool new things we're doing with CMake. Thanks for the reminder.

-Chris

[Fabio Pagani via llvm-dev]

For faster builds and rebuilds you should definitely read:
https://blogs.s-osg.org/an-introduction-to-accelerating-your-build-with-clang/
https://blogs.s-osg.org/a-conclusion-to-accelerating-your-build-with-clang/

Hope this helps! 

[Chris Bieneman via llvm-dev]

Fabio, the work I was mentioning here is an extension beyond those blog posts.

Some details:

* The “almost 40%” number I referred to is a multi-stage clang build. That means we build a host-capable compiler, then build the actual compiler we want to ship.

* I’m at Apple, so points 1 and 2 are already covered (we only use clang, and ld64 is a fast linker).

* Our system compiler is PGO+LTO’d, but our stage1 isn’t. Stage1 isn’t because the performance improvement of PGO+LTO is less than the time it takes to build, and stage1 is basically a throwaway.

* We are using Ninja and CMake, but this configuration isn’t really significantly faster than autoconf/make, and actually “ninja install” is slower in my tests than the old autoconf “make install”. The slowdown is almost entirely due to Ninja’s “all” target being a lot bigger.

* This performance is for clean builds, not incremental so ccache or shared libraries would not be a valid optimization

* We do use optimized tablegen

* “Build Less” is exactly what the LLVM_DISTRIBUTION_COMPONENTS enables, just in a friendly wrapper target.

-Chris

[Fabio Pagani via llvm-dev]

Hey Chris,

Sedat was asking for a way to "to speedup my build" and those blog posts were really helpful to me.
Anyway LLVM_DISTRIBUTION_COMPONENTS sounds very cool, hope you will push your code soon!

[Sedat Dilek via llvm-dev]

I got some more inspirations on how to speedup my build and integrated
the URLs into my scripts (attached).

For example to use GOLD as linker or to use '-O3' OptLevel maybe in
combination with LTO and PGO (using '-O3 -flto -fprofile-use').

Let's see when the v3.8.0 FINAL is released.

- Sedat -

[Mehdi Amini via llvm-dev]

> On Mar 2, 2016, at 4:22 PM, Sedat Dilek <sedat...@gmail.com> wrote:
>
> I got some more inspirations on how to speedup my build and integrated
> the URLs into my scripts (attached).
>
> For example to use GOLD as linker or to use '-O3' OptLevel maybe in
> combination with LTO and PGO (using '-O3 -flto -fprofile-use').

LTO *will* slow down dramatically the build.

--
Mehdi

> <build_llvm-toolchain_clang-cmake-ninja.sh><install_llvm-toolchain_clang-cmake-ninja.sh>

[fariborz jahanian via llvm-dev]

Building a binary with ‘LTO’, -O3, etc. will slow down the build. But the built binary could run much faster.
I am not sure what the intention is here.

- Fariborz

[Sedat Dilek via llvm-dev]

I had only a quick view on the blog-texts.

It might be that a CLANG generated with LTO/PGO speeds up the build.
Can you confirm this?

Can you confirm binutils-gold speed up the build?

Has LLVM an own linker?
Can be used? Speedup the build?

Yesterday night I loooked through available CMAKE/LLVM variables...

### GOLD
# CMAKE_LINKER:FILEPATH=/usr/bin/ld
# GOLD_EXECUTABLE:FILEPATH=/usr/bin/ld.gold
# LLVM_TOOL_GOLD_BUILD:BOOL=ON
### OPTLEVEL
# CMAKE_ASM_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
# CMAKE_CXX_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
# CMAKE_C_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
### LTO
# LLVM_TOOL_LLVM_LTO_BUILD:BOOL=ON
# LLVM_TOOL_LTO_BUILD:BOOL=ON
### PGO
# LLVM_USE_OPROFILE:BOOL=OFF
#### TABLEGEN
# LLVM_OPTIMIZED_TABLEGEN:BOOL=OFF

So '-O3' is default for a RELEASE build.

Not sure which of the LTO variables are suitable, maybe both.

PGO? Is that the correct variable?

The blog-text mentioned to use optimized-tablegen.
Good? Bad? Ugly?

Thanks in advance for answering my questions.

Best regards,
- Sedat -

[Tilmann Scheller via llvm-dev]

Hi Sedat,

On 03/03/2016 08:09 AM, Sedat Dilek via llvm-dev wrote:
> It might be that a CLANG generated with LTO/PGO speeds up the build.
> Can you confirm this?

Yes, a Clang host compiler built with LTO or PGO is generally faster
than an -O3 build.

Some things to keep in mind when building the Clang host compiler:

GCC:
- GCC 4.9 gives good results with PGO enabled (1.16x speedup over the
-O3 build), not so much with LTO (actually regresses performance over
the -O3 build, same for PGO vs PGO+LTO)
- GCC 5.1/5.2/5.3 can't build Clang with LTO enabled
(https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66027), that's supposed to
be fixed in GCC 5.4

Clang:
- PGO works and gives a good 1.12x speedup over the -O3 build
(produced about 270GB of profiling data when I tried this in December
last year, this should be addressed soon once the in-process profiling
data merging lands)
- LTO provides a 1.03x speedup over the -O3 build
- I have not tried LTO+PGO with full Clang bootstrap profiling data
but I would expect that it helps to increase the performance even further

> Can you confirm binutils-gold speed up the build?

Yes, gold is definitely faster than ld when building Clang/LLVM.

> Has LLVM an own linker?
> Can be used? Speedup the build?

I haven't tried it but lld can definitely link Clang/LLVM on x86-64 Linux.

> The blog-text mentioned to use optimized-tablegen.
> Good? Bad? Ugly?

Good, it helps to speed up debug builds.

Regards,

Tilmann

[Sedat Dilek via llvm-dev]

[ CCed all folks who answered me ]

Hi,

I have built my llvm-toolchain v3.8.0 (FINAL) with binutils-gold v1.11
in a 1st run.

When building with cmake/ninja there are 150 "Linking" lines...

$ grep Linking logs/3.8.0_clang-3-8-0_cmake-3-4-3_ninja-1-6-0_gold-1-11_compile-jobs-2_link-jobs-1/build-log_llvm-toolchain-3.8.0.txt
| wc -l
150

I have the following cmake-options realized...

*** SNIP ***

### CMAKE OPTIONS
# NOTE #1: CMake Version 2.8.8 is the minimum required(Ubuntu/precise
ships v2.8.7 officially)
# NOTE #2: For fast builds use recommended CMake >= v3.2 (used:
v3.4.3) and Ninja (used: v1.6.0)
# NOTE #3: How to check available cmake-options?
# EXAMPLE #3: cd $BUILD_DIR ; cmake ../llvm -LA | egrep $CMAKE_OPTS
#
# CMake binary
CMAKE="cmake"
# CMake compiler options
COMPILERS_CMAKE_OPTS="-DCMAKE_C_COMPILER=$COMPILER_CC
-DCMAKE_CXX_COMPILER=$COMPILER_CXX"
# NOTE-1: cmake/ninja: Use LLVM_PARALLEL_COMPILE_JOBS and
LLVM_PARALLEL_LINK_JOBS options
# NOTE-2: For fast builds use available (online) CPUs +1 or set values
explicitly
# NOTE-3: For fast and safe linking use bintils-gold and LINK_JOBS="1"
COMPILE_JOBS="2"
##COMPILE_JOBS=$(($(getconf _NPROCESSORS_ONLN)+1))
LINK_JOBS="1"
JOBS_CMAKE_OPTS="-DLLVM_PARALLEL_COMPILE_JOBS=$COMPILE_JOBS
-DLLVM_PARALLEL_LINK_JOBS=$LINK_JOBS"
# Cmake linker options (here: Use binutils-gold to speedup build)
LINKER="/usr/bin/ld.gold"
CMAKE_LINKER="$LINKER"
CMAKE_LINKER_OPTS="-DCMAKE_LINKER=$CMAKE_LINKER"
# CMake Generators
CMAKE_GENERATORS="Ninja"
GENERATORS_CMAKE_OPTS="-G $CMAKE_GENERATORS"
# CMake configure settings
PREFIX_CMAKE_OPTS="-DCMAKE_INSTALL_PREFIX=$PREFIX"
OPTIMIZED_CMAKE_OPTS="-DCMAKE_BUILD_TYPE=RELEASE"
ASSERTIONS_CMAKE_OPTS="-DLLVM_ENABLE_ASSERTIONS=ON"
TARGETS_CMAKE_OPTS="-DLLVM_TARGETS_TO_BUILD=X86"
CONFIGURE_CMAKE_OPTS="$PREFIX_CMAKE_OPTS $OPTIMIZED_CMAKE_OPTS
$ASSERTIONS_CMAKE_OPTS $TARGETS_CMAKE_OPTS"
# All CMake options
CMAKE_OPTS="$COMPILERS_CMAKE_OPTS $JOBS_CMAKE_OPTS $CMAKE_LINKER_OPTS
$GENERATORS_CMAKE_OPTS $CONFIGURE_CMAKE_OPTS"

*** SNAP ***

Is LINK_JOBS="2" speeding up the build?
One guy of you told me to use "1" to be on the safe side - that is my default.

Personally, I do not think this is very very efficiently - more than
with binutild-bfd Linker.

Using llvm-link (in a 2nd build) - good|bad|ugly?

[ TODO#S: Before doing a 2nd build (and in a 3rd run using more
optimized binaries) ]

How do I anable LTO via CMAKE?
How do I enable PGO via CMAKE?

Grepping for 'lto' 'pgo' gives no help in [1].
Searching there for '-fprofile' shows...

LLVM_PROFDATA_FILE:PATH Path to a profdata file to pass into clang’s
-fprofile-instr-use flag. This can only be specified if you’re
building with clang.

Unsure what to use!

From my build-script (attached)...

##### BEGIN *** SECTION WILL BE DELETED ***
#
# XXX: TRYOUT #1: Use GOLD as linker
# XXX: TRYOUT #2: Use '-O3' OptLevel
# XXX: TRYOUT #3: Use #2 in combination with LTO and PGO ('-O3 -flto
-fprofile-use')
# XXX: TRYOUT #4: Use optimized tablegen
#
### TRYOUT #1: GOLD <--- DONE

# CMAKE_LINKER:FILEPATH=/usr/bin/ld
# GOLD_EXECUTABLE:FILEPATH=/usr/bin/ld.gold
# LLVM_TOOL_GOLD_BUILD:BOOL=ON
#

### TRYOUT #2: OPTLEVEL '-O3' <--- NOP

# CMAKE_ASM_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
# CMAKE_CXX_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
# CMAKE_C_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
#

### TRYOUT #3: LTO AND PGO <--- UNSURE

# LLVM_TOOL_LLVM_LTO_BUILD:BOOL=ON
# LLVM_TOOL_LTO_BUILD:BOOL=ON

# LLVM_USE_OPROFILE:BOOL=OFF
#
#### TRYOUT #4: TABLEGEN
# LLVM_OPTIMIZED_TABLEGEN:BOOL=OFF
#
##### END *** SECTION WILL BE DELETED ***

Thanks for any help and ideas.

Regards,
- Sedat -

[1] http://llvm.org/releases/3.8.0/docs/CMake.html

[Sedat Dilek via llvm-dev]

Attached a v2 of my build-script with "cmake options" pretty-fied.

- Sedat -

[Sedat Dilek via llvm-dev]

My build-script as v3.

- Sedat -

[Chris Bieneman via llvm-dev]

LLVM_ENALBLE_LTO=On

How do I enable PGO via CMAKE?

I’m actually in the middle of writing some documentation on this right now because I recently made this process a lot easier. Let me give you the high level overview.

The way PGO works is that you build an instrumented compiler, then you run the instrumented compiler against sample source files. While the instrumented compiler runs it will output a bunch of files containing performance counters (.profraw files). After generating all the profraw files you use llvm-profdata to merge the files into a single profdata file that you can feed into the LLVM_PROFDATA_FILE option.

Alas, there is an easier way to generate the profdata file!

If you look in the clang repo at <clang>/cmake/caches/README.txt you’ll see an explanation of how to use the PGO CMake cache file. The basic idea is you run:

$ cmake -G <generator> -C <path_to_clang>/cmake/caches/PGO.cmake <source dir>

$ <build tool> stage2-instrumented-generate-profdata

If you let that run for a few hours or so, it will place a profdata file in your build directory. This takes a really long time because it builds clang twice, and you *must* have compiler-rt in your build tree.

This process uses any source files under the perf-training directory as training data as long as the source files are marked up with LIT-style RUN lines.

After it finishes you can use “find . -name clang.profdata” to find it, but it should be at a path something like:

<build dir>/tools/clang/stage2-instrumented-bins/utils/perf-training/clang.profdata

You can feed that file into the LLVM_PROFDATA_FILE option when you build your optimized compiler.

-Chris

<build_llvm-toolchain_clang-cmake-ninja.sh>_______________________________________________

[Sedat Dilek via llvm-dev]

On Mon, Mar 14, 2016 at 5:30 PM, Chris Bieneman <cbie...@apple.com> wrote:
[ brutal-snip ]
...

> [ TODO#S: Before doing a 2nd build (and in a 3rd run using more
> optimized binaries) ]
>
> How do I anable LTO via CMAKE?
>
>
> LLVM_ENALBLE_LTO=On
>

[ v4 of my build-script attached ]

Hi Chris,

thanks for the response!

That seems to work (see below).

$ cd $BUILD_DIR
$ grep -i lto CMakeCache.txt | grep ^[A-Z]
[ OUTPUT ]
LLVMLTO_LIB_DEPENDS:STATIC=
LLVM_ENALBLE_LTO:UNINITIALIZED=ON
LLVM_TOOL_LLVM_LTO_BUILD:BOOL=ON
LLVM_TOOL_LTO_BUILD:BOOL=ON
LTO_LIB_DEPENDS:STATIC=general;LLVMX86CodeGen;general;LLVMX86AsmPrinter;general;LLVMX86AsmParser;general;LLVMX86Desc;general;LLVMX86Info;general;LLVMX86Disassembler;general;LLVMCore;general;LLVMLTO;general;LLVMMC;general;LLVMMCDisassembler;general;LLVMSupport;general;LLVMTarget;
COMPILER_RT_HAS_FNO_LTO_FLAG:INTERNAL=1
LLVM_TOOL_LLVM_LTO_BUILD-ADVANCED:INTERNAL=1
LLVM_TOOL_LTO_BUILD-ADVANCED:INTERNAL=1
[ /OUTPUT ]

But is no more documented in
<http://llvm.org/releases/3.8.0/docs/CMake.html> or
<http://llvm.org/docs/CMake.html>.
I have seen LLVM_ENALBLE_LTO=On in some older docs.
Bring it back?
AFAICS, "cmake -LA | grep -i $pattern" die not show me this, but I
might be wrong.
( Some weeks ago, I dealt with LTO/PGO. )

Uuuugh, that sounds a bit complicated or at least for me :-).
I am not that familiar with CMake and the llvm-toolchain build-system.

I need something simple like...

PGO_CMAKE_OPTS="..."
CONFIGURE_CMAKE_OPTS="$CONFIGURE_CMAKE_OPTS $PGO_CMAKE_OPTS"

...in my build-script.

Thanks.

- Sedat -


> -Chris
>

[Sedat Dilek via llvm-dev]

OK, I got a typo, nevertheless LLVM_ENABLE_LTO does not work!

[ ... ]
-- Configuring done
-- Generating done
CMake Warning:
Manually-specified variables were not used by the project:

LLVM_ENABLE_LTO


-- Build files have been written to:
/home/wearefam/src/llvm-toolchain/llvm-build

Do you need more input?

Thanks!

- Sedat -

[Sedat Dilek via llvm-dev]

I found in "http://llvmweekly.org/issue/110"

[ llvm.src/cmake/modules/HandleLLVMOptions.cmake ]

option(LLVM_ENABLE_LTO "Enable link-time optimization" OFF)
append_if(LLVM_ENABLE_LTO "-flto"
CMAKE_CXX_FLAGS
CMAKE_C_FLAGS
CMAKE_EXE_LINKER_FLAGS
CMAKE_SHARED_LINKER_FLAGS)

Searching for...

$ grep LLVM_ENABLE_LTO llvm/cmake/modules/HandleLLVMOptions.cmake
[ OUTPUT EMPTY ]

...gives no output.

So, LLVM_ENABLE_LTO is not available for LLVM/Clang v3.8.0?!

Hmm, maybe I can backport (or git cherry-pick) SVN r259766.

- Sedat -

[1] http://llvmweekly.org/issue/110
[2] http://reviews.llvm.org/rL259766

[Sedat Dilek via llvm-dev]

Backport-patch and v6 of my build-script attached.

- Sedat -

[Sedat Dilek via llvm-dev]

With the backported patch I get...

[ build-log ]
...
[164/1813] Linking CXX executable bin/llvm-tblgen
[165/1813] Building CXX object lib/MC/CMakeFiles/LLVMMC.dir/MCAsmInfoELF.cpp.o
[166/1813] Building CXX object lib/MC/CMakeFiles/LLVMMC.dir/MCAsmStreamer.cpp.o
FAILED: : && /opt/llvm/bin/clang++-3.8 -fPIC
-fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter
-Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic
-Wno-long-long -Wcovered-switch-default -Wnon-virtual-dtor
-Wdelete-non-virtual-dtor -std=c++11 -fcolor-diagnostics
-ffunction-sections -fdata-sections -flto -O3 -flto
-Wl,-allow-shlib-undefined -Wl,-O3 -Wl,--gc-sections
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/AsmWriterEmitter.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/AsmWriterInst.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/Attributes.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CallingConvEmitter.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CodeEmitterGen.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CodeGenDAGPatterns.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CodeGenInstruction.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CodeGenMapTable.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CodeGenRegisters.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CodeGenSchedule.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CodeGenTarget.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/DAGISelEmitter.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/DAGISelMatcherEmitter.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/DAGISelMatcherGen.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/DAGISelMatcherOpt.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/DAGISelMatcher.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/DFAPacketizerEmitter.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/DisassemblerEmitter.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/FastISelEmitter.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/FixedLenDecoderEmitter.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/InstrInfoEmitter.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/IntrinsicEmitter.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/OptParserEmitter.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/PseudoLoweringEmitter.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/RegisterInfoEmitter.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/SubtargetEmitter.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/TableGen.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/X86DisassemblerTables.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/X86ModRMFilters.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/X86RecognizableInstr.cpp.o
utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CTagsEmitter.cpp.o -o
bin/llvm-tblgen lib/libLLVMSupport.a lib/libLLVMTableGen.a
lib/libLLVMSupport.a -lrt -ldl -ltinfo -lpthread -lz -lm
-Wl,-rpath,"\$ORIGIN/../lib" && :
/usr/bin/ld: /opt/llvm-toolchain-3.8.0/bin/../lib/LLVMgold.so: error
loading plugin
/usr/bin/ld: /opt/llvm-toolchain-3.8.0/bin/../lib/LLVMgold.so: error
in plugin cleanup (ignored)
clang-3.8: error: linker command failed with exit code 1 (use -v to
see invocation)
ninja: build stopped: subcommand failed.

Not sure what's going on or missing?

Any help appreciated.
Thanks.

- sed@ -

[Sedat Dilek via llvm-dev]

On Fri, Mar 4, 2016 at 11:28 AM, Tilmann Scheller
<til...@osg.samsung.com> wrote:

> Hi Sedat,
>
> On 03/03/2016 08:09 AM, Sedat Dilek via llvm-dev wrote:
>>
>> It might be that a CLANG generated with LTO/PGO speeds up the build.
>> Can you confirm this?
>
> Yes, a Clang host compiler built with LTO or PGO is generally faster than an
> -O3 build.
>
> Some things to keep in mind when building the Clang host compiler:
>
> GCC:
> - GCC 4.9 gives good results with PGO enabled (1.16x speedup over the -O3
> build), not so much with LTO (actually regresses performance over the -O3
> build, same for PGO vs PGO+LTO)
> - GCC 5.1/5.2/5.3 can't build Clang with LTO enabled
> (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66027), that's supposed to be
> fixed in GCC 5.4
>
> Clang:
> - PGO works and gives a good 1.12x speedup over the -O3 build (produced
> about 270GB of profiling data when I tried this in December last year, this
> should be addressed soon once the in-process profiling data merging lands)
> - LTO provides a 1.03x speedup over the -O3 build
> - I have not tried LTO+PGO with full Clang bootstrap profiling data but I
> would expect that it helps to increase the performance even further
>

I jumped over to LLVM v3.8.1 and trying again your speedup-build hints
- using GCC v4.9.2 and GNU/binutils v2.22 on a Ubuntu/precise AMD64
system.

$ uname -r
3.13.0-92-generic

My build-type is "release"...

BUILDTYPE_CMAKE_OPTS="-DCMAKE_BUILD_TYPE=RELEASE"

...for a "debug" build I have not enough disc-space.

Unfortunately, when I use and enable backported LTO-flag...

LTO_CMAKE_OPTS="-DLLVM_ENABLE_LTO=ON"

...my build breaks here.

Is LTO only available/useful for debug builds?

How do you pass "-flto" in your build-script (see my attached build-script).

Can you look at the attached build-log file?

Now, I am building with none of the speedup options.

>> Can you confirm binutils-gold speed up the build?
>
> Yes, gold is definitely faster than ld when building Clang/LLVM.
>

The same with GNU/gold and LTO-flag enabled.

>> Has LLVM an own linker?
>> Can be used? Speedup the build?
>
> I haven't tried it but lld can definitely link Clang/LLVM on x86-64 Linux.
>

Not tried.

>> The blog-text mentioned to use optimized-tablegen.
>> Good? Bad? Ugly?
>
> Good, it helps to speed up debug builds.
>

And for release builds?

- Sedat -