[LLVMdev] compiler-rt CMake build

[Greg Fitzgerald]

Regarding Brad King's CMake patches for LLVM:

http://thread.gmane.org/gmane.comp.compilers.llvm.cvs/173517

If compiler-rt could follow suit, the packaging for the toolchain
becomes a nice, clean package-manager-friendly DAG. Currently, the
compiler-rt CMake build depends on the just-built-clang to run its
test suite. It's more intuitive to me that either:

1) compiler-rt's sanitizer tests would live inside the clang project
2) compiler-rt would export a test suite that clang could optionally import

The former is simpler, but the latter allows the compiler-rt
developers to retain control over those tests, which are typically
updated alongside changes to compiler-rt's sanitizers. Thoughts?

Thanks,
Greg
_______________________________________________
LLVM Developers mailing list
LLV...@cs.uiuc.edu http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

[Alexey Samsonov]

On Tue, Feb 4, 2014 at 10:17 PM, Greg Fitzgerald <gar...@gmail.com> wrote:

Regarding Brad King's CMake patches for LLVM:

  http://thread.gmane.org/gmane.comp.compilers.llvm.cvs/173517

If compiler-rt could follow suit, the packaging for the toolchain
becomes a nice, clean package-manager-friendly DAG.  Currently, the
compiler-rt CMake build depends on the just-built-clang to run its
test suite.  It's more intuitive to me that either:

 1) compiler-rt's sanitizer tests would live inside the clang project
 2) compiler-rt would export a test suite that clang could optionally import

The former is simpler, but the latter allows the compiler-rt
developers to retain control over those tests, which are typically
updated alongside changes to compiler-rt's sanitizers.  Thoughts?

I'm pretty sure the compiler-rt tests should live in the compiler-rt project. You're right:

these tests should generally be updated in sync with the runtimes. I'm fine with creating

a large test suite comprising all individual tool-specific test suites we have now. I'm not

sure how we can "export" this to clang, though, and why the current situation is bad.

 

Thanks,
Greg

--

Alexey Samsonov, MSK

[Greg Fitzgerald]

The trouble is that the sanitizer tests are not really compiler-rt
tests. The original compiler-rt tests are, but not the sanitizer
tests. The sanitizer tests are "compiler-rt tests via clang". The
sanitizer runtime is a C++ library requiring only a C++ compiler to
build it (and to *unit* test it). The ASan lit tests, however,
require clang, because they are testing those C++ functions in the
context of a particular C/C++ compiler's instrumentation. I believe
these tests should not be called "compiler-rt tests". They are clang
tests. In fact, they are just-built-clang tests. They don't, for
example, verify the C++ library is useful to another compiler, another
version of clang, or simply as a C++ library. They are integration
tests (tests clang+asan), as opposed to unit tests (tests only the
asan runtime).

I understand that the integration tests are more relevant to the
sanitizers than they are clang, and so it make sense to me that they
reside in the compiler-rt repo alongside the sanitizers. But rather
than build them in the compiler-rt build, I think the directory should
be exported so that clang can optionally import it. The clang build
can then add those tests to the clang tests suite, and install the
runtime in whatever location is most appropriate for clang.

Why is this important now? The CMake developers are trying to make
the LLVM toolchain easy to build for package managers. Compiler-rt's
dependency on the just-built-clang makes that difficult. Without that
dependency, Compiler-rt could be built and tested against only an LLVM
package. And likewise, a Clang package would depend on Compiler-rt
(and do further testing on compiler-rt with those lit tests). But
with the dependency on the just-built-clang, there needs to be an
additional top-level package, let's call it ClangAndFriends.
ClangAndFriends would depend on Clang and Compiler-rt. Compiler-rt
would depend on the Clang package as well. Once both are built,
ClangAndFriends would copy the Clang install directory and then copy
the Compiler-rt install directory into it. This extra package and
extra copy would be unnecessary if the Compiler-rt build dropped its
dependency on the just-built-clang. After testing the sanitizers, the
Clang package would copy the runtime into its own install directory.

So what next? Wait for Brad King's LLVM patches to be integrated.
Then we tweak the compiler-rt build to optionally build against the
LLVM install directory (instead of in the context of the LLVM source
directory). Next, we use CMake to export the Sanitizer lit tests.
Lastly, Clang optionally imports the test suite and adds the sanitizer
runtime to its install directory. What do you think? Squeaky clean?

Thanks,
Greg

[Alexey Samsonov]

Hi Greg,

On Thu, Feb 6, 2014 at 12:55 AM, Greg Fitzgerald <gar...@gmail.com> wrote:

The trouble is that the sanitizer tests are not really compiler-rt
tests.  The original compiler-rt tests are, but not the sanitizer
tests.  The sanitizer tests are "compiler-rt tests via clang".  The
sanitizer runtime is a C++ library requiring only a C++ compiler to
build it (and to *unit* test it).  The ASan lit tests, however,
require clang, because they are testing those C++ functions in the
context of a particular C/C++ compiler's instrumentation.  I believe
these tests should not be called "compiler-rt tests".  They are clang
tests.

Well, not really. I agree that there is a significant difference between plain compiler-rt and

sanitizers:

1) compiler-rt is more or less a regular C++ library, with the exception that a compiler

sometimes creates a calls into compiler-rt. compiler-rt can be built with any C compiler,

and can be fairly easy unit-tested.

2) sanitizer libraries are special in a way that they more heavily rely on compiler instrumentation and

code transformations. sanitizers can be built with any C++ compiler, but "unit-testing" is far

less important for them, and although we have regular unit tests (that don't expect

instrumentation) for some sanitizers, we are really more concentrated on testing

compiler+run-time together.

But our tests (call them "integration tests" or "toolchain tests") are not Clang-specific - they

should work (and we'd love to make them work) with GCC version of ASan as well. So, in

a perfect world, we would have a "sanitizer-tests" test suite, which we can run with

any compiler we specify at configuration time, be it just-built Clang, or GCC [1].

 

 In fact, they are just-built-clang tests.  They don't, for
example, verify the C++ library is useful to another compiler, another
version of clang, or simply as a C++ library.  They are integration
tests (tests clang+asan), as opposed to unit tests (tests only the
asan runtime).

I understand that the integration tests are more relevant to the
sanitizers than they are clang, and so it make sense to me that they
reside in the compiler-rt repo alongside the sanitizers.  But rather
than build them in the compiler-rt build, I think the directory should
be exported so that clang can optionally import it.  The clang build
can then add those tests to the clang tests suite, and install the
runtime in whatever location is most appropriate for clang.

Why is this important now?  The CMake developers are trying to make
the LLVM toolchain easy to build for package managers.  Compiler-rt's
dependency on the just-built-clang makes that difficult.  Without that
dependency, Compiler-rt could be built and tested against only an LLVM
package.

Why does compiler-rt needs to be tested against LLVM package? All

compiler-rt runtimes can be built and unit-tested with any C/C++ host compiler,

if they don't depend on just-built-Clang. Or do you mean LLVM testing

utilities (llvm-lit, FileCheck) here? If yes, maybe it makes sense to move *them*

to another package as well?

 

 And likewise, a Clang package would depend on Compiler-rt
(and do further testing on compiler-rt with those lit tests).

That's a bit weird - what if compiler-rt is not supported on a platform, where Clang is supported?

 

 But
with the dependency on the just-built-clang, there needs to be an
additional top-level package, let's call it ClangAndFriends.
ClangAndFriends would depend on Clang and Compiler-rt.  Compiler-rt
would depend on the Clang package as well.  Once both are built,
ClangAndFriends would copy the Clang install directory and then copy
the Compiler-rt install directory into it.

Can we instead make a package SanitizerTests (or whatever)? It won't even

need to have any dependencies - instead we should make it configurable -

SanitizerTests should take as a parameter a path to the compiler, and use

it to run the lit-tests and GTest-style unit tests that require instrumentation.

 

 This extra package and
extra copy would be unnecessary if the Compiler-rt build dropped its
dependency on the just-built-clang.  After testing the sanitizers, the
Clang package would copy the runtime into its own install directory.

So what next?  Wait for Brad King's LLVM patches to be integrated.
Then we tweak the compiler-rt build to optionally build against the
LLVM install directory (instead of in the context of the LLVM source
directory).

Please note that it makes a lot of sense to built compiler-rt (and sanitizers) with just-built

Clang. In fact, even though we should support building it with another compilers (gcc, MSVC),

using just-built-Clang should be a default scenario, like it is in configure+make build.

In this way we'll be able to build compiler-rt libs for platforms that just-build

Clang can target, and host compiler probably can't...

 

 Next, we use CMake to export the Sanitizer lit tests.
Lastly, Clang optionally imports the test suite and adds the sanitizer
runtime to its install directory.  What do you think?  Squeaky clean?

Thanks,
Greg

[1] Except that we frequently add new features to sanitizers and modify

compiler<->runtime interface, so the test-suite is kind of tied to the ToT Clang.

But, at the moment we merge Sanitizer runtime into GCC tree at revison X,

we can cut off Sanitizer testsuite at revision X as well, and expect it to work.

--

Alexey Samsonov, MSK

[Brad King]

On 02/06/2014 08:12 AM, Alexey Samsonov wrote:
> Please note that it makes a lot of sense to built compiler-rt (and sanitizers) with just-built
> Clang. In fact, even though we should support building it with another compilers (gcc, MSVC),
> using just-built-Clang should be a default scenario, like it is in configure+make build.

This is possible with CMake using the ExternalProject module:

http://www.cmake.org/cmake/help/v2.8.12/cmake.html#module:ExternalProject

The add_llvm_external_project macro already in LLVM could be taught
to call ExternalProject_Add instead of add_subdirectory. This could
create a custom target with dependencies appropriately configured to
build after Clang. The custom target would run CMake to configure the
project like an outside build and then launch the build.

IIUC there is a desire for Clang to be able to be built externally to
LLVM rather than subsumed into its build process. The top-level
CMakeLists.txt file of Clang already has code to do that, though it
can be much cleaner after my patches to LLVM to provide LLVMConfig.cmake
are integrated. Then it will even be possible to build Clang using
CMake against a LLVM that was built and installed using configure+make.

If one would like to drive compiler-rt as part of testing a Clang built
outside of LLVM then the appropriate place for the ExternalProject_Add
call would be inside the build of Clang.

Either way, from a quick glance at the top-level CMakeLists.txt file
in compiler-rt it appears to want to know a bunch of information about
Clang rather than LLVM. In this case having Clang export enough info
for applications to write find_package(Clang) would make sense. This
is possible to do an can be implemented using techniques similar to
that in my proposed LLVMConfig.cmake patch series.

-Brad

[Jean-Daniel Dupas]

Le 6 févr. 2014 à 16:20, Brad King <brad...@kitware.com> a écrit :

> On 02/06/2014 08:12 AM, Alexey Samsonov wrote:
>> Please note that it makes a lot of sense to built compiler-rt (and sanitizers) with just-built
>> Clang. In fact, even though we should support building it with another compilers (gcc, MSVC),
>> using just-built-Clang should be a default scenario, like it is in configure+make build.
>
> This is possible with CMake using the ExternalProject module:
>
> http://www.cmake.org/cmake/help/v2.8.12/cmake.html#module:ExternalProject
>
> The add_llvm_external_project macro already in LLVM could be taught
> to call ExternalProject_Add instead of add_subdirectory. This could
> create a custom target with dependencies appropriately configured to
> build after Clang. The custom target would run CMake to configure the
> project like an outside build and then launch the build.
>
> IIUC there is a desire for Clang to be able to be built externally to
> LLVM rather than subsumed into its build process. The top-level
> CMakeLists.txt file of Clang already has code to do that, though it
> can be much cleaner after my patches to LLVM to provide LLVMConfig.cmake
> are integrated. Then it will even be possible to build Clang using
> CMake against a LLVM that was built and installed using configure+make.
>
> If one would like to drive compiler-rt as part of testing a Clang built
> outside of LLVM then the appropriate place for the ExternalProject_Add
> call would be inside the build of Clang.

Look like what we need to build compiler-rt. Actually the makefile responsible to launch the compiler-rt build is in clang/runtime/compiler-rt.
Using ExternalProject_Add, we can probably add the CMake part beside that existing file, so everything will be at the same place.

> Either way, from a quick glance at the top-level CMakeLists.txt file
> in compiler-rt it appears to want to know a bunch of information about
> Clang rather than LLVM. In this case having Clang export enough info
> for applications to write find_package(Clang) would make sense. This
> is possible to do an can be implemented using techniques similar to
> that in my proposed LLVMConfig.cmake patch series.
>
> -Brad
>
> _______________________________________________
> LLVM Developers mailing list
> LLV...@cs.uiuc.edu http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

-- Jean-Daniel

[Alexey Samsonov]

On Thu, Feb 6, 2014 at 7:57 PM, Jean-Daniel Dupas <devl...@shadowlab.org> wrote:

Le 6 févr. 2014 à 16:20, Brad King <brad...@kitware.com> a écrit :

> On 02/06/2014 08:12 AM, Alexey Samsonov wrote:
>> Please note that it makes a lot of sense to built compiler-rt (and sanitizers) with just-built
>> Clang. In fact, even though we should support building it with another compilers (gcc, MSVC),
>> using just-built-Clang should be a default scenario, like it is in configure+make build.
>
> This is possible with CMake using the ExternalProject module:
>
> http://www.cmake.org/cmake/help/v2.8.12/cmake.html#module:ExternalProject
>
> The add_llvm_external_project macro already in LLVM could be taught
> to call ExternalProject_Add instead of add_subdirectory.  This could
> create a custom target with dependencies appropriately configured to
> build after Clang.  The custom target would run CMake to configure the
> project like an outside build and then launch the build.
>
> IIUC there is a desire for Clang to be able to be built externally to
> LLVM rather than subsumed into its build process.  The top-level
> CMakeLists.txt file of Clang already has code to do that, though it
> can be much cleaner after my patches to LLVM to provide LLVMConfig.cmake
> are integrated.  Then it will even be possible to build Clang using
> CMake against a LLVM that was built and installed using configure+make.
>
> If one would like to drive compiler-rt as part of testing a Clang built
> outside of LLVM then the appropriate place for the ExternalProject_Add
> call would be inside the build of Clang.

Look like what we need to build compiler-rt. Actually the makefile responsible to launch the compiler-rt build is in clang/runtime/compiler-rt.
Using ExternalProject_Add, we can probably add the CMake part beside that existing file, so everything will be at the same place.

OK, I feel like I need to learn more about ExternalProject_Add magic. I have a few

quick questions for people with knowledge - currently:

1) "compiler-rt"'s CMake needs to know about targets in the Clang build tree ("clang") and

in LLVM build tree ("FileCheck", llvm-* command-line tools, googletest etc.), and uses common

macro from LLVM's CMake modules like "configute_lit_site_cfg", "parse_arguments".

2) top-level targets from compiler-rt's CMake files are visible at the root of LLVM's build tree,

so that I can run "make check-asan" or even "make clang_rt.asan-x86_64" from the root of

the build tree.

Will we easily retain all these capabilities if we turn compiler-rt into an external project?

 

> Either way, from a quick glance at the top-level CMakeLists.txt file
> in compiler-rt it appears to want to know a bunch of information about
> Clang rather than LLVM.  In this case having Clang export enough info
> for applications to write find_package(Clang) would make sense.  This
> is possible to do an can be implemented using techniques similar to
> that in my proposed LLVMConfig.cmake patch series.
>
> -Brad
>

> _______________________________________________
> LLVM Developers mailing list
> LLV...@cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

-- Jean-Daniel





_______________________________________________
LLVM Developers mailing list
LLV...@cs.uiuc.edu         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

--

Alexey Samsonov, MSK

[Brad King]

On 02/06/2014 01:02 PM, Alexey Samsonov wrote:
> OK, I feel like I need to learn more about ExternalProject_Add magic.

It is the CMake equivalent to a configure+make build target that
runs configure+make for another project.

> I have a few quick questions for people with knowledge - currently:
> 1) "compiler-rt"'s CMake needs to know about targets in the Clang build tree ("clang") and
> in LLVM build tree ("FileCheck", llvm-* command-line tools, googletest etc.), and uses common
> macro from LLVM's CMake modules like "configute_lit_site_cfg", "parse_arguments".

This will be possible once LLVM and Clang are taught to provide proper
content in LLVMConfig.cmake and ClangConfig.cmake package configuration
files. The patch series I proposed in the thread Greg linked at the
start of this thread is the first step to do that.

> 2) top-level targets from compiler-rt's CMake files are visible at the root of LLVM's build tree,
> so that I can run "make check-asan" or even "make clang_rt.asan-x86_64" from the root of
> the build tree.

This would not work with ExternalProject_Add because the CMake
running on LLVM or Clang would not see the targets in compiler-rt
since the project will not even be processed until build (make) time.
In return the compiler-rt would be able to build using the just-built
Clang because it will now be available when CMake runs on compiler-rt.
(IIUC this is currently the case for configure+make.) Also compiler-rt
could now be built outside of a LLVM/Clang that was built unaware of
compiler-rt.

It is also possible to make project/compiler-rt build optionally
with add_subdirectory instead of ExternalProject_Add, as it does now,
with the cost of not using the just-built Clang.

[Alexey Samsonov]

On Fri, Feb 7, 2014 at 1:59 AM, Brad King <brad...@kitware.com> wrote:

On 02/06/2014 01:02 PM, Alexey Samsonov wrote:
> OK, I feel like I need to learn more about ExternalProject_Add magic.

It is the CMake equivalent to a configure+make build target that
runs configure+make for another project.

> I have a few quick questions for people with knowledge - currently:
> 1) "compiler-rt"'s CMake needs to know about targets in the Clang build tree ("clang") and
> in LLVM build tree ("FileCheck", llvm-* command-line tools, googletest etc.), and uses common
> macro from LLVM's CMake modules like "configute_lit_site_cfg", "parse_arguments".

This will be possible once LLVM and Clang are taught to provide proper
content in LLVMConfig.cmake and ClangConfig.cmake package configuration
files.  The patch series I proposed in the thread Greg linked at the
start of this thread is the first step to do that.

> 2) top-level targets from compiler-rt's CMake files are visible at the root of LLVM's build tree,
> so that I can run "make check-asan" or even "make clang_rt.asan-x86_64" from the root of
> the build tree.

This would not work with ExternalProject_Add because the CMake
running on LLVM or Clang would not see the targets in compiler-rt
since the project will not even be processed until build (make) time.
In return the compiler-rt would be able to build using the just-built
Clang because it will now be available when CMake runs on compiler-rt.
(IIUC this is currently the case for configure+make.)  Also compiler-rt
could now be built outside of a LLVM/Clang that was built unaware of
compiler-rt.

I see the benefit in using just-built Clang, but it's a pity to lose the ability to

add convenient top-level targets from compiler-rt project. It also means

that we'd need to add "fake" top-level custom targets (that would configure +

build + do something) in compiler-rt build tree if we want to, say,

run compiler-rt test suite as a part of "make check-all" command.

I'll take a look at your patches for ExternalProject that were recently

submitted.

 

It is also possible to make project/compiler-rt build optionally
with add_subdirectory instead of ExternalProject_Add, as it does now,
with the cost of not using the just-built Clang.

-Brad

--

Alexey Samsonov, MSK

[Chandler Carruth]

Just to try and share some perspective of developers on Clang and LLVM...

On Thu, Feb 6, 2014 at 7:20 AM, Brad King <brad...@kitware.com> wrote:

IIUC there is a desire for Clang to be able to be built externally to
LLVM rather than subsumed into its build process.  The top-level
CMakeLists.txt file of Clang already has code to do that, though it
can be much cleaner after my patches to LLVM to provide LLVMConfig.cmake
are integrated.

While there is a strong desire to do this by some developers, it is purely for pragmatic reasons: it makes their builds *significantly* faster because xcode doesn't load the entire project, etc.

I don't believe there is *any* interest in breaking the hard version lock between LLVM and Clang. We rely on that heavily, and it's likely never going away.

I think the default and expected build mode should still be a merged tree where the bits and pieces desired are checked out into an LLVM checkout, and the build system can handle whatever partial build there is provided the dependencies are in place.

 

Then it will even be possible to build Clang using
CMake against a LLVM that was built and installed using configure+make.

I actually doubt that this is a particularly interesting build configuration. Perhaps it is for distributions or packagers, but frankly, I expect it to bitrot immediately. I don't think any devs will realistically use it. Maybe I'm wrong though...

[Tobias Grosser]

At best, we would have some buildbots using this configuration (we could
switch existing onces to this scheme).

I don't think we need it for clang, but many external projects would
like to use cmake to build against LLVM as installed by debian e.g. In
this case, it is great if even an autoconf installation of LLVM provides
the relevant cmake pieces.

Cheers,
Tobias

[Brad King]

On 02/10/2014 06:08 AM, Chandler Carruth wrote:
> While there is a strong desire to do this by some developers, it is purely
> for pragmatic reasons

[snip]

> I think the default and expected build mode should still be a merged tree

Okay, thanks for the clarification.

On 02/10/2014 08:52 AM, Tobias Grosser wrote:
> it is great if even an autoconf installation of LLVM provides
> the relevant cmake pieces.

As of r201053 this is now the case :)

-Brad

[Alexey Samsonov]

Hi Brad,

I have a few questions regarding ExternalProject_Add. For me it doesn't really work as expected.

I add the following code to the tools/clang/runtime/CMakeLists.txt to configure compiler-rt as external project:

  ExternalProject_Add(compiler-rt                                                                         

    #DEPENDS clang clang++ llvm-config                                                              

    PREFIX ${CMAKE_BINARY_DIR}/projects/compiler-rt

    SOURCE_DIR ${COMPILER_RT_SRC_ROOT}                                                                    

    CMAKE_ARGS -DCMAKE_C_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/clang                                     

               -DCMAKE_CXX_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/clang++                                 

               -DCMAKE_BUILD_TYPE=Release                                                                 

               -DLLVM_CONFIG_PATH=${LLVM_RUNTIME_OUTPUT_INTDIR}/llvm-config                               

               -DCOMPILER_RT_OUTPUT_DIR=${LLVM_LIBRARY_OUTPUT_INTDIR}/clang/${CLANG_VERSION}              

               -DCOMPILER_RT_INSTALL_PATH=lib${LLVM_LIBDIR_SUFFIX}/clang/${CLANG_VERSION}                 

               # -DCOMPILER_RT_INCLUDE_TESTS=ON                                                           

    INSTALL_COMMAND ""                                                                                    

    )

  add_dependencies(compiler-rt clang clang++ llvm-config)

1) Looks like "DEPENDS" option is just broken - docs here (http://www.kitware.com/media/html/BuildingExternalProjectsWithCMake2.8.html)

state that you can pass CMake targets there, but if I uncomment that line, I get errors like:

CMake Error at /usr/local/share/cmake-2.8/Modules/ExternalProject.cmake:720 (message):

  External project "clang" has no stamp_dir

Call Stack (most recent call first):

  /usr/local/share/cmake-2.8/Modules/ExternalProject.cmake:932 (ExternalProject_Get_Property)

  /usr/local/share/cmake-2.8/Modules/ExternalProject.cmake:1488 (_ep_get_step_stampfile)

  /usr/local/share/cmake-2.8/Modules/ExternalProject.cmake:1702 (_ep_add_configure_command)

  tools/clang/runtime/CMakeLists.txt:18 (ExternalProject_Add)

as if "clang" was supposed to be another external project. FTR, I use cmake 2.8.10.2.

2) The dependencies don't act as expected: if I run "make compiler-rt", it builds Clang, uses it to configure

compiler-rt and builds compiler-rt. But if I then change the Clang sources, and re-run "make compiler-rt",

Clang is re-built, but compiler-rt is *not* re-configured or re-built, while I definitely want this.

3) The same doc (http://www.kitware.com/media/html/BuildingExternalProjectsWithCMake2.8.html) states that

"One drawback, however, is ExternalProject’s lack of full dependency analysis. Changes in header files of an external project may

not cause an incremental rebuild of the affected sources that depend on those headers."

Looks like even if I modify *sources* under projects/compiler-rt, and re-run "make compiler-rt" from the build

directory, it doesn't re-build the compiler-rt libraries. This makes incremental development simply impossible.

What am I doing wrong?

-- 

Alexey Samsonov, MSK

[Alexey Samsonov]

Hm, this thread: http://cmake.3232098.n2.nabble.com/ExternalProject-Add-Automatic-Incremental-Rebuilds-Add-Sources-to-IDE-td7585544.html

suggests that ExternalProject is not good at all for tracking source file changes, and suggests to use add_subdirectory approach and pass

information around in variables. Looks like we just can't express rules like "use this just-built compiler to build those libraries" in terms of CMake *sigh*

We can try to configure new build trees with CMake manually, though, similar to the way it's done in add_tablegen() macro.

--

Alexey Samsonov, MSK

[Brad King]

On 02/21/2014 09:49 AM, Alexey Samsonov wrote:
> 1) Looks like "DEPENDS" option is just broken

> I use cmake 2.8.10.2.

Dependencies on normal targets were accidentally broken in 2.8.9 and
later fixed in 2.8.11 (along with a test case to keep it working):

http://www.cmake.org/Bug/view.php?id=13849
http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=c0f1af92

> if I then change the Clang sources, and re-run "make compiler-rt",
> Clang is re-built, but compiler-rt is *not* re-configured or re-built

[snip]

> even if I modify *sources* under projects/compiler-rt

> it doesn't re-build the compiler-rt libraries.

ExternalProject was originally designed for two purposes:

* Build third-party dependences on which local development will
not be done

* Manage a "superbuild" project that has no binaries of its own
but has a bunch of ExternalProject_Add calls to drive set up
of a bunch of build trees. Local development then occurs in
each tree manually.

Using ExternalProject for a recursive-make-like structure was
only later proposed as a use case. In order to make the build
step always run you need to make it depend on an extra step that
is always out of date:

ExternalProject_Add_Step(compiler-rt always-build
COMMENT "Always building MyCOnly..."
DEPENDERS build
ALWAYS 1
)

[Brad King]

On 02/21/2014 10:31 AM, Brad King wrote:
> Using ExternalProject for a recursive-make-like structure was
> only later proposed as a use case. In order to make the build
> step always run you need to make it depend on an extra step that
> is always out of date:
>
> ExternalProject_Add_Step(compiler-rt always-build
> COMMENT "Always building MyCOnly..."

s/MyCOnly/compiler-rt/

> DEPENDERS build
> ALWAYS 1
> )

In order to simplify this in the future, I've looked at adding
a BUILD_ALWAYS option to ExternalProject_Add:

ExternalProject: Add option to always run the build step
http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=73e5c6ae

This would allow one to request that the external build step run
on every build of the main project without adding an extra step
manually.

[Alexey Samsonov]

Great, thanks for suggestion! We can even use "DEPENDERS configure" instead of "DEPENDERS build" here.

And we'd better "clobber" the compiler-rt build tree if any of the dependencies change (we need to rebuild

every compiler-rt source from scratch if clang has changed).

I will also try to add support for running tests in compiler-rt build tree. I think this can be achieved by smth. like:

add_custom_target(check-compiler-rt

  COMMAND ${CMAKE_BUILD_TOOL} check-all

  DEPENDS compiler-rt

  WORKING_DIRECTORY ${COMPILER_RT_BUILD_DIR})

-Brad

--

Alexey Samsonov, MSK

[Brad King]

On 02/21/2014 11:20 AM, Alexey Samsonov wrote:
> We can even use "DEPENDERS configure" instead of "DEPENDERS build" here.

Just "build" should be sufficient because the generated build system
in compiler-rt knows how to re-run CMake if any of the inputs have
changed. This is just as you do not need to re-run "cmake ." by hand
after editing CMakeLists.txt and can just type "make".

[Alexey Samsonov]

On Fri, Feb 21, 2014 at 9:03 PM, Brad King <brad...@kitware.com> wrote:

On 02/21/2014 11:20 AM, Alexey Samsonov wrote:
> We can even use "DEPENDERS configure" instead of "DEPENDERS build" here.

Just "build" should be sufficient because the generated build system
in compiler-rt knows how to re-run CMake if any of the inputs have
changed.  This is just as you do not need to re-run "cmake ." by hand
after editing CMakeLists.txt and can just type "make".

I am not convinced. Suppose LLVMConfig.cmake (which we read in standalone compiler-rt build system) has changed.

Then we should re-run cmake for standalone compiler-rt even though none of its cmake or source files was changed.

Could you take a look if http://llvm-reviews.chandlerc.com/D2868 looks reasonable?

 

-Brad

--

Alexey Samsonov, MSK

[Brad King]

On 02/24/2014 08:53 AM, Alexey Samsonov wrote:

> On Fri, Feb 21, 2014 at 9:03 PM, Brad King wrote:
> Just "build" should be sufficient because the generated build system
> in compiler-rt knows how to re-run CMake if any of the inputs have
> changed.
>

> Suppose LLVMConfig.cmake has changed.

Every file CMake processes as part of configuring the project ends up in
the build-time dependencies to re-run CMake.

> Could you take a look if http://llvm-reviews.chandlerc.com/D2868 looks reasonable?

It looks pretty good to me. I added some inline comments.

[Alexey Samsonov]

On Mon, Feb 24, 2014 at 6:05 PM, Brad King <brad...@kitware.com> wrote:

On 02/24/2014 08:53 AM, Alexey Samsonov wrote:

> On Fri, Feb 21, 2014 at 9:03 PM, Brad King wrote:
>     Just "build" should be sufficient because the generated build system
>     in compiler-rt knows how to re-run CMake if any of the inputs have
>     changed.
>

> Suppose LLVMConfig.cmake has changed.

Every file CMake processes as part of configuring the project ends up in
the build-time dependencies to re-run CMake.

Alright, I've switched this to

  ExternalProject_Add_Step(compiler-rt force-rebuild

    DEPENDERS build

    ALWAYS 1

    )

However, looks like this works with Unix Makefiles generators, but doesn't work for Ninja.

When I run "ninja compiler-rt" twice in the build tree, I see the following message on the second run:

  ninja: no work to do.

I've verified that compiler-rt is not rebuilt even if I change its sources. Is it a CMake bug?

 

> Could you take a look if http://llvm-reviews.chandlerc.com/D2868 looks reasonable?

It looks pretty good to me.  I added some inline comments.

-Brad

--

Alexey Samsonov, MSK

[Brad King]

On 02/24/2014 09:44 AM, Alexey Samsonov wrote:
> ExternalProject_Add_Step(compiler-rt force-rebuild
> DEPENDERS build
[snip]

> doesn't work for Ninja.

What version of CMake did you use for this test? Also 2.8.10?

> Is it a CMake bug?

Yes, though there have been some changes related to this upstream
since 2.8.10, IIRC.

I think you can just go with your original "DEPENDERS configure"
approach for now.

[Alexey Samsonov]

On Mon, Feb 24, 2014 at 6:57 PM, Brad King <brad...@kitware.com> wrote:

On 02/24/2014 09:44 AM, Alexey Samsonov wrote:
>   ExternalProject_Add_Step(compiler-rt force-rebuild
>     DEPENDERS build
[snip]

> doesn't work for Ninja.

What version of CMake did you use for this test?  Also 2.8.10?

Yes, 2.8.10.2 to be exact.

 

> Is it a CMake bug?

Yes, though there have been some changes related to this upstream
since 2.8.10, IIRC.

I think you can just go with your original "DEPENDERS configure"
approach for now.

"DEPENDERS configure" doesn't help this Ninja issue.

 

-Brad

--

Alexey Samsonov, MSK

[Brad King]

On 02/24/2014 10:02 AM, Alexey Samsonov wrote:
> Yes, 2.8.10.2 to be exact.

Well, the upstream change I made to simplify this feature in
the future:

ExternalProject: Add option to always run the build step
http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=73e5c6ae

includes a test that touches a file in the external project
source. It passes our test suite with the Ninja generator.

Please see if you can reproduce this with CMake 'master' from
Git:

git clone git://cmake.org/cmake.git CMake &&
mkdir CMake-build &&
cd CMake-build &&
../CMake/bootstrap --parallel=8 &&
make -j 8 &&
bin/cmake --version

Thanks,

[Alexey Samsonov]

On Mon, Feb 24, 2014 at 8:26 PM, Brad King <brad...@kitware.com> wrote:

On 02/24/2014 10:02 AM, Alexey Samsonov wrote:
> Yes, 2.8.10.2 to be exact.

Well, the upstream change I made to simplify this feature in
the future:

 ExternalProject: Add option to always run the build step
 http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=73e5c6ae

includes a test that touches a file in the external project
source.  It passes our test suite with the Ninja generator.

Please see if you can reproduce this with CMake 'master' from
Git:

Yep, I can reproduce it with latest CMake and ninja:

$ cmake --version

cmake version 3.0.20140225-gcd8c

$ ninja --version

1.4.0.git

I configure LLVM with -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON

(with applied patches http://llvm-reviews.chandlerc.com/D2868 and http://llvm-reviews.chandlerc.com/D2874)

Then I run

$ ninja compiler-rt

twice. The first run builds the libraries, the second shows only: "ninja: no work to do." message.

 

 git clone git://cmake.org/cmake.git CMake &&
 mkdir CMake-build &&
 cd CMake-build &&
 ../CMake/bootstrap --parallel=8 &&
 make -j 8 &&
 bin/cmake --version

Thanks,
-Brad

--

Alexey Samsonov, MSK

[Brad King]

On 02/25/2014 03:28 AM, Alexey Samsonov wrote:
> Then I run
> $ ninja compiler-rt
> twice. The first run builds the libraries, the second shows only:
> "ninja: no work to do." message.

I'm able to reproduce that, thanks. This is a subtle interaction
between always-out-of-date rules and order-only dependencies.

The problem is that the build.ninja file has the rule:

build projects/compiler-rt/src/compiler-rt-stamp/compiler-rt-force-reconfigure: phony || bin/clang bin/llvm-config

to express the always-out-of-date force-reconfigure rule. The Ninja
documentation says:

"If a phony build statement is written without any dependencies,
the target will be considered out of date if it does not exist."

However, the rule has dependencies because the Ninja generator implements
add_dependencies by adding order-only dependencies to every individual
rule in the dependent target on all dependencies.

It looks like the Ninja generator needs to write a rule is always out
of date but still has order-only dependencies. I think this will work:

build projects/compiler-rt/src/compiler-rt-stamp/compiler-rt-force-reconfigure: phony | always || bin/clang bin/llvm-config
build always: phony

I've recorded this issue in the CMake issue tracker:

http://www.cmake.org/Bug/view.php?id=14771

[Alexey Samsonov]

Hi Brad,

Thanks for investigating this. Do you think it makes sense to land my ExternalProject_Add patch

so that others can experiment with it? I can add quit with a fatal_error/warning if the build tree rules

are generated with Ninja. However, there is a problem with Unix Makefiles as well: parallelism doesn't

work when I run "make check-compiler-rt -j8" in the original build tree, presumably because we call

"cd /path/to/compiler-rt/build/tree && make check-all" there.

On Tue, Feb 25, 2014 at 7:02 PM, Brad King <brad...@kitware.com> wrote:

On 02/25/2014 03:28 AM, Alexey Samsonov wrote:
> Then I run
> $ ninja compiler-rt
> twice. The first run builds the libraries, the second shows only:
> "ninja: no work to do." message.

I'm able to reproduce that, thanks.  This is a subtle interaction
between always-out-of-date rules and order-only dependencies.

The problem is that the build.ninja file has the rule:

 build projects/compiler-rt/src/compiler-rt-stamp/compiler-rt-force-reconfigure: phony || bin/clang bin/llvm-config

to express the always-out-of-date force-reconfigure rule.  The Ninja
documentation says:

 "If a phony build statement is written without any dependencies,
  the target will be considered out of date if it does not exist."

However, the rule has dependencies because the Ninja generator implements
add_dependencies by adding order-only dependencies to every individual
rule in the dependent target on all dependencies.

It looks like the Ninja generator needs to write a rule is always out
of date but still has order-only dependencies.  I think this will work:

 build projects/compiler-rt/src/compiler-rt-stamp/compiler-rt-force-reconfigure: phony | always || bin/clang bin/llvm-config
 build always: phony

I've recorded this issue in the CMake issue tracker:

 http://www.cmake.org/Bug/view.php?id=14771

-Brad

--

Alexey Samsonov, MSK

[Brad King]

On 02/26/2014 12:43 PM, Alexey Samsonov wrote:
> Do you think it makes sense to land my ExternalProject_Add patch
> so that others can experiment with it? I can add quit with a
> fatal_error/warning if the build tree rules are generated with Ninja.

Since it is conditional on LLVM_BUILD_EXTERNAL_COMPILER_RT, yes.

> parallelism doesn't work when I run "make check-compiler-rt -j8"
> in the original build tree, presumably because we call
> "cd /path/to/compiler-rt/build/tree && make check-all" there.

Right. The ExternalProject module has a special case for the
Makefile generators to make with $(MAKE) instead of "make":

http://cmake.org/gitweb?p=cmake.git;a=blob;f=Modules/ExternalProject.cmake;hb=v2.8.12.2#l846

so that flags like -j propagate automatically. You could do
that too:

if(CMAKE_GENERATOR MATCHES "Make")
set(check_compiler_rt "$(MAKE)" "check-all")
else()
set(check_compiler_rt ${CMAKE_COMMAND} --build .
--target check-all --config $<CONFIGURATION>)
endif()

ExternalProject_Get_Property(compiler-rt BINARY_DIR)
add_custom_target(check-compiler-rt
COMMAND ${check_compiler_rt}
DEPENDS compiler-rt
WORKING_DIRECTORY ${BINARY_DIR}
VERBATIM
)

[Alexey Samsonov]

On Wed, Feb 26, 2014 at 9:58 PM, Brad King <brad...@kitware.com> wrote:

On 02/26/2014 12:43 PM, Alexey Samsonov wrote:
> Do you think it makes sense to land my ExternalProject_Add patch
> so that others can experiment with it? I can add quit with a
> fatal_error/warning if the build tree rules are generated with Ninja.

Since it is conditional on LLVM_BUILD_EXTERNAL_COMPILER_RT, yes.

Submitted as r202367.

 

> parallelism doesn't work when I run "make check-compiler-rt -j8"
> in the original build tree, presumably because we call
> "cd /path/to/compiler-rt/build/tree && make check-all" there.

Right.  The ExternalProject module has a special case for the
Makefile generators to make with $(MAKE) instead of "make":

 http://cmake.org/gitweb?p=cmake.git;a=blob;f=Modules/ExternalProject.cmake;hb=v2.8.12.2#l846

so that flags like -j propagate automatically.  You could do
that too:

  if(CMAKE_GENERATOR MATCHES "Make")
    set(check_compiler_rt "$(MAKE)" "check-all")
  else()
    set(check_compiler_rt ${CMAKE_COMMAND} --build .
      --target check-all --config $<CONFIGURATION>)
  endif()

  ExternalProject_Get_Property(compiler-rt BINARY_DIR)
  add_custom_target(check-compiler-rt
    COMMAND ${check_compiler_rt}
    DEPENDS compiler-rt
    WORKING_DIRECTORY ${BINARY_DIR}
    VERBATIM
    )

This worked, thanks! Currently I also print fatal_error message if I detect Ninja as a CMAKE_GENERATOR.

--

Alexey Samsonov, MSK

[Greg Fitzgerald]

> ExternalProject_Add(compiler-rt ...)

So that was quite the experiment. Looking at
clang/runtime/CMakeLists.txt, I'm not seeing a lot of bang for buck
here, and it looks like this file is prone to bit rot. I think we
should consider punting on this one and looking for a more incremental
strategy. For instance, how about starting by moving the call
'add_llvm_external_project(compiler-rt)' from
llvm/projects/CMakesLists.txt to clang/runtime? We can tweak this
macro (or a clang variant) to optionally do something like:

include(AddCompilerRt)

That way if CMAKE_PREFIX_PATH includes a path to the compiler-rt
install directory, it can import all the same build targets that would
be created by add_subdirectory(compiler-rt).

If I were to use CMake's ExternalProject feature, it would be from a
top-level superproject. I would use it to manage dependencies and to
populate CMAKE_PREFIX_PATH for each dependency. CMake, Make or a
package manager, it wouldn't matter which you used for this part (I've
used Make for this). Ultimately, you want to end up with a top-level
build that performs the following:

mkdir -p llvm/out && cd llvm/out
cmake .. -DCMAKE_INSTALL_PREFIX=ship
ninja check-all install
cd -

mkdir -p compiler-rt/out && cd compiler-rt/out
cmake .. -DCMAKE_PREFIX_PATH=`pwd`/../../llvm/out/ship
-DCMAKE_INSTALL_PREFIX=ship
ninja check-all install
cd -

mkdir -p clang/out && cd clang/out
cmake .. -DCMAKE_PREFIX_PATH=`pwd`/../../llvm/out/ship:`pwd`/../../compiler-rt/out/ship
-DCMAKE_INSTALL_PREFIX=ship
ninja check-all install
cd -

In this scenario, the compiler-rt build creates a cmake module that
clang includes. It uses that cmake module to find the compiler-rt
install targets and copy them into "lib/clang/<version>". The clang
build then calls 'add_subdirectory' on the compiler-rt 'test'
directory, pointing it to the just-built-clang. So the dependency
tree for clang is:

sanitizer-tests -> clang -> compiler-rt -> c-compiler

Thoughts?

Thanks,
Greg

[Alexey Samsonov]

On Thu, Mar 20, 2014 at 10:12 PM, Greg Fitzgerald <gar...@gmail.com> wrote:

> ExternalProject_Add(compiler-rt ...)

So that was quite the experiment.  Looking at
clang/runtime/CMakeLists.txt, I'm not seeing a lot of bang for buck
here, and it looks like this file is prone to bit rot.

Could you please elaborate on this? In fact, I don't plan to give up on this experiment.

On the contrary, I wanted to move LLVM_BUILD_EXTERNAL_COMPILER_RT to the

bots, migrate the workflow of our team to use this mode, and eventually make it the default.

The problem is that we want top-level targets from compiler-rt build tree be visible in the llvm/clang

top-level build tree. I thought this could be achieved by ugly propagation of top-level compiler-rt

targets with add_custom_target() command. (like I did this for check-compiler-rt command).

 

 I think we
should consider punting on this one and looking for a more incremental
strategy.  For instance, how about starting by moving the call
'add_llvm_external_project(compiler-rt)' from
llvm/projects/CMakesLists.txt to clang/runtime?  We can tweak this
macro (or a clang variant) to optionally do something like:

    include(AddCompilerRt)

That way if CMAKE_PREFIX_PATH includes a path to the compiler-rt
install directory, it can import all the same build targets that would
be created by add_subdirectory(compiler-rt).

Probably I don't understand how CMAKE_PREFIX_PATH works. How can top-level

targets from one build tree be visible in another build tree?

... But we want to build compiler-rt libraries with just-built Clang as well. This is what

Makefile-based build does, and what we can benefit from.

--

Alexey Samsonov, MSK

[Greg Fitzgerald]

Hi Alexey,

CMAKE_PREFIX_PATH is a convenient mechanism for exposing prebuilt
install directories to a CMake build. It contains a colon-separated
list of paths. Each path points to a directory that contains
directory names that are meaningful to CMake, including 'bin', 'lib'
and 'share'.

If, for example, you want the path to 'llvm-config', instead of
creating a new CMake variable, you instead write
"find_program(LLVM_CONFIG_PATH llvm-config)". CMake will search for
an executable named 'llvm-config' in each 'bin' directory of each path
in CMAKE_PREFIX_PATH.

In a simplest case, where there is only one dependency, using
CMAKE_PREFIX_PATH means we do not have to standardize on a variable
name. For llvm-config, should it be LLVM_CONFIG_PATH or
COMPILER_RT_LLVM_CONFIG_PATH? In a component build, the former is
sufficient, but in the monolithic build, it makes more sense to
namespace every variable. We can avoid painting that bike shed using
CMAKE_PREFIX_PATH.

When one component has multiple dependencies from another, using
CMAKE_PREFIX_PATH really starts saving keystrokes. You can use it to
find headers (sanitizer/asan_interface.h), binaries (FileCheck),
libraries, or CMake modules.

A good alternative to CMAKE_PREFIX_PATH is to create one variable per
install directory. So compiler-rt might accept LLVM_INSTALL_DIR, and
clang accept COMPILER_RT_INSTALL_DIR and LLVM_INSTALL_DIR. There are
two advantages to this route:

1) No need for CMake features such as 'find_program'. Instead, you
can be explicit:
${LLVM_INSTALL_DIR}/bin/llvm-config
2) No need to organize the install directory in any particular way.
If a small component wants to combine its 'bin' and 'lib' directories
into one top-level directory, that's no problem.

The disadvantage of this system is the need to standardize across
components on a naming convention. COMPILER_RT_DIR, COMPILER-RT_DIR,
COMPILER_RT_INSTALL_DIR, COMPILER-RT_SHIP_DIR? Ideally you'd only
have to write "deps = llvm compiler-rt" and a package manager could
automatically translate that to names that match the CMake options
(LLVM_DIR, COMPILER_RT_DIR). But like before, using CMAKE_PREFIX_PATH
avoids the issue.


Regarding the use of ExternalProject within Clang, I think it could
create more problems than it solves. For one, it can't be used from
within the monolithic build (unless you have a duplicate llvm build,
or point CompilerRT to a subset of the currently-building LLVM build).
Secondly, it's not a general enough solution for other LLVM
components. Say, for example, you wanted to use ExternalProject to
build LLVM from both Clang and CompilerRT. If the parameters passed
to LLVM were identical, CMake could detect that and not build a
duplicate copy. If they parameters are different in any way, you have
a problem. Do you duplicate the build or flag an error? We can avoid
this issue by allowing a super-project or package manager build LLVM
and pass its path to both Clang and CompilerRT.


Regarding the use of just-built-clang for both building CompilerRT and
running its test suite, please consider relaxing that to a
stage0-clang for building and a just-built-clang for testing. Using
the just-built-clang to build compiler-rt is rough on package
managers. You're basically stating that the compiler-rt build depends
on some unreleased feature of clang to *build* compiler-rt. To a
package manager, that means it should first build clang, then use that
clang to build compiler-rt, and then build a version of clang that
includes compiler-rt. It implies that every time there is a new
commit to clang, compiler-rt should be rebuilt and retested. If the
compiler-rt build depended only a just-released-clang, then a commit
to clang would imply only that the sanitizer tests should be rebuilt
and tested.

-Greg

[Alexey Samsonov]

Hi Greg,

Thanks for the great detailed explanations. I should check if we can simplify

clang - compiler-rt build systems relationship with this.

Regarding the use of ExternalProject within Clang, I think it could
create more problems than it solves.  For one, it can't be used from
within the monolithic build (unless you have a duplicate llvm build,
or point CompilerRT to a subset of the currently-building LLVM build).

Could you elaborate on this? That's exactly what I'm trying to do now -

when I call "ExternalProject_Add(compiler-rt)" from Clang build tree, I

use just-built llvm-config to configure it, and set up the paths so that

"make compiler-rt" in compiler-rt build tree will build the libraries in the

proper locations, where Clang driver expects to find them. Maybe, I don't

understand what you mean when you say "monolithic build"

 

 Secondly, it's not a general enough solution for other LLVM
components.  Say, for example, you wanted to use ExternalProject to
build LLVM from both Clang and CompilerRT.

Why would one want to build LLVM from Clang and CompilerRT? My approach

is targeted at solving a very specific problem.

 

 If the parameters passed
to LLVM were identical, CMake could detect that and not build a
duplicate copy.  If they parameters are different in any way, you have
a problem.  Do you duplicate the build or flag an error?  We can avoid
this issue by allowing a super-project or package manager build LLVM
and pass its path to both Clang and CompilerRT.


Regarding the use of just-built-clang for both building CompilerRT and
running its test suite, please consider relaxing that to a
stage0-clang for building and a just-built-clang for testing.

What if we don't have stage0-clang, and use gcc as a host compiler?

 

 Using
the just-built-clang to build compiler-rt is rough on package
managers.  You're basically stating that the compiler-rt build depends
on some unreleased feature of clang to *build* compiler-rt.

No, I want "use just-built clang" to be the default workflow. Currently, you

may use any host compiler to build compiler-rt sources (gcc, Clang, MSVC),

and we certainly want to keep it that way and not rely on some non-standard

or Clang-specific features. But we probably want to ship/release Clang with

a set of compiler-rt libraries that are built with *this* Clang. Again, one use case

when this can be necessary is targeting different platforms. For example, if

you use linux-x86 and you build LLVM/Clang capable of targeting both x86 and arm,

it certainly makes sense to build necessary compiler-rt libraries for all arches that Clang

can target. But you host compiler will most likely not be able to build arm version of

compiler-rt libs.

 

 To a
package manager, that means it should first build clang, then use that
clang to build compiler-rt, and then build a version of clang that
includes compiler-rt.

As I mentioned earlier, it's enough to build clang, and use it to build compiler-rt

in a proper location, no futher steps will be necessary.

--

Alexey Samsonov, MSK

[Greg Fitzgerald]

Hi Alexey,

> What if we don't have stage0-clang?

I'd like to answer this question in detail. I think if we're on the
same page about which compiler to use to build CompilerRT and where it
comes from, then the rest will, with any luck, fall into place.

Here is a dependency diagram for building and running the sanitizer test suite:

http://yuml.me/0324b101

Note that it mentions 3 compilers: Clang, the C++ compiler used to
build Clang, and a C++ cross-compiler used to build CompilerRT. If no
cross-compiler exists, we can bootstrap by building a Stage-0 Clang
with a C++ compiler. The full dependency diagram then becomes:

http://yuml.me/2633fbbd

When a C++ cross-compiler already exists (either by building it or
using clang-3.4), the dependency graph can be reduced by using the
same compiler to build Clang, LLVM and CompilerRT:

http://yuml.me/4b0473ae

http://yuml.me/9ff774bc

When building in this way, CompilerRT can expect the
CMAKE_CXX_COMPILER to be a cross-compiler with the same command-line
interface as Clang (cross-compile using -target and --sysroot flags).
This Stage-0 compiler cannot be used to run the Sanitizer test suite.
Instead, the Clang build should use add_subdirectory() to add the
CompilerRT 'test' directory to its build. No need for the CMake's
ExternalProject feature.

-Greg

[Alexey Samsonov]

On Tue, Mar 25, 2014 at 6:36 AM, Greg Fitzgerald <gar...@gmail.com> wrote:

Hi Alexey,

> What if we don't have stage0-clang?

I'd like to answer this question in detail.  I think if we're on the
same page about which compiler to use to build CompilerRT and where it
comes from, then the rest will, with any luck, fall into place.

Here is a dependency diagram for building and running the sanitizer test suite:

Why do you state that Clang depends on compiler-rt libs? Currently it doesn't:

compiler-rt libs are not necessary to link "clang" binary (as LLVM libs are).

фтв compiler-rt libs are optional in the sense that "clang" binary works perfectly fine if

they are missing (though not all of the features are supported).

IMO, the dependency diagram for sanitizer tests (in LLVM/Clang/compiler-rt build tree)

should be: http://yuml.me/27f7eca5

It's also interesting that "sanitizer test suite" might be used as a standalone test suite

for any compiler with a gcc-style commandline-interface: http://yuml.me/e984f26f

In particular, we can use sanitizer test suite from compiler-rt repository

to test sanitizers in GCC (this can be done already, and we will probably perform such a

testing during the next merge of sanitizer runtimes to GCC).

--

Alexey Samsonov, MSK

[Alexey Samsonov]

On Tue, Mar 25, 2014 at 4:57 PM, Alexey Samsonov <sams...@google.com> wrote:

On Tue, Mar 25, 2014 at 6:36 AM, Greg Fitzgerald <gar...@gmail.com> wrote:

Hi Alexey,

> What if we don't have stage0-clang?

I'd like to answer this question in detail.  I think if we're on the
same page about which compiler to use to build CompilerRT and where it
comes from, then the rest will, with any luck, fall into place.

Here is a dependency diagram for building and running the sanitizer test suite:

Why do you state that Clang depends on compiler-rt libs? Currently it doesn't:

compiler-rt libs are not necessary to link "clang" binary (as LLVM libs are).

фтв compiler-rt libs are optional in the sense that "clang" binary works perfectly fine if

they are missing (though not all of the features are supported).

Sorry, I meant "and compiler-rt libs are optional".

--

Alexey Samsonov, MSK