[LLVMdev] Sanitizers libs in Compiler-RT

[Renato Golin]

Hi all,

Can anyone explain me what was the rationale behind putting the sanitizer libraries in compiler-rt?

The sanitizers only work properly in x86_64 and I don't see anyone even testing to any other platform, while compiler-rt should be a substitute for libgcc on all platforms, at least that's the goal, and linking the success of the RT library to the success of the sanitizers is a bit of an oversight, IMHO.

I'm spending 99% of the time to make compiler-rt compiler on ARM using CMake by fiddling the sanitizers' tests, lit config files and CMake scripts, which is a bit annoying...

Long term, would it be possible / desirable to split them into compiler-rt and sanitizer-rt?

cheers,

--renato

[Alp Toker]

I don't know about the specific problems you've faced on ARM, but I'd
like to add that these problems also exist on x86, where the sanitizers
aren't fully supported on OS X, Windows or even various Linux
distributions outside the configuration the authors have.

Moreover the project seems to have a different development and review
policy to the rest of LLVM and it's been difficult at times to align
that with general interests on the clang side.

That said, let's engage the developers before jumping to conclusions.
The sanitizer work happens off in a corner and we tend to only notice it
when it gets in our way. If the sanitizers are spun off, it should be
done to make those developers work easier, not just ours.

Alp.

--
http://www.nuanti.com
the browser experts

_______________________________________________
LLVM Developers mailing list
LLV...@cs.uiuc.edu http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

[Alexey Samsonov]

Hi Renato,

On Thu, Jan 30, 2014 at 11:07 PM, Renato Golin <renato...@linaro.org> wrote:

Hi all,

Can anyone explain me what was the rationale behind putting the sanitizer libraries in compiler-rt?

The sanitizers only work properly in x86_64 and I don't see anyone even testing to any other platform, while compiler-rt should be a substitute for libgcc on all platforms, at least that's the goal, and linking the success of the RT library to the success of the sanitizers is a bit of an oversight, IMHO.

I'm spending 99% of the time to make compiler-rt compiler on ARM using CMake by fiddling the sanitizers' tests, lit config files and CMake scripts, which is a bit annoying...

If you're only interested in building compiler-rt library for ARM, and not interested in supporting sanitizers on that platform, you can just avoid building them on ARM. You're right that it takes explicit effort to enable building sanitizers on ARM, but IMO that's not accidental or bad - we *don't* want to build them unless there is an interest in support.

Long term, would it be possible / desirable to split them into compiler-rt and sanitizer-rt?

cheers,

--renato

_______________________________________________
LLVM Developers mailing list
LLV...@cs.uiuc.edu         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

--

Alexey Samsonov, MSK

[Reid Kleckner]

On Thu, Jan 30, 2014 at 11:07 AM, Renato Golin <renato...@linaro.org> wrote:

Hi all,

Can anyone explain me what was the rationale behind putting the sanitizer libraries in compiler-rt?

http://lists.cs.uiuc.edu/pipermail/llvmdev/2011-November/045444.html

Basically, compiler-rt contains the only runtime libraries we ship with Clang.  The sanitizers are runtime libraries shipped with clang (they have some version dependence), so they went in compiler-rt.  Now they are starting to feel much larger than compiler-rt, so perhaps they should be split out.

Or we can leave them there and solve your ARM build problems a different way.

[Renato Golin]

On 30 January 2014 20:33, Reid Kleckner <r...@google.com> wrote:

Basically, compiler-rt contains the only runtime libraries we ship with Clang.  The sanitizers are runtime libraries shipped with clang (they have some version dependence), so they went in compiler-rt.  Now they are starting to feel much larger than compiler-rt, so perhaps they should be split out.

Or we can leave them there and solve your ARM build problems a different way.

Right, I apologise for my lack of specificity... I do remember that thread, what I was referring to was exactly what you said.

My premise:

1. I want to make Compiler-RT work on ARM and be a drop-in replacement for libgcc, to the point where it'll become the standard compiler library for LLVM in the near future. My bold plans are to have that in 3.5, but I won't be unhappy if we at least get it working until then.

2. I don't want to have the huge effort right now, to port a library that was mainly developed by and on x86_64 *just because* I want compiler-rt to work on ARM.

This can work in several ways:

A. I compile the whole lot, but only grab the compiler-rt library when packaging.

B. I change the CMalke files to only compiler RT on ARM, or any other architecture that wants only RT

C. We split the libraries

If the consensus is that the sanitizers are a lot bigger than RT and should split for that reason, I'm fine with it. If not, I'm also fine with having a CMake configuration to only build what's needed. I'd be less fine with compiling everything, testing (or ignoring) everything, because that'll strain our native bots (that are still dead slow) and will also increase the signal-to-noise ration on bugs and crashes, but ultimately, I would be ok with it, as a first approach.

Makes sense?

cheers,

--renato

[Reid Kleckner]

I don't see any compelling reason to split the sanitizers out today.  I think the sanitizer guys would probably take a patch to CMakeLists.txt to disable the sanitizer RTLs.

I'm told that ASan supports ARM, but it's maybe not production quality, so I don't think disabling all sanitizers on non-x86 architectures will work.

[Renato Golin]

On 30 January 2014 21:50, Reid Kleckner <r...@google.com> wrote:

I don't see any compelling reason to split the sanitizers out today.

Clear, next.

I think the sanitizer guys would probably take a patch to CMakeLists.txt to disable the sanitizer RTLs.

Right. I'll first make it build and test, than I'll disable via a flag, since other people can work in parallel to make it work on ARM.

I'm told that ASan supports ARM, but it's maybe not production quality, so I don't think disabling all sanitizers on non-x86 architectures will work.

I hear Android folks use it, this is why I want to enable them by default. Even if the tests don't mean much, at least there's a seed from where they can pick that up and complete the port.

I'm fine on building them on my own, but buildbots and release builds should not need to, if it's not suitable.

 

Thanks, that answers all my questions. I'll proceed with my changes.

--renato

[Yury Gribov]

Renato Golin wrote:
>> I'm told that ASan supports ARM, but it's maybe not production
>> quality, so I don't think disabling all sanitizers on non-x86
>> architectures will work.
>
> I hear Android folks use it, this is why I want to enable them by
> default. Even if the tests don't mean much, at least there's a seed from
> where they can pick that up and complete the port.

We have been using ASan on ARM for last several months and we haven't
seen any serious problem in runtime library.
We are using GCC's ASan though.

-Y

[Chandler Carruth]

On Thu, Jan 30, 2014 at 1:54 PM, Renato Golin <renato...@linaro.org> wrote:

On 30 January 2014 21:50, Reid Kleckner <r...@google.com> wrote:

I don't see any compelling reason to split the sanitizers out today.

Clear, next.

Just as a side note, I had some thoughts about the organization of this stuff a while back that I wanted to replay here.

Fundamentally, I feel like we're in particular getting a few things conflated. Maybe separating them out helps:

- There is the repository "compiler-rt" that holds all of the runtime libraries that (if desired) need to be shipped along side the compiler. They're separated so that they can be omitted when they aren't needed, and potentially to isolate an unusual desired property: we would really like to (eventually) build them with the just-built compiler rather than the host compiler (where possible, clearly this requires the target compiler to be executable on the host).

- There is the core runtime library. Historically this was called 'compiler-rt' informally, but perhaps better called 'libclang_rt', which provides the core necessary runtime library facilities to compile C or C++ applications. It's analogous to libgcc but without some of the unwinding code (as I understand it, there may be details I'm wrong about here or glossing over, but it's not relevant to the organization of things).

- There are several optional runtime libraries to support specific compiler / toolchain features. These include the sanitizers and profiling libraries.

I think all of the libraries here make sense in the same repository because of the shared concerns of building runtime libraries. For example, it would be useful to compile them with the just-built-clang when target is executable on the host, and it would be useful to automatically cross compile versions of them for as many targets as are possible on the given host and supported by the host compiler (if used) and the just-built-clang.

However, the organization of the tree is ... very hard to understand. originally, there was only the one 'libclang_rt' library that had its generic C99 implementation in 'lib', and architecture-specific assembly routines when desirable in architecture subdirectories of lib. When we added new libraries, we put them in subdirectories of lib, making the whole thing kind of a mess. My suggestion to fix this was to create a "core" subdirectory of lib to contain the code used for 'libclang_rt'. "core" is a terrible name, but i've no better. suggestions welcome there. Then we would have a more sensible organization of the 'lib' tree.

I also think it might be useful to have a single large test tree (much like with llvm or clang) that has subdirectories for the various tests rather than test directories under lib/asan/ and friends, but maybe the sanitizer folks have objections to that. consistency seems a compelling reason here, but there might be other compelling concerns.

Now, the build system has always been problematic because the build system for this tree is *hard* and no one who has worked on it has really had the time to do an extremely thorough job and finish all aspects. It's a huge project. Right now, the makefile system does a good job of using the just-built-clang and can do a limited amount of cross-building runtimes for other targets. But the makefile system of compiler-rt is also terribly, terribly complex, doesn't follow the conventions of LLVM's makefiles, and is generally painful to maintain and update, so folks have been reluctant to flesh out its support for new libraries and other new things.

The CMake system is much cleaner in some respects (a bit less opaque to the folks trying to maintain it), but CMake makes it much harder to use the just-built-clang, especially for the C++ runtime code. The consequence is that we've never finished either the cross-building or just-built-clang-hosting features that are desirable. Any work toward these would be really awesome to see, but is a huge pile of work. Finally, when I was originally doing the CMake build for this I didn't understand what really needed to be done to build and use the core 'libclang_rt' library, and so I don't think I got it right. Some folks have sent patches to improve it, but I suspect it still really needs more work to be a solid system to use instead of libgcc. So I'm really excited about your emails. =] Having a more self-contained stack would be a significant improvement.

I think the obvious incremental steps are to disable building any parts of compiler-rt that don't build cleanly. There should never be a requirement for you to port an optional runtime library unless you need it. =] I think having good ports is important, but that should never block progress getting other thinsg ported and working well.

If it helps to reorganize things, I'm happy to even help there as much as I can. I agree that the organization isn't great, but I *really* didn't want to fight the makefile build system to do the reorganization myself, so its something that has lingered too long.

Sorry for the long ramble, but hopefully this gives you some of tho context.

[David Chisnall]

On 31 Jan 2014, at 08:12, Chandler Carruth <chan...@google.com> wrote:

> - There is the core runtime library. Historically this was called 'compiler-rt' informally, but perhaps better called 'libclang_rt', which provides the core necessary runtime library facilities to compile C or C++ applications. It's analogous to libgcc but without some of the unwinding code (as I understand it, there may be details I'm wrong about here or glossing over, but it's not relevant to the organization of things).

For some reason, the (generic, language-agnostic) unwind code is in libcxxabi. There was some discussion about moving it into the compiler-rt repository, where it would make sense. No one objected, but I'd rather not move it without a 'yes' from someone who is actually working on the code currently (I'd like to start factoring out the #ifdefs into a cleaner platform / architecture layer).

On FreeBSD, the 'libgcc' that we ship is now mostly code from the compiler-rt repository and it would be nice to have libgcc_s from the same source.

David

[Chandler Carruth]

On Fri, Jan 31, 2014 at 12:23 AM, David Chisnall <David.C...@cl.cam.ac.uk> wrote:

> - There is the core runtime library. Historically this was called 'compiler-rt' informally, but perhaps better called 'libclang_rt', which provides the core necessary runtime library facilities to compile C or C++ applications. It's analogous to libgcc but without some of the unwinding code (as I understand it, there may be details I'm wrong about here or glossing over, but it's not relevant to the organization of things).

For some reason, the (generic, language-agnostic) unwind code is in libcxxabi.  There was some discussion about moving it into the compiler-rt repository, where it would make sense.  No one objected, but I'd rather not move it without a 'yes' from someone who is actually working on the code currently (I'd like to start factoring out the #ifdefs into a cleaner platform / architecture layer).

Maybe shoot a fresh email and directly CC various folks like howard, doug, and nick? Seems reasonable to me.

I thought some of it still need libunwind (whichever of the various versions you like)? It would be good to get a nice, clean implementation of that functionality (whether based on one with an MIT license if there is such or not) if we don't already have it.

[Alexey Samsonov]

Chandler,

Thanks for the great overview.

On Fri, Jan 31, 2014 at 12:12 PM, Chandler Carruth <chan...@google.com> wrote:

On Thu, Jan 30, 2014 at 1:54 PM, Renato Golin <renato...@linaro.org> wrote:

On 30 January 2014 21:50, Reid Kleckner <r...@google.com> wrote:

I don't see any compelling reason to split the sanitizers out today.

Clear, next.

Just as a side note, I had some thoughts about the organization of this stuff a while back that I wanted to replay here.

Fundamentally, I feel like we're in particular getting a few things conflated. Maybe separating them out helps:

- There is the repository "compiler-rt" that holds all of the runtime libraries that (if desired) need to be shipped along side the compiler. They're separated so that they can be omitted when they aren't needed, and potentially to isolate an unusual desired property: we would really like to (eventually) build them with the just-built compiler rather than the host compiler (where possible, clearly this requires the target compiler to be executable on the host).

- There is the core runtime library. Historically this was called 'compiler-rt' informally, but perhaps better called 'libclang_rt', which provides the core necessary runtime library facilities to compile C or C++ applications. It's analogous to libgcc but without some of the unwinding code (as I understand it, there may be details I'm wrong about here or glossing over, but it's not relevant to the organization of things).

- There are several optional runtime libraries to support specific compiler / toolchain features. These include the sanitizers and profiling libraries.

I think all of the libraries here make sense in the same repository because of the shared concerns of building runtime libraries. For example, it would be useful to compile them with the just-built-clang when target is executable on the host, and it would be useful to automatically cross compile versions of them for as many targets as are possible on the given host and supported by the host compiler (if used) and the just-built-clang.

However, the organization of the tree is ... very hard to understand. originally, there was only the one 'libclang_rt' library that had its generic C99 implementation in 'lib', and architecture-specific assembly routines when desirable in architecture subdirectories of lib. When we added new libraries, we put them in subdirectories of lib, making the whole thing kind of a mess. My suggestion to fix this was to create a "core" subdirectory of lib to contain the code used for 'libclang_rt'. "core" is a terrible name, but i've no better. suggestions welcome there. Then we would have a more sensible organization of the 'lib' tree.

I can't suggest a better name than "core" (maybe, "runtime"?), but would *love* to put 'libclang_rt' into a separate directory under /compiler-rt/lib :) We should've done it long ago, IMO.

I also think it might be useful to have a single large test tree (much like with llvm or clang) that has subdirectories for the various tests rather than test directories under lib/asan/ and friends, but maybe the sanitizer folks have objections to that. consistency seems a compelling reason here, but there might be other compelling concerns.

Yes, I don't like the way testing is organized either. Originally it made sense to put sanitizer lit tests under lib/xsan/, but now we've got a lot of sanitizers, and a lot of duplicating configuration code. As Renato mentioned, browsing

through lit configs is a bit painful. I would like to proceed with re-organizing the tests for sanitizers, unless anyone objects. Kostya?

Now, the build system has always been problematic because the build system for this tree is *hard* and no one who has worked on it has really had the time to do an extremely thorough job and finish all aspects. It's a huge project. Right now, the makefile system does a good job of using the just-built-clang and can do a limited amount of cross-building runtimes for other targets. But the makefile system of compiler-rt is also terribly, terribly complex, doesn't follow the conventions of LLVM's makefiles, and is generally painful to maintain and update, so folks have been reluctant to flesh out its support for new libraries and other new things.

Note that it would be really hard to add support for running test suites into Makefile build system. It's also good for the one-time build only, not for continuous development - I don't think it respects dependencies correctly.

 

The CMake system is much cleaner in some respects (a bit less opaque to the folks trying to maintain it), but CMake makes it much harder to use the just-built-clang, especially for the C++ runtime code. The consequence is that we've never finished either the cross-building or just-built-clang-hosting features that are desirable.

I once tried to implement a pseudo-build-system for compiler-rt on top of CMake, so that we use "just-built" Clang instead of a host compiler, but failed miserably. Maybe I was doing wrong things, but I got the impression

that CMake isn't suited for switching the compiler on the fly :)

 

Any work toward these would be really awesome to see, but is a huge pile of work. Finally, when I was originally doing the CMake build for this I didn't understand what really needed to be done to build and use the core 'libclang_rt' library, and so I don't think I got it right. Some folks have sent patches to improve it, but I suspect it still really needs more work to be a solid system to use instead of libgcc. So I'm really excited about your emails. =] Having a more self-contained stack would be a significant improvement.

I think the obvious incremental steps are to disable building any parts of compiler-rt that don't build cleanly.

As I mentioned above, currently in CMake build system we try to avoid building anything if we're not sure we can produce a working and correct library on the host platform. That is, I still don't see what the problem is - it's relatively easy to enable building just the compiler-rt library on ARM and not enable building sanitizers on ARM.

 

There should never be a requirement for you to port an optional runtime library unless you need it. =] I think having good ports is important, but that should never block progress getting other thinsg ported and working well.

If it helps to reorganize things, I'm happy to even help there as much as I can. I agree that the organization isn't great, but I *really* didn't want to fight the makefile build system to do the reorganization myself, so its something that has lingered too long.

Sorry for the long ramble, but hopefully this gives you some of tho context.

_______________________________________________
cfe-dev mailing list
cfe...@cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev

--

Alexey Samsonov, MSK

[Kostya Serebryany]

Yes, I don't like the way testing is organized either. Originally it made sense to put sanitizer lit tests under lib/xsan/, but now we've got a lot of sanitizers, and a lot of duplicating configuration code. As Renato mentioned, browsing

through lit configs is a bit painful. I would like to proceed with re-organizing the tests for sanitizers, unless anyone objects. Kostya?

Any such changes would horrify me, but if you are willing to do them -- great.

Once thing which I'd like to have improved: we currently have several lit tests in asan or msan or tsan or lsan which really ought to be run for all of the tools (e.g. tests for interceptors)

--kcc 

[Iain Sandoe]

Hi Alexey,

On 31 Jan 2014, at 08:50, Alexey Samsonov wrote:

> Thanks for the great overview.

+1
( and +1 for David's comment about moving the unwind stuff into this area)

>
> On Fri, Jan 31, 2014 at 12:12 PM, Chandler Carruth <chan...@google.com> wrote:
>
> On Thu, Jan 30, 2014 at 1:54 PM, Renato Golin <renato...@linaro.org> wrote:
> On 30 January 2014 21:50, Reid Kleckner <r...@google.com> wrote:
> I don't see any compelling reason to split the sanitizers out today.
>
> Clear, next.
>
> Just as a side note, I had some thoughts about the organization of this stuff a while back that I wanted to replay here.
>
> Fundamentally, I feel like we're in particular getting a few things conflated. Maybe separating them out helps:
>
> - There is the repository "compiler-rt" that holds all of the runtime libraries that (if desired) need to be shipped along side the compiler. They're separated so that they can be omitted when they aren't needed, and potentially to isolate an unusual desired property: we would really like to (eventually) build them with the just-built compiler rather than the host compiler (where possible, clearly this requires the target compiler to be executable on the host).
>
> - There is the core runtime library. Historically this was called 'compiler-rt' informally, but perhaps better called 'libclang_rt', which provides the core necessary runtime library facilities to compile C or C++ applications. It's analogous to libgcc but without some of the unwinding code (as I understand it, there may be details I'm wrong about here or glossing over, but it's not relevant to the organization of things).
>
> - There are several optional runtime libraries to support specific compiler / toolchain features. These include the sanitizers and profiling libraries.
>
> I think all of the libraries here make sense in the same repository because of the shared concerns of building runtime libraries. For example, it would be useful to compile them with the just-built-clang when target is executable on the host, and it would be useful to automatically cross compile versions of them for as many targets as are possible on the given host and supported by the host compiler (if used) and the just-built-clang.
>
> However, the organization of the tree is ... very hard to understand. originally, there was only the one 'libclang_rt' library that had its generic C99 implementation in 'lib', and architecture-specific assembly routines when desirable in architecture subdirectories of lib. When we added new libraries, we put them in subdirectories of lib, making the whole thing kind of a mess. My suggestion to fix this was to create a "core" subdirectory of lib to contain the code used for 'libclang_rt'. "core" is a terrible name, but i've no better. suggestions welcome there. Then we would have a more sensible organization of the 'lib' tree.
>
> I can't suggest a better name than "core" (maybe, "runtime"?), but would *love* to put 'libclang_rt' into a separate directory under /compiler-rt/lib :) We should've done it long ago, IMO.
>
> I also think it might be useful to have a single large test tree (much like with llvm or clang) that has subdirectories for the various tests rather than test directories under lib/asan/ and friends, but maybe the sanitizer folks have objections to that. consistency seems a compelling reason here, but there might be other compelling concerns.
>
> Yes, I don't like the way testing is organized either. Originally it made sense to put sanitizer lit tests under lib/xsan/, but now we've got a lot of sanitizers, and a lot of duplicating configuration code. As Renato mentioned, browsing
> through lit configs is a bit painful. I would like to proceed with re-organizing the tests for sanitizers, unless anyone objects. Kostya?
>
>
> Now, the build system has always been problematic because the build system for this tree is *hard* and no one who has worked on it has really had the time to do an extremely thorough job and finish all aspects. It's a huge project. Right now, the makefile system does a good job of using the just-built-clang and can do a limited amount of cross-building runtimes for other targets. But the makefile system of compiler-rt is also terribly, terribly complex, doesn't follow the conventions of LLVM's makefiles, and is generally painful to maintain and update, so folks have been reluctant to flesh out its support for new libraries and other new things.
>
> Note that it would be really hard to add support for running test suites into Makefile build system. It's also good for the one-time build only, not for continuous development - I don't think it respects dependencies correctly.

Please could you expand upon this?
Certainly, one can expect to test the runtime on the host?
likewise, any pointers to where you see wrong dependencies would be appreciated.

At present, config & make appears to build a sensible lib using the just-built compiler [and in the general case, that's the only sensible solution, since the host might not have cross-tools for the set of archs you want to support with llvm/clang]

In some future ideal world, we might cook up a cross-testing environment - of course, it's a given that that would require testers to have access to the target hardware they wanted to test on.

Iain

>
>
> The CMake system is much cleaner in some respects (a bit less opaque to the folks trying to maintain it), but CMake makes it much harder to use the just-built-clang, especially for the C++ runtime code. The consequence is that we've never finished either the cross-building or just-built-clang-hosting features that are desirable.
>
> I once tried to implement a pseudo-build-system for compiler-rt on top of CMake, so that we use "just-built" Clang instead of a host compiler, but failed miserably. Maybe I was doing wrong things, but I got the impression
> that CMake isn't suited for switching the compiler on the fly :)
>
> Any work toward these would be really awesome to see, but is a huge pile of work. Finally, when I was originally doing the CMake build for this I didn't understand what really needed to be done to build and use the core 'libclang_rt' library, and so I don't think I got it right. Some folks have sent patches to improve it, but I suspect it still really needs more work to be a solid system to use instead of libgcc. So I'm really excited about your emails. =] Having a more self-contained stack would be a significant improvement.
>
>
> I think the obvious incremental steps are to disable building any parts of compiler-rt that don't build cleanly.
>
> As I mentioned above, currently in CMake build system we try to avoid building anything if we're not sure we can produce a working and correct library on the host platform. That is, I still don't see what the problem is - it's relatively easy to enable building just the compiler-rt library on ARM and not enable building sanitizers on ARM.
>
> There should never be a requirement for you to port an optional runtime library unless you need it. =] I think having good ports is important, but that should never block progress getting other thinsg ported and working well.
>
> If it helps to reorganize things, I'm happy to even help there as much as I can. I agree that the organization isn't great, but I *really* didn't want to fight the makefile build system to do the reorganization myself, so its something that has lingered too long.
>
>
> Sorry for the long ramble, but hopefully this gives you some of tho context.
>
> _______________________________________________
> cfe-dev mailing list
> cfe...@cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
>
>
>
> --
> Alexey Samsonov, MSK

[Alexey Samsonov]

Well, LLVM/Clang's configure+make and compiler-rt's make are disjoint - when you run "make" in Clang build tree, at one point it simply invokes, the Makefile in compiler-rt directory. But if we want more functionality, like running the tests, we ought to have a single build system, single set of "targets" (binaries, libraries, test suites) with dependencies between them. For example, I want to have "make check-asan" command which I can run from the root of the LLVM/Clang's build tree. More, if I run "make check-asan", I need to scan the dependencies of that test suite, and:

1) If ASan runtime has changed, re-build it, and re-run tests.

2) If FileCheck or lit sources have changed, I should re-build them (but not ASan runtime) and re-run tests.

3) If Clang sources have changed, I should re-build Clang, re-build ASan runtime (*) and re-run tests. 

With CMake we're able to do all that (except for (*), as we build runtimes with host compiler) at the moment.

 

Certainly, one can expect to test the runtime on the host?
likewise, any pointers to where you see wrong dependencies would be appreciated.

IIRC, "make check-all" in configure+make build tree doesn't re-build the necessary libraries automatically.

 

At present, config & make appears to build a sensible lib using the just-built compiler [and in the general case, that's the only sensible solution, since the host might not have cross-tools for the set of archs you want to support with llvm/clang]

Yes, that's what configure+make is good at.

In some future ideal world, we might cook up a cross-testing environment - of course, it's a given that that would require testers to have access to the target hardware they wanted to test on.

Iain

>
>
> The CMake system is much cleaner in some respects (a bit less opaque to the folks trying to maintain it), but CMake makes it much harder to use the just-built-clang, especially for the C++ runtime code. The consequence is that we've never finished either the cross-building or just-built-clang-hosting features that are desirable.
>
> I once tried to implement a pseudo-build-system for compiler-rt on top of CMake, so that we use "just-built" Clang instead of a host compiler, but failed miserably. Maybe I was doing wrong things, but I got the impression
> that CMake isn't suited for switching the compiler on the fly :)
>
> Any work toward these would be really awesome to see, but is a huge pile of work. Finally, when I was originally doing the CMake build for this I didn't understand what really needed to be done to build and use the core 'libclang_rt' library, and so I don't think I got it right. Some folks have sent patches to improve it, but I suspect it still really needs more work to be a solid system to use instead of libgcc. So I'm really excited about your emails. =] Having a more self-contained stack would be a significant improvement.
>
>
> I think the obvious incremental steps are to disable building any parts of compiler-rt that don't build cleanly.
>
> As I mentioned above, currently in CMake build system we try to avoid building anything if we're not sure we can produce a working and correct library on the host platform. That is, I still don't see what the problem is - it's relatively easy to enable building just the compiler-rt library on ARM and not enable building sanitizers on ARM.
>
> There should never be a requirement for you to port an optional runtime library unless you need it. =] I think having good ports is important, but that should never block progress getting other thinsg ported and working well.
>
> If it helps to reorganize things, I'm happy to even help there as much as I can. I agree that the organization isn't great, but I *really* didn't want to fight the makefile build system to do the reorganization myself, so its something that has lingered too long.
>
>
> Sorry for the long ramble, but hopefully this gives you some of tho context.
>
> _______________________________________________
> cfe-dev mailing list
> cfe...@cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
>
>
>
> --
> Alexey Samsonov, MSK

> _______________________________________________
> LLVM Developers mailing list
> LLV...@cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

--

Alexey Samsonov, MSK

[Evgeniy Stepanov]

As Kostya mentioned, there are many tests in compiler-rt that should
be shared between several sanitizers. This has bitten us more than
once in the past.

On Fri, Jan 31, 2014 at 1:04 PM, Iain Sandoe <ia...@codesourcery.com> wrote:

I'd very much like to have this for Android target.

[Renato Golin]

On 31 January 2014 09:22, Alexey Samsonov <sams...@google.com> wrote:

1) If ASan runtime has changed, re-build it, and re-run tests.

2) If FileCheck or lit sources have changed, I should re-build them (but not ASan runtime) and re-run tests.

3) If Clang sources have changed, I should re-build Clang, re-build ASan runtime (*) and re-run tests. 

Yeah, this is the major pain I'm having right now, as there isn't a target that I can make to re-compile the rt dependencies, not even check-all does that. (I have no idea how it gets compiled in the first place!).

Maybe anyone knows of a way I can force it to re-compile, like removing the build directory just for the rt or something.

IIRC, "make check-all" in configure+make build tree doesn't re-build the necessary libraries automatically.

Correct.

--renato

[Alexey Samsonov]

On Fri, Jan 31, 2014 at 1:35 PM, Renato Golin <renato...@linaro.org> wrote:

On 31 January 2014 09:22, Alexey Samsonov <sams...@google.com> wrote:

1) If ASan runtime has changed, re-build it, and re-run tests.

2) If FileCheck or lit sources have changed, I should re-build them (but not ASan runtime) and re-run tests.

3) If Clang sources have changed, I should re-build Clang, re-build ASan runtime (*) and re-run tests. 

Yeah, this is the major pain I'm having right now, as there isn't a target that I can make to re-compile the rt dependencies, not even check-all does that. (I have no idea how it gets compiled in the first place!).

 

Maybe anyone knows of a way I can force it to re-compile, like removing the build directory just for the rt or something.

In CMake build system you can just run "make compiler-rt".

Not sure how to do this in configure+make w/o "make clean"...

 

IIRC, "make check-all" in configure+make build tree doesn't re-build the necessary libraries automatically.

Correct.

--renato

--

Alexey Samsonov, MSK

[Renato Golin]

On 31 January 2014 08:50, Alexey Samsonov <sams...@google.com> wrote:

That is, I still don't see what the problem is - it's relatively easy to enable building just the compiler-rt library on ARM and not enable building sanitizers on ARM.

For some reason, when I added the compiler-rt directory (even before my CMake changes), the Clang tests *required* the Asan libraries in lib/clang/3.5/linux, which is why I added it in the first place. I think this is wrong and should be fixed (though I have no idea how) in the Clang CMake files.

With that fixed, I could just enable the RT libs and worry about the Asan later. Right now, I can't.

--renato

[Renato Golin]

On 31 January 2014 09:47, Alexey Samsonov <sams...@google.com> wrote:

In CMake build system you can just run "make compiler-rt".

Not sure how to do this in configure+make w/o "make clean"...

Make compiler-rt doesn't re-make it on changes. Nor does make check-asan or check-all.

cheers,

--renato

[Alexey Samsonov]

On Fri, Jan 31, 2014 at 1:47 PM, Renato Golin <renato...@linaro.org> wrote:

On 31 January 2014 08:50, Alexey Samsonov <sams...@google.com> wrote:

That is, I still don't see what the problem is - it's relatively easy to enable building just the compiler-rt library on ARM and not enable building sanitizers on ARM.

For some reason, when I added the compiler-rt directory (even before my CMake changes), the Clang tests *required* the Asan libraries in lib/clang/3.5/linux, which is why I added it in the first place. I think this is wrong and should be fixed (though I have no idea how) in the Clang CMake files.

Huh? What tests are you talking about? I thought that tests for Clang driver verifiy that the command "clang++ -fsanitize=address ...." produce a correct linker invocation (with path to /lib/clang/3.5/linux/libclang_rt.asan-arm.a), but don't actually require the libraries to be built there. If it's not the case, we should fix that.

With that fixed, I could just enable the RT libs and worry about the Asan later. Right now, I can't.

--renato

--

Alexey Samsonov, MSK

[Alexey Samsonov]

Right now, in my CMake build tree I've ran "make compiler-rt", then changed compiler-rt/lib/absvsi2.c, then ran "make compiler-rt" again, and saw that lib/clang/3.5/lib/linux/libclang_rt.x86_64.a was indeed rebuilt.

 

cheers,

--renato

--

Alexey Samsonov, MSK

[Renato Golin]

On 31 January 2014 10:01, Alexey Samsonov <sams...@google.com> wrote:

Right now, in my CMake build tree I've ran "make compiler-rt", then changed compiler-rt/lib/absvsi2.c, then ran "make compiler-rt" again, and saw that lib/clang/3.5/lib/linux/libclang_rt.x86_64.a was indeed rebuilt.

I was changing the lit.config files, probably they're not in the dependency graph?

--renato

[Renato Golin]

On 31 January 2014 09:59, Alexey Samsonov <sams...@google.com> wrote:

Huh? What tests are you talking about? I thought that tests for Clang driver verifiy that the command "clang++ -fsanitize=address ...." produce a correct linker invocation (with path to /lib/clang/3.5/linux/libclang_rt.asan-arm.a), but don't actually require the libraries to be built there. If it's not the case, we should fix that.

I got linker errors, since /lib/clang/3.5/linux/libclang_rt.asan-armv7l.a (or arm for that matter) wasn't there, since "arm" is not a recognized architecture.

My point is that Clang sanitizer tests should *only* run IFF:

1. Compiler-rt is present AND

2. The architecture is recognized AND

3. The Asan/Lsan/UBsan etc libraries were enabled via some build option (individually checked).

I may be wrong, but right now, it seems that they all run once compiler-rt is available.

cheers,

--renato

[Renato Golin]

Correction, not only the lit.config but also the unit-tests within asan (source files). They didn't trigger their own build either.

cheers,

--renato

[Alexey Samsonov]

On Fri, Jan 31, 2014 at 2:30 PM, Renato Golin <renato...@linaro.org> wrote:

On 31 January 2014 09:59, Alexey Samsonov <sams...@google.com> wrote:

Huh? What tests are you talking about? I thought that tests for Clang driver verifiy that the command "clang++ -fsanitize=address ...." produce a correct linker invocation (with path to /lib/clang/3.5/linux/libclang_rt.asan-arm.a), but don't actually require the libraries to be built there. If it's not the case, we should fix that.

I got linker errors, since /lib/clang/3.5/linux/libclang_rt.asan-armv7l.a (or arm for that matter) wasn't there, since "arm" is not a recognized architecture.

What tests under tools/clang/test were failing and with what errors? I thought that we usually test Clang with -### option, that is, only look at the

args we're going to pass to linker, not actually invoke it.

My point is that Clang sanitizer tests should *only* run IFF:

1. Compiler-rt is present AND

2. The architecture is recognized AND

3. The Asan/Lsan/UBsan etc libraries were enabled via some build option (individually checked).

I may be wrong, but right now, it seems that they all run once compiler-rt is available.

I think that Clang tests just shouldn't depend on the presence of compiler-rt and should always run.

 

cheers,

--renato

--

Alexey Samsonov, MSK

[Alexey Samsonov]

How can I reproduce it?

 

cheers,

--renato

--

Alexey Samsonov, MSK

[Renato Golin]

On 31 January 2014 10:54, Alexey Samsonov <sams...@google.com> wrote:

How can I reproduce it?

Do you have an ARM box? ;) I think any non-supported platform (mips?) would do, but now I'm not sure any more. It takes so long to build things that I get lost in the process.

I'm posting the Clang patch soon, which should make my compiler-rt changes harmless when they land. If they work, I think we should just get something in and trim the problems as they appear, one by one, rather than chase ghosts in this thread (which is now just us).

Let's close this here, and follow up on the two phab reviews.

cheers,

--renato

[Nick Kledzik]

On Jan 31, 2014, at 12:29 AM, Chandler Carruth <chan...@google.com> wrote:
> I thought some of it still need libunwind (whichever of the various versions you like)? It would be good to get a nice, clean implementation of that functionality (whether based on one with an MIT license if there is such or not) if we don't already have it.

The unwinder in libcxxabi implements both the _Unwind_* functions needed by libcxxabi and the unw_* functions that are the “libunwind” API. There is nothing more needed.


On Jan 31, 2014, at 12:23 AM, David Chisnall <David.C...@cl.cam.ac.uk> wrote

> On 31 Jan 2014, at 08:12, Chandler Carruth <chan...@google.com> wrote:
>
>> - There is the core runtime library. Historically this was called 'compiler-rt' informally, but perhaps better called 'libclang_rt', which provides the core necessary runtime library facilities to compile C or C++ applications. It's analogous to libgcc but without some of the unwinding code (as I understand it, there may be details I'm wrong about here or glossing over, but it's not relevant to the organization of things).
>
> For some reason, the (generic, language-agnostic) unwind code is in libcxxabi. There was some discussion about moving it into the compiler-rt repository, where it would make sense. No one objected, but I'd rather not move it without a 'yes' from someone who is actually working on the code currently (I'd like to start factoring out the #ifdefs into a cleaner platform / architecture layer).

The logic was that libcxxabi is the biggest client of the unwinder, and well, compiler-rt was already complicated enough ;-) That said, if we had a nice clean, scalable model for organizing all the runtime support libraries, I’d be happy to migrate the unwinder there.

Also, to help explain my bias, at Apple, the unwinder (and libc++abi) are dylibs that ship as part of the OS. They have nothing to do with the compiler.


A couple of other random thoughts about compiler-rt:

* One of the makefile dimensions of complexity is the ability to build optimized, profile, and debug copies of everything. This was once needed at Apple, but no longer is necessary.

* One of the interesting things about compiler-rt is the static library to dynamic library migration (e.g. libgcc.a vs libgcc_s.so, or on Darwin libclang_*.a vs libSystem.dylib). If the shared library ships independently from the compiler, then the compiler may need a .a file that can ship with it that contains any support functions not available in a shared library on the target. Currently, it is a very manual process to figure out which functions are needed where.

* It would be nice if the clang build system could output a list of all possible support functions it might need for compiler being built. That list could drive what parts of compiler-rt need to be built.

So, to me an ideal build system for compiler-rt would not just compile the snippets of code, it would figure out which snippets to build based on what the compiler needs and what the OS needs/provides.

-Nick

[Renato Golin]

On 1 February 2014 00:44, Nick Kledzik <kle...@apple.com> wrote:

* One of the interesting things about compiler-rt is the static library to dynamic library migration (e.g. libgcc.a vs libgcc_s.so, or on Darwin libclang_*.a vs libSystem.dylib).  If the shared library ships independently from the compiler, then the compiler may need a .a file that can ship with it that contains any support functions not available in a shared library on the target.  Currently, it is a very manual process to figure out which functions are needed where.

* It would be nice if the clang build system could output a list of all possible support functions it might need for compiler being built.  That list could drive what parts of compiler-rt need to be built.

So, to me an ideal build system for compiler-rt would not just compile the snippets of code, it would figure out which snippets to build based on what the compiler needs and what the OS needs/provides.

Hi Nick,

These features would be great, but I think it's too complex to migrate from what we have today to that scenario and people won't buy-in that easily.

I think the fact that all sanitizers, profilers and possibly unwinders have their libraries in there is a good reason for us to have separate builds of each library and treat "Compiler-RT" as a repository for run-time libraries, rather than a library in itself.

I'm in favour of naming what we call today "compiler-rt" to "libclang" and moving libcxxabi into it as "libunwind" or anything relevant, but to softly migrate the interactions between RT and Clang over time.

A few steps like:

1. Move libraries in, rename, compile everything everytime

2. Separate libraries' build systems, disable via flags / arch support (on both clang and rt)

3. Get Clang to list *libraries* needed, and make RT's make system to only build those

4. Get Clang to list *functions* needed, and update RT's make system accordingly

I think having step 3 would be amazing, but 2 is already good for all practical purposes. Step 4 is way past *my* needs, but I don't think it's a bad thing to have.

cheers,

--renato

[Iain Sandoe]

Hi Nick,

A few of my colleagues and I have been giving these general issues some serious thought - with a view to floating an abstract for a BOF proposal for Edinburgh - seems like a topic with a fair few interests to be resolved.

I think it is probably clear that we can't achieve everything that everyone wants in a single step - but also that having an agreed "end goal" and a possible route to get there is essential if we want to avoid chaos.

On 1 Feb 2014, at 00:44, Nick Kledzik wrote:

> On Jan 31, 2014, at 12:29 AM, Chandler Carruth <chan...@google.com> wrote:
>> I thought some of it still need libunwind (whichever of the various versions you like)? It would be good to get a nice, clean implementation of that functionality (whether based on one with an MIT license if there is such or not) if we don't already have it.
> The unwinder in libcxxabi implements both the _Unwind_* functions needed by libcxxabi and the unw_* functions that are the “libunwind” API. There is nothing more needed.

So that component *could* comprise a stand-alone unwind dylib/so/a?

> On Jan 31, 2014, at 12:23 AM, David Chisnall <David.C...@cl.cam.ac.uk> wrote
>> On 31 Jan 2014, at 08:12, Chandler Carruth <chan...@google.com> wrote:
>>
>>> - There is the core runtime library. Historically this was called 'compiler-rt' informally, but perhaps better called 'libclang_rt', which provides the core necessary runtime library facilities to compile C or C++ applications. It's analogous to libgcc but without some of the unwinding code (as I understand it, there may be details I'm wrong about here or glossing over, but it's not relevant to the organization of things).
>>
>> For some reason, the (generic, language-agnostic) unwind code is in libcxxabi. There was some discussion about moving it into the compiler-rt repository, where it would make sense. No one objected, but I'd rather not move it without a 'yes' from someone who is actually working on the code currently (I'd like to start factoring out the #ifdefs into a cleaner platform / architecture layer).
>
> The logic was that libcxxabi is the biggest client of the unwinder, and well, compiler-rt was already complicated enough ;-) That said, if we had a nice clean, scalable model for organizing all the runtime support libraries, I’d be happy to migrate the unwinder there.
>
> Also, to help explain my bias, at Apple, the unwinder (and libc++abi) are dylibs that ship as part of the OS. They have nothing to do with the compiler.

Thus, a comment about language runtimes.

So, [in respect of language runtime libraries] a useful model moving forward should be that the compiler does not assume that target runtimes are installed as part of the compiler. Essentially, the target system (even if it is the same as the host) might not have the compiler installed.

However, the compiler itself needs runtimes (or at least stub libraries) to link against for each supported target - so ideally (IMO) a set of sysroots for these supported targets would be installed relative to compiler - such that the whole package is completely relocatable. In my ideal world the sysroots would be installed in a "SDK/sysroots" directory parallel to the bin, lib etc. so that they remain distinct from the compiler itself. It might actually be tidier to consider the host as "just another cross target" - so that the entire process is uniform.

====

A comment about the unwind library:

Assume:
(1) that there is only sensibly one unwinder [a design that allows multiple unwinders would solve some problems]
(2) that there might be other clients for the unwind library in addition to c++ (e.g. an Ada, Fortran, other ObjC dialects .. etc.)
It might be that new FEs would choose interoperability with c++ (even quite likely) - but it seems a little unreasonable for the compiler to impose that as a pre-constraint.

(3) categories of system

(a) ones where the entire stack is clang-based like modern OSX or freeBSD10.
(b) ones where the clang components must interface with an existing unwinder (e.g. older OSX or Linux)
(c) bare metal.

We have found over the course of darwin's history that binding the unwinder into a library (although that was not a Darwin-specific decision) made it difficult to maintain things in proper isolation - and made it very difficult to have tidy implementations of "3rd party" compiler ports.

Given that clang is a '3rd party' port to all the systems in categories (b) and (c) above - it would be a good idea to make maintenance and installation easier.

In fact, what this argues for is probably an unwinder.dylib / unwinder.so / unwinder.a since this component can be corrected or replaced without needing to install dependent libs (ABI constraints notwithstanding).

> A couple of other random thoughts about compiler-rt:
>
> * One of the makefile dimensions of complexity is the ability to build optimized, profile, and debug copies of everything. This was once needed at Apple, but no longer is necessary.
>
> * One of the interesting things about compiler-rt is the static library to dynamic library migration (e.g. libgcc.a vs libgcc_s.so, or on Darwin libclang_*.a vs libSystem.dylib). If the shared library ships independently from the compiler, then the compiler may need a .a file that can ship with it that contains any support functions not available in a shared library on the target. Currently, it is a very manual process to figure out which functions are needed where.
>
> * It would be nice if the clang build system could output a list of all possible support functions it might need for compiler being built. That list could drive what parts of compiler-rt need to be built.
>
> So, to me an ideal build system for compiler-rt would not just compile the snippets of code, it would figure out which snippets to build based on what the compiler needs and what the OS needs/provides.

This "library of fragments" would be probably the only runtime candidate for installation inside the actual compiler tree since it's essentially bound to a compiler version.

cheers
Iain

[Alex Rosenberg]

On Jan 30, 2014, at 1:50 PM, Reid Kleckner <r...@google.com> wrote:

On Thu, Jan 30, 2014 at 1:19 PM, Renato Golin <renato...@linaro.org> wrote:

On 30 January 2014 20:33, Reid Kleckner <r...@google.com> wrote:

Basically, compiler-rt contains the only runtime libraries we ship with Clang.  The sanitizers are runtime libraries shipped with clang (they have some version dependence), so they went in compiler-rt.  Now they are starting to feel much larger than compiler-rt, so perhaps they should be split out.

Or we can leave them there and solve your ARM build problems a different way.

Right, I apologise for my lack of specificity... I do remember that thread, what I was referring to was exactly what you said.

My premise:

1. I want to make Compiler-RT work on ARM and be a drop-in replacement for libgcc, to the point where it'll become the standard compiler library for LLVM in the near future. My bold plans are to have that in 3.5, but I won't be unhappy if we at least get it working until then.

2. I don't want to have the huge effort right now, to port a library that was mainly developed by and on x86_64 *just because* I want compiler-rt to work on ARM.

This can work in several ways:

A. I compile the whole lot, but only grab the compiler-rt library when packaging.

B. I change the CMalke files to only compiler RT on ARM, or any other architecture that wants only RT

C. We split the libraries

If the consensus is that the sanitizers are a lot bigger than RT and should split for that reason, I'm fine with it. If not, I'm also fine with having a CMake configuration to only build what's needed. I'd be less fine with compiling everything, testing (or ignoring) everything, because that'll strain our native bots (that are still dead slow) and will also increase the signal-to-noise ration on bugs and crashes, but ultimately, I would be ok with it, as a first approach.

I don't see any compelling reason to split the sanitizers out today.  I think the sanitizer guys would probably take a patch to CMakeLists.txt to disable the sanitizer RTLs.

One organizational thought to add:

For us, the sanitizers contain behaviors/syscalls that are not valid on consumer hardware. They perform "debug" functionality and thus can only operate on special developer kits. Additionally, we need to put this type of code into a dynamic library instead of a static library to avoid versioning problems.

I agree that this is all part of compiler_rt as a project/repo, but the sanitizers should remain segregated into their own runtime library.

Alex

[Chandler Carruth]

On Sat, Feb 1, 2014 at 8:41 AM, Alex Rosenberg <al...@ohmantics.com> wrote:

One organizational thought to add:

For us, the sanitizers contain behaviors/syscalls that are not valid on consumer hardware. They perform "debug" functionality and thus can only operate on special developer kits. Additionally, we need to put this type of code into a dynamic library instead of a static library to avoid versioning problems.

I agree that this is all part of compiler_rt as a project/repo, but the sanitizers should remain segregated into their own runtime library.

I think many, many folks need the sanitizers to be segregated into their own runtime libraries.

In fact, from a functional perspective, many of the sanitizers' runtimes are really mutually exclusive and should never even be in the same runtime library.

[David Chisnall]

On 1 Feb 2014, at 00:44, Nick Kledzik <kle...@apple.com> wrote:

> On Jan 31, 2014, at 12:23 AM, David Chisnall <David.C...@cl.cam.ac.uk> wrote
>> On 31 Jan 2014, at 08:12, Chandler Carruth <chan...@google.com> wrote:
>>
>>> - There is the core runtime library. Historically this was called 'compiler-rt' informally, but perhaps better called 'libclang_rt', which provides the core necessary runtime library facilities to compile C or C++ applications. It's analogous to libgcc but without some of the unwinding code (as I understand it, there may be details I'm wrong about here or glossing over, but it's not relevant to the organization of things).
>>
>> For some reason, the (generic, language-agnostic) unwind code is in libcxxabi. There was some discussion about moving it into the compiler-rt repository, where it would make sense. No one objected, but I'd rather not move it without a 'yes' from someone who is actually working on the code currently (I'd like to start factoring out the #ifdefs into a cleaner platform / architecture layer).
>
> The logic was that libcxxabi is the biggest client of the unwinder, and well, compiler-rt was already complicated enough ;-) That said, if we had a nice clean, scalable model for organizing all the runtime support libraries, I’d be happy to migrate the unwinder there.

On FreeBSD, the main consumers of the unwind library are:

- libgcc_s / compiler-rt (our libgcc_s.so actually uses compiler-rt code here) for the C personality function.
- libcxxrt for the C++ personality function
- libobjc for the Objective-C and Objective-C++ personality functions (libobjcxx on older versions where the C++ ABI library is not dynamically linked).

Thus we'd like to import and contribute to a cleaner unwind library, but for us the code will end up in a libgcc_s replacement, and so having it in the same place as the rest of the libgcc_s replacement code (compiler-rt) seems more obvious.

Having a library as a child of one of its consumers is a bit odd.

> Also, to help explain my bias, at Apple, the unwinder (and libc++abi) are dylibs that ship as part of the OS. They have nothing to do with the compiler.

We also ship compiler-rt as part of the OS, not specifically as part of the compiler (the same is true of libgcc* and crt*), so I don't understand this argument.

David

[Sean Silva]

On Sat, Feb 1, 2014 at 6:10 AM, Renato Golin <renato...@linaro.org> wrote:

On 1 February 2014 00:44, Nick Kledzik <kle...@apple.com> wrote:

* One of the interesting things about compiler-rt is the static library to dynamic library migration (e.g. libgcc.a vs libgcc_s.so, or on Darwin libclang_*.a vs libSystem.dylib).  If the shared library ships independently from the compiler, then the compiler may need a .a file that can ship with it that contains any support functions not available in a shared library on the target.  Currently, it is a very manual process to figure out which functions are needed where.

* It would be nice if the clang build system could output a list of all possible support functions it might need for compiler being built.  That list could drive what parts of compiler-rt need to be built.

So, to me an ideal build system for compiler-rt would not just compile the snippets of code, it would figure out which snippets to build based on what the compiler needs and what the OS needs/provides.

Hi Nick,

These features would be great, but I think it's too complex to migrate from what we have today to that scenario and people won't buy-in that easily.

I think the fact that all sanitizers, profilers and possibly unwinders have their libraries in there is a good reason for us to have separate builds of each library and treat "Compiler-RT" as a repository for run-time libraries, rather than a library in itself.

I'm in favour of naming what we call today "compiler-rt" to "libclang" and moving libcxxabi into it as "libunwind" or anything relevant, but to softly migrate the interactions between RT and Clang over time.

FYI the name "libclang" is already in use <http://clang.llvm.org/docs/Tooling.html#libclang>, so a different name would need to be chosen.

-- Sean Silva

 

A few steps like:

1. Move libraries in, rename, compile everything everytime

2. Separate libraries' build systems, disable via flags / arch support (on both clang and rt)

3. Get Clang to list *libraries* needed, and make RT's make system to only build those

4. Get Clang to list *functions* needed, and update RT's make system accordingly

I think having step 3 would be amazing, but 2 is already good for all practical purposes. Step 4 is way past *my* needs, but I don't think it's a bad thing to have.

cheers,

--renato

[Renato Golin]

On 3 February 2014 02:36, Sean Silva <chiso...@gmail.com> wrote:

FYI the name "libclang" is already in use <http://clang.llvm.org/docs/Tooling.html#libclang>, so a different name would need to be chosen.

Yes, my laziness is immense. I think the proposed name is libclang_rt etc.

--renato

[Evgeniy Stepanov]

Hi,

> For us, the sanitizers contain behaviors/syscalls that are not valid on
> consumer hardware. They perform "debug" functionality and thus can only
> operate on special developer kits. Additionally, we need to put this type of
> code into a dynamic library instead of a static library to avoid versioning
> problems.

Could you elaborate on this? I can't imagine how making sanitizer
runtime a shared library would help with the versioning problems,
other than create them. As a static library it is always in sync with
the compiler (unless someone wants to distribute them with an OS, of
course).

[David Chisnall]

On 3 Feb 2014, at 08:32, Evgeniy Stepanov <eug...@google.com> wrote:

>> For us, the sanitizers contain behaviors/syscalls that are not valid on
>> consumer hardware. They perform "debug" functionality and thus can only
>> operate on special developer kits. Additionally, we need to put this type of
>> code into a dynamic library instead of a static library to avoid versioning
>> problems.
>
> Could you elaborate on this? I can't imagine how making sanitizer
> runtime a shared library would help with the versioning problems,
> other than create them. As a static library it is always in sync with
> the compiler (unless someone wants to distribute them with an OS, of
> course).

There are two issues with compatibility in the sanitiser libraries:

- Compiler-Runtime interfaces
- Runtime-OS interfaces

The first is made easier with a static library, the latter is made much harder because modifying the OS (including libc and system calls) can break the static library.

Both are made easy with a shared library. The first can be addressed using symbol versioning, the second by updating the shared library whenever the underlying interfaces change.

David