building shared libraries with tsan?

[Toebs Douglass]

So, Gents,

Back to the problem in hand.

I've modified my makefiles to use gcc for compiling and linking.

However, I'm still confused.

I have a library, which can be built as a static or shared library.

I have an executable, which links to the library.

The tsan use instructions are;

"Simply compile your program with -fsanitize=thread -fPIE and link it
with -fsanitize=thread -pie."

I've read -pie is only for executables. I think then these instructions
are then for producing an executable.

So I think my question must be this : do I need to build the library
*and* the executable with tsan, or do I only need to build the
executable with tsan?

If it's only the exe, then fine, I know what to do.

However, if I need to build the library with tsan, well, the library can
be static or dynamic.

If I'm building dynamic, then I must use -fPIC. Do I then use PIE or
PIC with the executable? I think I must use PIC...

If I'm building an archive, I think I can then use PIE, since it'll be
linked directly into the executable, which would then mean I can as
specified in the instructions use PIE to build the executable...
however, I've read -static should not be used with -pie, which confuses
the hell out of me. Is it actually meant that archives *typically* are
not built -pie, so don't use -pie with -static?

[Dmitry Vyukov]

On Sun, Oct 12, 2014 at 2:25 AM, Toebs Douglass <to...@winterflaw.net> wrote:
> So, Gents,
>
> Back to the problem in hand.
>
> I've modified my makefiles to use gcc for compiling and linking.
>
> However, I'm still confused.
>
> I have a library, which can be built as a static or shared library.
>
> I have an executable, which links to the library.
>
> The tsan use instructions are;
>
> "Simply compile your program with -fsanitize=thread -fPIE and link it with
> -fsanitize=thread -pie."
>
> I've read -pie is only for executables. I think then these instructions are
> then for producing an executable.
>
> So I think my question must be this : do I need to build the library *and*
> the executable with tsan, or do I only need to build the executable with
> tsan?
>
> If it's only the exe, then fine, I know what to do.
>
> However, if I need to build the library with tsan, well, the library can be
> static or dynamic.
>
> If I'm building dynamic, then I must use -fPIC. Do I then use PIE or PIC
> with the executable? I think I must use PIC...

For executable you use -fPIE (which stands for Position-Independent
*Executable*).

> If I'm building an archive, I think I can then use PIE, since it'll be

For static archives it depends on what it will be linked to. If it is
linked only to executable, then you can use -fPIE; if it is linked to
shared libraries (and optionally to executables), then you need to use
-fPIC.

> linked directly into the executable, which would then mean I can as
> specified in the instructions use PIE to build the executable... however,
> I've read -static should not be used with -pie, which confuses the hell out
> of me. Is it actually meant that archives *typically* are not built -pie,
> so don't use -pie with -static?

Why do you want to use -static? It's not for creation of static
archives. And it's not supported with tsan, don't use it.

[Toebs Douglass]

Dimitry Vyukov wrote:
> On Sun, Oct 12, 2014 at 2:25 AM, Toebs Douglass <to...@winterflaw.net> wrote:
>> If I'm building dynamic, then I must use -fPIC. Do I then use PIE or PIC
>> with the executable? I think I must use PIC...
>
> For executable you use -fPIE (which stands for Position-Independent
> *Executable*).

As I understand it, PIC and PIE do the same thing but in a different
way, and PIE is more optimal than PIC but can only be used with executables.

My concern is that if I've built some of the code with PIC (the
libraries) and then the executable with PIE, *will it all still work?*
i.e. I've now built a final executable which in part is using one method
(PIE) and in part another (PIC). Will this not cause problems?

>> If I'm building an archive, I think I can then use PIE, since it'll be
>
> For static archives it depends on what it will be linked to. If it is
> linked only to executable, then you can use -fPIE; if it is linked to
> shared libraries (and optionally to executables), then you need to use
> -fPIC.

Yes. I meant only to the executable, and I understand what you say in
general. This of course harks back to my earlier point - it looks like
you cannot mix PIE and PIC.

>> linked directly into the executable, which would then mean I can as
>> specified in the instructions use PIE to build the executable... however,
>> I've read -static should not be used with -pie, which confuses the hell out
>> of me. Is it actually meant that archives *typically* are not built -pie,
>> so don't use -pie with -static?
>
> Why do you want to use -static? It's not for creation of static
> archives. And it's not supported with tsan, don't use it.

I don't want to and I'm not using -static.

The reason I mentioned it was because of the *implications* of -static
and -pie being mutually exclusive. If that is true, then it seems to
imply I cannot make static libraries with -pie (which in fact looks to
be possible - so as I say, it confuses me).

I'm afraid I still don't know how to build tsan with a shared library.

I have to build the shared library with PIC - that's mandatory for
shared libraries. Do I build with the shared library with
-fthread=sanitize? if so, do I then build the executable with PIE, or
with PIC?

[Dmitry Vyukov]

On Sun, Oct 12, 2014 at 12:49 PM, Toebs Douglass <to...@winterflaw.net> wrote:
> Dimitry Vyukov wrote:
>>
>> On Sun, Oct 12, 2014 at 2:25 AM, Toebs Douglass <to...@winterflaw.net>
>> wrote:
>>>
>>> If I'm building dynamic, then I must use -fPIC. Do I then use PIE or PIC
>>> with the executable? I think I must use PIC...
>>
>>
>> For executable you use -fPIE (which stands for Position-Independent
>> *Executable*).
>
>
> As I understand it, PIC and PIE do the same thing but in a different way,
> and PIE is more optimal than PIC but can only be used with executables.
>
> My concern is that if I've built some of the code with PIC (the libraries)
> and then the executable with PIE, *will it all still work?* i.e. I've now
> built a final executable which in part is using one method (PIE) and in part
> another (PIC). Will this not cause problems?

Mixing PIC and PIE is OK.

>>> If I'm building an archive, I think I can then use PIE, since it'll be
>>
>>
>> For static archives it depends on what it will be linked to. If it is
>> linked only to executable, then you can use -fPIE; if it is linked to
>> shared libraries (and optionally to executables), then you need to use
>> -fPIC.
>
>
> Yes. I meant only to the executable, and I understand what you say in
> general. This of course harks back to my earlier point - it looks like you
> cannot mix PIE and PIC.
>
>>> linked directly into the executable, which would then mean I can as
>>> specified in the instructions use PIE to build the executable... however,
>>> I've read -static should not be used with -pie, which confuses the hell
>>> out
>>> of me. Is it actually meant that archives *typically* are not built
>>> -pie,
>>> so don't use -pie with -static?
>>
>>
>> Why do you want to use -static? It's not for creation of static
>> archives. And it's not supported with tsan, don't use it.
>
>
> I don't want to and I'm not using -static.
>
> The reason I mentioned it was because of the *implications* of -static and
> -pie being mutually exclusive. If that is true, then it seems to imply I
> cannot make static libraries with -pie (which in fact looks to be possible -
> so as I say, it confuses me).

If you are not using -static, let's forget about it.

> I'm afraid I still don't know how to build tsan with a shared library.
>
> I have to build the shared library with PIC - that's mandatory for shared
> libraries. Do I build with the shared library with -fthread=sanitize? if
> so, do I then build the executable with PIE, or with PIC?

You build shared libraries with -fthread=sanitize -fPIC
You build executables with -fthread=sanitize -fPIE

[Toebs Douglass]

Dimitry Vyukov wrote:
>> My concern is that if I've built some of the code with PIC (the libraries)
>> and then the executable with PIE, *will it all still work?* i.e. I've now
>> built a final executable which in part is using one method (PIE) and in part
>> another (PIC). Will this not cause problems?
>
> Mixing PIC and PIE is OK.

Excellent. That makes things much clearer.

>> I have to build the shared library with PIC - that's mandatory for shared
>> libraries. Do I build with the shared library with -fthread=sanitize? if
>> so, do I then build the executable with PIE, or with PIC?
>
> You build shared libraries with -fthread=sanitize -fPIC
> You build executables with -fthread=sanitize -fPIE

Good. Thankyou!

One final question, regarding TLS.

I understand TSAN heavily uses TLS, and -shared -fPIC TLS has very, very
poor performance.

On that basis, what I really want to do is compile to a static library -
and, ideally, with PIE rather than PIC, since PIE better than PIC for TLS.

I'll be linking the static library only to the test executable, so I
think this should be fine.

Does this all sound okay to you?

[Evgeniy Stepanov]

AFAIK, all TSan TLS variables are declared in the TSan runtime
library, which is statically linked to the main executable, and use
initial_exec TLS model. In this case TLS access is fast even from
shared libraries.

>
> --
> You received this message because you are subscribed to the Google Groups
> "thread-sanitizer" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to thread-sanitiz...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Dmitry Vyukov]

On Sun, Oct 12, 2014 at 1:03 PM, Toebs Douglass <to...@winterflaw.net> wrote:

Thread-local accesses happen within tsan runtime, not within
instrumented code. So it's irrelevant how you compile your code. In
tsan runtime we use __attribute__((tls_model("initial-exec"))) to make
the TLS accesses reasonably fast even in PIC code.
However, all calls into tsan runtime from shared libraries are slower
(because they go through PLT). If you use gcc, there is nothing you
can do with it because tsan runtime itself is a shared library. If you
use clang, tsan runtime is a static library; so for performance
reasons it makes sense to build your code as static libraries as well.
But I've never measured what performance impact it has. Maybe it's just 1%.

[Toebs Douglass]

Dimitry Vyukov wrote:
> Thread-local accesses happen within tsan runtime, not within
> instrumented code. So it's irrelevant how you compile your code.

Ah. It wasn't clear to me whether or not there was link-time work which
affected TLS.

> In
> tsan runtime we use __attribute__((tls_model("initial-exec"))) to make
> the TLS accesses reasonably fast even in PIC code.

Yes.

> However, all calls into tsan runtime from shared libraries are slower
> (because they go through PLT). If you use gcc, there is nothing you
> can do with it because tsan runtime itself is a shared library. If you
> use clang, tsan runtime is a static library; so for performance
> reasons it makes sense to build your code as static libraries as well.
> But I've never measured what performance impact it has. Maybe it's just 1%.

I've done some reading on this, but it's not something I know much
about. I've seen 10%/20% banded about, but much more for -shared -fPIC
(not sure if this is with or without initial-exec).

I've rolled my own TLS (which is to say, it's an argument which you pass
round) because I couldn't trust TLS to be performant or available on all
platforms (I target the freestanding implementation, and I'm actually
only using C89 because of Windows - that may change, I'm fed up with MS
not supporting C and I'll be moving to Linux with my next laptop); I
only use it for per-thread PRNG state for exponential back-off on CAS.

I'd use per-CPU if I could, but as you know, it's not straightforward.