shared libaries, tsan, pic and pie

[Toebs Douglass]

Hi!

I have a library of lock-free data structures (in C and I'm using gcc).
The library internally can have race conditions in its own functions.

I have a test executable, which links to this library. The test
executable itself also needs tsan, as it does some abstraction layer
testing, which can have race conditions.

The build instructions are;

"Simply compile your program with -fsanitize=thread -fPIE and link it
with -fsanitize=thread -pie."

What is the correct way here to use tsan?

I'm thinking I need to build the lock-free library as shared library
(-fPIC, where PIC of course has to happen anyway as we're a .so) (as I
have read archives can't use position independent code, although it's
not apparent to me why not) and with tsan, and then also build the text
executable with tsan (-fPIE) (I've been told I cannot build executables
with -fPIC).

However, when I try to build the lock-free library as a shared library
with tsan, it fails;

ld -shared -soname=liblfds700.so.1 -fsanitize=thread -pic
../../obj/lfds700_ahash_cleanup.o ../../obj/lfds700_ahash_get.o
../../obj/lfds700_ahash_init.o ../../obj/lfds700_ahash_iterate.o
../../obj/lfds700_ahash_put.o ../../obj/lfds700_ahash_query.o
../../obj/lfds700_aslist_cleanup.o
../../obj/lfds700_aslist_get_and_set.o ../../obj/lfds700_aslist_init.o
../../obj/lfds700_aslist_link.o ../../obj/lfds700_aslist_query.o
../../obj/lfds700_aubtree_cleanup.o ../../obj/lfds700_aubtree_find.o
../../obj/lfds700_aubtree_get.o ../../obj/lfds700_aubtree_init.o
../../obj/lfds700_aubtree_link.o ../../obj/lfds700_aubtree_query.o
../../obj/lfds700_counter_cleanup.o ../../obj/lfds700_counter_init.o
../../obj/lfds700_counter_read.o ../../obj/lfds700_counter_register.o
../../obj/lfds700_freelist_cleanup.o
../../obj/lfds700_freelist_gc_reuse.o ../../obj/lfds700_freelist_init.o
../../obj/lfds700_freelist_pop.o ../../obj/lfds700_freelist_push.o
../../obj/lfds700_freelist_query.o ../../obj/lfds700_liblfds_globals.o
../../obj/lfds700_liblfds_cleanup.o ../../obj/lfds700_liblfds_init.o
../../obj/lfds700_liblfds_pal_helpers.o
../../obj/lfds700_liblfds_query.o ../../obj/lfds700_queue_mpmc_cleanup.o
../../obj/lfds700_queue_mpmc_dequeue.o
../../obj/lfds700_queue_mpmc_enqueue.o
../../obj/lfds700_queue_mpmc_gc_reuse.o
../../obj/lfds700_queue_mpmc_init.o ../../obj/lfds700_queue_mpmc_query.o
../../obj/lfds700_queue_spsc_cleanup.o
../../obj/lfds700_queue_spsc_dequeue.o
../../obj/lfds700_queue_spsc_enqueue.o
../../obj/lfds700_queue_spsc_init.o ../../obj/lfds700_queue_spsc_query.o
../../obj/lfds700_ringbuffer_cleanup.o
../../obj/lfds700_ringbuffer_get_and_put.o
../../obj/lfds700_ringbuffer_init.o ../../obj/lfds700_ringbuffer_query.o
../../obj/lfds700_gc_cleanup.o ../../obj/lfds700_gc_init.o
../../obj/lfds700_gc_query.o ../../obj/lfds700_gc_register.o
../../obj/lfds700_gc_reuse.o ../../obj/lfds700_gc_scan.o
../../obj/lfds700_stlist_cleanup.o ../../obj/lfds700_stlist_get.o
../../obj/lfds700_stlist_init.o ../../obj/lfds700_stlist_link.o
../../obj/lfds700_stlist_query.o ../../obj/lfds700_stlist_unlink.o
../../obj/lfds700_stack_cleanup.o ../../obj/lfds700_stack_gc_reuse.o
../../obj/lfds700_stack_init.o ../../obj/lfds700_stack_pop.o
../../obj/lfds700_stack_push.o ../../obj/lfds700_stack_query.o -o
../../bin/liblfds700.so.1.0.0

ld: warning: cannot find entry symbol _start; defaulting to 0000000000000db0
../../obj/lfds700_ahash_cleanup.o: In function `lfds700_ahash_cleanup':
/tmp/temp/liblfds/liblfds7.0.0/liblfds700/build/linux_usermode_gcc_and_gnumake/../../src/lfds700_addonly_hash/lfds700_ahash_cleanup.c:11:
undefined reference to `__tsan_func_entry'
/tmp/temp/liblfds/liblfds7.0.0/liblfds700/build/linux_usermode_gcc_and_gnumake/../../src/lfds700_addonly_hash/lfds700_ahash_cleanup.c:15:
undefined reference to `__assert_fail'

[snip many tens of thousands of lines of output)

The first error, cannot find _start, I understand prolly to originate
from me using ld directly and not passing in all the necessary args -
although having said that, the code compiles to a .so just fine if tsan
is not in use.

I think that -fsanitize=thread is behind the scenes causing GCC to link
to libstan, but that this here is not happening.

[Dmitry Vyukov]

Hi,

-fsanitize=thread is a driver flag, not linker.
Why do you link with ld? This looks wrong. You need to use 'gcc
-shared' to link shared libraries.


p.s. archives can contain PIC code. In fact tsan runtime clang is a
static library (.a) built with -fPIC.
p.s. I believe executables can be built with both -fPIE and -fPIC.
-fPIC will lead to slightly slower code, but both should work.

[Toebs Douglass]

Dmitry Vyukov wrote:

Apologies for what may be a second thread - I can only reply in-thread
if I log into Google Groups; I emailed the original question, but
received no email when you replied. I've used the same subject, in the
hope it will be appended to the thread.

> -fsanitize=thread is a driver flag, not linker.

Okay. I've just googled about driver flags; I'd never heard of the
compiler driver before. I presume it is a flag which directs a range of
related behaviours.

> Why do you link with ld? This looks wrong. You need to use 'gcc
> -shared' to link shared libraries.

I just always have... ld does linking, you link to produce shared
libraries. It works when not using tsan, so it's been okay for some years.

> p.s. archives can contain PIC code. In fact tsan runtime clang is a
> static library (.a) built with -fPIC.

Right. It made no sense to me that it would not be possible.

> p.s. I believe executables can be built with both -fPIE and -fPIC.
> -fPIC will lead to slightly slower code, but both should work.

Yes. This I thought also, from what I'd read about TLS. I begin to
suspect there's a good deal of incorrect information out on the net, in
this matter.

Thankyou for you reply.

However, I'm still not clear about what to do :-)

I've just found a little bit more information, from the clang docs for
AddressSanitizer. They say;

"Simply compile and link your program with -fsanitize=address flag. The
AddressSanitizer run-time library should be linked to the final
executable, so make sure to use clang (not ld) for the final link step.
When linking shared libraries, the AddressSanitizer run-time is not
linked, so -Wl,-z,defs may cause link errors (don’t use it with
AddressSanitizer)."

This indicates I do *nothing* different with my library; I only need to
build the final executable with the necessary thread sanitizer flags.

[Toebs Douglass]

Dmitry Vyukov wrote:
> -fsanitize=thread is a driver flag, not linker.

To explain my usage, the build instructions are thus;

https://code.google.com/p/thread-sanitizer/wiki/CppManual

"Simply compile your program with -fsanitize=thread -fPIE and link it
with -fsanitize=thread -pie."

Now imagine you have a makefile with GCC for compiling and LD for
linking. Those instructions, if you don't know better, come to this;

1. add "-fsanitize=thread -fPIE" to your GCC call (compiling)
2. add "-fsanitize=thread -pie" to your LD call (linking)

You see?

I thought I recognized your name, BTW - thanks for 1024cores, there's
some nice stuff on there.

[Evgeniy Stepanov]

With ld you need to pass all the crt*.o stuff, standard libs, linker
pass etc manually. Does you Makefile take care of this, and why?

> --
> You received this message because you are subscribed to the Google Groups
> "thread-sanitizer" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to thread-sanitiz...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Toebs Douglass]

Evgeniy Stepanov wrote:
> With ld you need to pass all the crt*.o stuff, standard libs, linker
> pass etc manually. Does you Makefile take care of this, and why?

This is a library makefile - it builds a shared library or a static
library. It does not build an executable. As far as I know, this means
I do not need any of the extra stuff (crt*.o, etc)?

Having said that... the makefile for the executable *also* uses ld, and
it doesn't specify the crt stuff. It does work, or at least, appear to
work (in the sense that I can run the binary). I do specify -lc, and I
do have a rule for the link phase.

As to why - I would ask you why would you want to operate ld through
gcc? ld is a separate tool, with its own command line args and separate
task. I feel it more right to explicitly show and control its operation.

I can see a reason for using ld through gcc, which is that gcc will
handle some linking details for you, which otherwise you must do
manually. You only have to write that once, though.

[Evgeniy Stepanov]

Right, the driver takes care of lots of little details, like, for
example, you'd need different linker flags if you build an binary with
any sanitizer with clang versus gcc (different runtime library name).
What's worse, runtime libraries (static ones) are located at a
non-standard path in the compiler toolchain, which linker normally
does not know anything about. Driver knows that -fsanitize=thread
implies -pie. Driver knows the correct set of crt objects (and the
ld.so path) for your libc. I could go on :)

[Dmitry Vyukov]

On Fri, Oct 10, 2014 at 5:48 PM, Toebs Douglass <to...@winterflaw.net> wrote:
> Dmitry Vyukov wrote:
>>
>> -fsanitize=thread is a driver flag, not linker.
>
>
> To explain my usage, the build instructions are thus;
>
> https://code.google.com/p/thread-sanitizer/wiki/CppManual
>
> "Simply compile your program with -fsanitize=thread -fPIE and link it with
> -fsanitize=thread -pie."
>
> Now imagine you have a makefile with GCC for compiling and LD for linking.
> Those instructions, if you don't know better, come to this;
>
> 1. add "-fsanitize=thread -fPIE" to your GCC call (compiling)
> 2. add "-fsanitize=thread -pie" to your LD call (linking)
>
> You see?

Linking with ld is a very weird, not promoted way to build programs.
In the gcc compiler suite C compiler is cc1, and for linking gnu ld
linker is usually used. But users should use compiler driver - gcc -
for both tasks. gcc invokes either the compiler (cc1) or the linker
(ld) as necessary.

You should use something along the lines of:
$ gcc source.c -c -fPIC -fsanitize=thread
for compilation
and:
$ gcc -shared -o lib.so -fsanitize=thread source.o
for linking.

Then driver will take care of linking in all required libraries (like
the one that provides _start symbol and the one that provides
__tsan_func_entry symbol).

> I thought I recognized your name, BTW - thanks for 1024cores, there's some
> nice stuff on there.

Thanks
you are welcome

[Dmitry Vyukov]

On Fri, Oct 10, 2014 at 6:23 PM, Toebs Douglass <to...@winterflaw.net> wrote:
> Evgeniy Stepanov wrote:
>> With ld you need to pass all the crt*.o stuff, standard libs, linker
>> pass etc manually. Does you Makefile take care of this, and why?
>
> This is a library makefile - it builds a shared library or a static library.
> It does not build an executable. As far as I know, this means I do not need
> any of the extra stuff (crt*.o, etc)?
>
> Having said that... the makefile for the executable *also* uses ld, and it
> doesn't specify the crt stuff. It does work, or at least, appear to work
> (in the sense that I can run the binary). I do specify -lc, and I do have a
> rule for the link phase.
>
> As to why - I would ask you why would you want to operate ld through gcc?

The same reason you don't use *the compiler* -- cc1 for compilation.
It's much harder.

> ld is a separate tool, with its own command line args and separate task. I
> feel it more right to explicitly show and control its operation.
>
> I can see a reason for using ld through gcc, which is that gcc will handle
> some linking details for you, which otherwise you must do manually. You
> only have to write that once, though.
>

[Toebs Douglass]

Evgeniy Stepanov wrote:
> Right, the driver takes care of lots of little details, like, for
> example, you'd need different linker flags if you build an binary with
> any sanitizer with clang versus gcc (different runtime library name).
> What's worse, runtime libraries (static ones) are located at a
> non-standard path in the compiler toolchain, which linker normally
> does not know anything about. Driver knows that -fsanitize=thread
> implies -pie. Driver knows the correct set of crt objects (and the
> ld.so path) for your libc. I could go on :)

Right - I think this is what I've discovered.

You *can't* build using tsan and ld (perhaps you can in principle, but
I've after considerable - a day or two now - of effort not found the
necessary documentation or any hint of the issues involved, so I can't,
not in a reasonable fashion). You *have* to use gcc to link.

My makefiles, which compile and then link, are broken by tsan. I may be
wrong, but I don't think that's reasonable. It *should* be possible to
compile with gcc and then link with ld.

[Dmitry Vyukov]

On Fri, Oct 10, 2014 at 8:43 PM, Toebs Douglass <to...@winterflaw.net> wrote:
> Evgeniy Stepanov wrote:
>>
>> Right, the driver takes care of lots of little details, like, for
>> example, you'd need different linker flags if you build an binary with
>> any sanitizer with clang versus gcc (different runtime library name).
>> What's worse, runtime libraries (static ones) are located at a
>> non-standard path in the compiler toolchain, which linker normally
>> does not know anything about. Driver knows that -fsanitize=thread
>> implies -pie. Driver knows the correct set of crt objects (and the
>> ld.so path) for your libc. I could go on :)
>
>
> Right - I think this is what I've discovered.
>
> You *can't* build using tsan and ld (perhaps you can in principle, but I've
> after considerable - a day or two now - of effort not found the necessary
> documentation or any hint of the issues involved, so I can't, not in a
> reasonable fashion). You *have* to use gcc to link.

Well, you don't have to use gcc. And there is nothing in particular to document.
Link once with gcc and with -v flag. That will give you full command
line with ld invocation. Copy the flags from there.

> My makefiles, which compile and then link, are broken by tsan. I may be
> wrong, but I don't think that's reasonable. It *should* be possible to
> compile with gcc and then link with ld.
>
>

[Yuri Gribov]

On Fri, Oct 10, 2014 at 8:43 PM, Toebs Douglass <to...@winterflaw.net> wrote:

> My makefiles, which compile and then link, are broken by tsan. I may be
> wrong, but I don't think that's reasonable. It *should* be possible to
> compile with gcc and then link with ld.

In my experience using plain ld is asking for trouble precisely
because innocently looking compiler options may result in vast changes
in LDFLAGS.

-Y

[toebs.d...@gmail.com]

Hi all.

Google are silently blocking my emails to this thread - certainly from the email address I originally was using and seemingly also from a gmail account!  as such for the last two hours my most recent post has not been accepted and so everyone else has been replying in error (and wasting their time).  I've spent the last two hours trying in various ways to get this post accepted.

I've now logged in to post.

I made my most recent email and then immediately after, having read Dmitry's explanation, another, this;

Dmitry Vyukov wrote:

Linking with ld is a very weird, not promoted way to build programs.
In the gcc compiler suite C compiler is cc1, and for linking gnu ld
linker is usually used. But users should use compiler driver - gcc -
for both tasks. gcc invokes either the compiler (cc1) or the linker
(ld) as necessary.

I see.  I've always thought of gcc as the C compiler.

You should use something along the lines of:
$ gcc source.c -c -fPIC -fsanitize=thread
for compilation
and:
$ gcc -shared -o lib.so -fsanitize=thread source.o
for linking.

That's about what it is, except the second command uses ld rather than gcc.

Then driver will take care of linking in all required libraries (like
the one that provides _start symbol and the one that provides
__tsan_func_entry symbol).

Yes.

Thankyou for explaining about gcc/cc1/ld.

[toebs.d...@gmail.com]

And I've just noticed I'm going to be emailed the replies to my post, which is on by default but *not* what I wanted.

I have to say, this has not been a pleasent experience.  The tragedy of the decline of Usenet and the rise of the nannying web.

[toebs.d...@gmail.com]

Test.

[Secret Private]

Dmitry Vyukov wrote:

This reply was originally made moments after my most recent post.

However, I think where I made two quick posts, google groups is I believe now silently blocking me.  I have waited now about two hours and my posts have not turned up, so I'm now posting from this gmail account.

[Toebs Douglass]

[Toebs Douglass]

[Dmitry Vyukov]

Hi Toebs,

Google is not silencing your emails.
We have turned on pre-moderation in the group, which is absolutely
necessary for a public group, because there are huge amounts of spam
coming.

Part of the problem was that you were using different email addresses.
Also some of your emails were classified as spam. I don't know what is
the reason for that. It may be an issue in classification algorithm or
it could be caused by seemingly duplicate emails or something else.

All your emails must appear in the group now. And also your emails now
have permission to post w/o per-moderation.

Sorry once again.