Dylib on Mac OSX with ASan

[Christian Holler]

Hey all,

we recently ran into the problem that on Mac, when building a dylib, the
default behavior for undefined symbols is to error. As far as I
understood on Linux, it is expected that shared libraries have undefined
symbols because the runtime is only linked to the executable. I assume
this is the same for dylibs? If so, what is the proper switch to allow
so? A colleague of mine found -undefined dynamic_lookup but ran into
other problems then, so I wanted to make sure that this is the right way.

Thanks in advance,

Chris

[Alexander Potapenko]

Hi Christian,

are you using "clang -faddress-sanitizer" to link the dynamic libraries
(note that this is the only correct way to link with ASan, manually
adding the runtime library to the ld command line is too tricky to
manage)?
It should automatically add "-undefined dynamic_lookup" to the list of
ld options.

Yes, the runtime can only be linked to the executable, so if you are
about to load your instrumented library into a non-instrumented binary
(that obviously doesn't have the runtime), you'll run into trouble.
Otherwise everything should be ok.

--
Alexander Potapenko
Software Engineer
Google Moscow

[Christian Holler]

Hey Alex,

thanks, that was indeed a problem. It seems that NSS did not pass
-faddress-sanitizer for LDFLAGS when linking dylibs. I fixed that,
however, I get this now:

/Users/choller/llvm/build/Release+Asserts/bin/clang -faddress-sanitizer
-framework Cocoa -lobjc -framework ExceptionHandling
-Wl,-executable_path,/Users/choller/mozilla-central/objdir-ff-asan/dist/bin
-faddress-sanitizer -v -o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/libnssutil3.dylib
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/quickder.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/secdig.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/derdec.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/derenc.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/dersubr.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/dertime.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/errstrs.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/nssb64d.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/nssb64e.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/nssrwlk.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/nssilock.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/oidstring.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/portreg.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/secalgid.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/secasn1d.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/secasn1e.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/secasn1u.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/secitem.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/secload.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/secoid.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/sectime.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/secport.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/templates.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/utf8.o
-L/Users/choller/mozilla-central/objdir-ff-asan/security/manager/../../dist/lib
-L/Users/choller/mozilla-central/objdir-ff-asan/dist/lib -lplc4 -lplds4
-lnspr4
clang version 3.1 (trunk 152341)
Target: x86_64-apple-darwin11.3.0
Thread model: posix
"/usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld" -dynamic
-arch x86_64 -macosx_version_min 10.6.0 -o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/libnssutil3.dylib
-lcrt1.10.6.o
-L/Users/choller/mozilla-central/objdir-ff-asan/security/manager/../../dist/lib
-L/Users/choller/mozilla-central/objdir-ff-asan/dist/lib -framework
Cocoa -lobjc -framework ExceptionHandling -executable_path
/Users/choller/mozilla-central/objdir-ff-asan/dist/bin
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/quickder.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/secdig.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/derdec.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/derenc.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/dersubr.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/dertime.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/errstrs.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/nssb64d.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/nssb64e.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/nssrwlk.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/nssilock.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/oidstring.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/portreg.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/secalgid.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/secasn1d.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/secasn1e.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/secasn1u.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/secitem.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/secload.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/secoid.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/sectime.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/secport.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/templates.o
/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/utf8.o -lplc4
-lplds4 -lnspr4
/Users/choller/llvm/build/Release+Asserts/bin/../lib/clang/3.1/lib/darwin/libclang_rt.asan_osx.a
-lstdc++ -framework CoreFoundation -lSystem
/Users/choller/llvm/build/Release+Asserts/bin/../lib/clang/3.1/lib/darwin/libclang_rt.osx.a
Undefined symbols for architecture x86_64:
"_main", referenced from:
start in crt1.10.6.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see
invocation)
make[7]: ***
[/Users/choller/mozilla-central/objdir-ff-asan/nss/nssutil/libnssutil3.dylib]
Error 1

The first one seems to be the compiler call for creating the dylib (-o
[...]/libnssutil3.dylib), the second the associated ld call. What I'm
wondering is why the asan RT is passed in that ld call. It shouldn't for
a dylib, should it?

Cheers,

Chris

[Alexander Potapenko]

Maybe because there's no -dynamiclib or -bundle flag?
I think Clang is trying to build an executable for you.

--

[Christian Holler]

You're right, the flags were messed up, it's all fixed now (NSS
buildsystem integration isn't straight forward sometimes..)

I get a startup crash now:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000000
[Switching to process 78567 thread 0x2803]
0x0000000000000000 in ?? ()
(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x00007fff8995810a in _dispatch_queue_drain ()
#2 0x00007fff89957f66 in _dispatch_queue_invoke ()
#3 0x00007fff89957760 in _dispatch_worker_thread2 ()
#4 0x00007fff82ab13da in _pthread_wqthread ()
#5 0x00007fff82ab2b85 in start_wqthread ()

Ever seen this?

Chris

[Alexander Potapenko]

On Tue, Apr 3, 2012 at 11:46 PM, Christian Holler <cho...@mozilla.com> wrote:
> You're right, the flags were messed up, it's all fixed now (NSS
> buildsystem integration isn't straight forward sometimes..)
>
> I get a startup crash now:
>
> Program received signal EXC_BAD_ACCESS, Could not access memory.
> Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000000
> [Switching to process 78567 thread 0x2803]
> 0x0000000000000000 in ?? ()
> (gdb) bt
> #0  0x0000000000000000 in ?? ()
> #1  0x00007fff8995810a in _dispatch_queue_drain ()
> #2  0x00007fff89957f66 in _dispatch_queue_invoke ()
> #3  0x00007fff89957760 in _dispatch_worker_thread2 ()
> #4  0x00007fff82ab13da in _pthread_wqthread ()
> #5  0x00007fff82ab2b85 in start_wqthread ()
>
>
> Ever seen this?
>
> Chris

This looks like an attempt to execute a NULL callback. Maybe something
is linked wrong?
Can you please attach the output of nm for your binary and the dynamic
library (you can filter it for __asan, if there's something
confidential)?
For example, the build system may strip the __asan_init symbol from
the binary, then the dylib won't be able to call it.

Can I also take a look at the output with ASAN_OPTIONS=verbosity=2?

Finally, which OS X and Clang versions are you using?

[Christian Holler]

Hey Alex,

thanks for your analysis. Attached is the nm output (I grepped for
__asan, not because there is something confidential but because Firefox
has lots of symbols, if you need the full output, let me know) of both
the binary and the nssutil dylib (although I don't know if the NSS dylib
is involved here at all). I also attached the verbosity log but
compressed it because it had 450 kb.

LLVM/Clang version is trunk rev 152341. uname -a says the kernel is
Darwin 11.3.0 so I assume it's Mac OS X 10.7.3.

If you need anything else I'd be glad to provide it so we can solve this
issue :)

Cheers,

Chris

[Alexander Potapenko]

This is most certainly
http://code.google.com/p/address-sanitizer/issues/detail?id=53
Can you please update Clang to at least r153249 and check that?

[Christian Holler]

That was it :) works now (on exactly that rev), thanks a lot! :)

Chris