AddressSanitizer enabled shared object dlopen'd in non-AddressSanitizer executable

[mjoc...@gmail.com]

Hi,

Is there anyway to build a shared object with asan support and load this in a non-asan executable?

The third party executable has support for loading shared objects to provide extra functionality but it is not built with asan; this results in an undefined symbol error.

A similar thread previously addressed this point with regard to python and the conclusion was to rebuild python with asan. Unfortunately this is not an option; is there any work around?

Thanks,

Mark

[Konstantin Serebryany]

On Wed, Feb 5, 2014 at 4:06 PM, <mjoc...@gmail.com> wrote:

Hi,

Is there anyway to build a shared object with asan support and load this in a non-asan executable?

Short answer -- no. 

 

The third party executable has support for loading shared objects to provide extra functionality but it is not built with asan; this results in an undefined symbol error.

A similar thread previously addressed this point with regard to python and the conclusion was to rebuild python with asan. Unfortunately this is not an option; is there any work around?

Correct. The best option with python is to rebuild python with asan. 

For your case, you may try to build asan run time as a shared library and LD_PRELOAD it into your process. 

But this is really tricky and I don't recommend this unless you are desperate. 

--kcc 

 

Thanks,

Mark

--
You received this message because you are subscribed to the Google Groups "address-sanitizer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to address-saniti...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Yuri Gribov]

>> Is there anyway to build a shared object with asan support and load this
>> in a non-asan executable?

Which platform are we talking about?

> For your case, you may try to build asan run time as a shared library and
> LD_PRELOAD it into your process.
> But this is really tricky and I don't recommend this unless you are
> desperate.

FYI this works out-of-the-box with GCC ASan (which allows dynamic ASan runtime).

-Y

[mjoc]

Thanks Kcc and Yuri,

Running on Linux.

I have been using clang, I will try gcc and LD_PRELOAD.

Thanks,

mjoc

[Konstantin Serebryany]

You can build asan.so from LLVM tree like this:

cd lvm/projects/compiler-rt/lib

clang -o asan.so  -DASAN_USE_PREINIT_ARRAY=0 -fno-exceptions -fno-rtti -I. -O2 -fPIC -shared asan/*.cc sanitizer_common/*.cc interception/*.cc lsan/lsan_common.cc lsan/lsan_common_linux.cc 

LD_PRELOAD may appear to work for simple cases, but the devil is in the detail.

What if your process creates other processes? They will also have asan.so preloaded. 

Sometimes it's ok, sometimes not.

We may be able to support building asan as a DSO in clang tree as well,

but I don't want to do this unless we know that someone really needs it and this is the right way to solve some problem.

--kcc 

--

[Konstantin Serebryany]

On Thu, Feb 6, 2014 at 1:44 PM, Konstantin Serebryany <konstantin....@gmail.com> wrote:

You can build asan.so from LLVM tree like this:

cd lvm/projects/compiler-rt/lib

clang -o asan.so  -DASAN_USE_PREINIT_ARRAY=0 -fno-exceptions -fno-rtti -I. -O2 -fPIC -shared asan/*.cc sanitizer_common/*.cc interception/*.cc lsan/lsan_common.cc lsan/lsan_common_linux.cc 

also add "-ldl -lpthread -lm"  to this command line

[Yuri Gribov]

> but I don't want to do this unless we know that someone really needs it
> and this is the right way to solve some problem.

Some benefits of dynamic runtime:
1) reduced memory overhead (important for embedded world)
2) it's also rather handy when you have a bunch of apps and libs some
of which are sanitized and some aren't (in real world you don't always
have the luxury of full app rebuild); LD_PRELOAD would allow you to
override malloc and friends and enable ASan even in that case

-Y

[Konstantin Serebryany]

There is no disagreement here.

Note that the only way ASAN is used on Android, Mac and iOS (and Windows?) is through dynamic run-time.

But: we are using dynamic run-time for python in one of our settings

and the experience is so painful that we don't want to advertise asan-as-dso on Linux,

where static run-time works well for the majority of use cases.

There are some bugs specific to using asan as a shared library, e.g. 

https://code.google.com/p/address-sanitizer/issues/detail?id=147

--kcc 

[mjoc]

Thanks guys,

gcc48 worked a charm.

For our use case, the DSO is the only option.

A third party application which cannot be recompiled with the required flags, internal teams developing DSOs and trying to get to the bottom of some stack smashing.

Thanks,

mjoc

[Konstantin Serebryany]

I created https://code.google.com/p/address-sanitizer/wiki/AsanAsDso to keep the info in one place.

Feedback and additions are welcome

[Yuri Gribov]

Cool! My comments below.

> Smaller disk usage

How about "Smaller memory footprint"?

> __asan_init is not called from preinit_array and so there is a risk that an instrumented code will get called before __asan_init

1) Should probably decribe symptoms here (call to NULL).
2) Can we add a check to REAL() to produce a nice diagnostic message
in this case?
3) We once discussed making __asan_init a constructor to solve this
problem (discussion and patch available here:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58937).

> the binary is linked against incompatible DSO version

Nice one. I wonder whether we can catch this in preloaded libasan
(e.g. search for all older __asan_init_vXXX using dlsym at start).

>> There are some bugs specific to using asan as a shared library, e.g.

Can you add other bugs as well?

Some proposal from me:
* could you add a note about spurious "failed to intercept" warnings?
(see http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58680 for reference)

BTW should we acronimize as ASAN or ASan?

-Y

[Konstantin Serebryany]

On Thu, Feb 6, 2014 at 8:43 PM, Yuri Gribov <tetr...@gmail.com> wrote:

Cool! My comments below.

> Smaller disk usage

How about "Smaller memory footprint"?

Done.  

> __asan_init is not called from preinit_array and so there is a risk that an instrumented code will get called before __asan_init

1) Should probably decribe symptoms here (call to NULL).

We tried to eliminate all such failures and we don't know of any existing one.

Added "(may cause SEGV at startup; still unlikely)" 

2) Can we add a check to REAL() to produce a nice diagnostic message
in this case?

There are a few such checks.

 

3) We once discussed making __asan_init a constructor to solve this
problem (discussion and patch available here:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58937).

__asan_init is a constructor, but there could be other constructors that run before it. 

 

> the binary is linked against incompatible DSO version

Nice one. I wonder whether we can catch this in preloaded libasan
(e.g. search for all older __asan_init_vXXX using dlsym at start).

This should be detected at load time  precisely due to __asan_init_vXXX mismatch. 

 

>> There are some bugs specific to using asan as a shared library, e.g.

Can you add other bugs as well?

When I find/remember them.

I afraid we did not document all of the cases like this yet because we have very little experience with dynamic run-time on linux. 

 

Some proposal from me:
* could you add a note about spurious "failed to intercept" warnings?
(see http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58680 for reference)

Hm? What about that? How would you phrase it? 

 

BTW should we acronimize as ASAN or ASan?

Emm. I equally like all of {asan,ASAN,ASan} and I don't care much. 

[Yuri Gribov]

>> 2) Can we add a check to REAL() to produce a nice diagnostic message
>> in this case?
>
> There are a few such checks.

I don't think that REAL() currently checks anything:

~/llvm/trunk/projects/compiler-rt/lib$ grep NULL interception/* -l
interception/interception_win.cc

We seem to rely on callers to make sure that these variables are
properly initialized.

>> 3) We once discussed making __asan_init a constructor to solve this
>> problem (discussion and patch available here:
>> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58937).
>
>
> __asan_init is a constructor, but there could be other constructors that run
> before it.

Are you sure? I don't see any ctors in compiler-rt trunk:
~/llvm/trunk/projects/compiler-rt/lib/asan$ grep constructor * | grep
-v TestCases
asan_thread.cc: // On Android, libc constructor is called _after_
asan_init, and cleans up
asan_thread.h: // NOTE: There is no AsanThread constructor. It is allocated

>> Some proposal from me:
>> * could you add a note about spurious "failed to intercept" warnings?
>> (see http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58680 for reference)
>
> Hm? What about that? How would you phrase it?

Added to Cons section in wiki.

>> BTW should we acronimize as ASAN or ASan?
>
> Emm. I equally like all of {asan,ASAN,ASan} and I don't care much.

I don't care much but I vaguely remember that wiki once explicitly
insisted on "ASan".

-Y

[Konstantin Serebryany]

On Fri, Feb 7, 2014 at 12:24 PM, Yuri Gribov <tetr...@gmail.com> wrote:

>> 2) Can we add a check to REAL() to produce a nice diagnostic message
>> in this case?
>
> There are a few such checks.

I don't think that REAL() currently checks anything:

REAL doesn't check anything. Some interceptors do the checks, e.g.

INTERCEPTOR(char*, strdup, const char *s) {

  if (!asan_inited) return internal_strdup(s);

  ENSURE_ASAN_INITED();

...

 

~/llvm/trunk/projects/compiler-rt/lib$ grep NULL interception/* -l
interception/interception_win.cc

We seem to rely on callers to make sure that these variables are
properly initialized.

>> 3) We once discussed making __asan_init a constructor to solve this
>> problem (discussion and patch available here:
>> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58937).
>
>
> __asan_init is a constructor, but there could be other constructors that run
> before it.

Are you sure? I don't see any ctors in compiler-rt trunk:

The compiler eimits a CTOR function that call __asan_init

clang  -fsanitize=address foo.cc && objdump -d ./a.out 

000000000047dbd0 <asan.module_ctor>:

  47dbd0:       55                      push   %rbp

  47dbd1:       48 89 e5                mov    %rsp,%rbp

  47dbd4:       48 83 ec 10             sub    $0x10,%rsp

  47dbd8:       e8 d3 d6 fe ff          callq  46b2b0 <__asan_init_v3>

 

~/llvm/trunk/projects/compiler-rt/lib/asan$ grep constructor *  | grep
-v TestCases
asan_thread.cc:      // On Android, libc constructor is called _after_
asan_init, and cleans up
asan_thread.h:  // NOTE: There is no AsanThread constructor. It is allocated

>> Some proposal from me:
>> * could you add a note about spurious "failed to intercept" warnings?
>> (see http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58680 for reference)
>
> Hm? What about that? How would you phrase it?

Added to Cons section in wiki.

I don't really like the phrasing. 

First, debug output concerns only developers, not (most) users.

Second, what does this output mean?  

I suspect that this means the same thing as with -static-libstdc++ -- asan failed to intercept one of the functions is wants to intercept. 

For memcpy&friends it means that asan will not find bugs there.

For some other functions (e.g. longjmp) it means that asan may report false positives.

This problem needs deeper analysis before I can recommend asan-DSO to anyone. 

--kcc 

[Yuri Gribov]

>> I don't think that REAL() currently checks anything:
>
> REAL doesn't check anything. Some interceptors do the checks

That's what I meant to say - instead or relying on interceptors (I'm
pretty sure we missed something out or may miss in future), just check
it directly in REAL().

>> Are you sure? I don't see any ctors in compiler-rt trunk:
>
> The compiler eimits a CTOR function that call __asan_init

Is this one coming from asan_preinit? What about unsanitized
executable with preloaded ASan runtime? My proposal was to add ctor
directly to ASan runtime.

> First, debug output concerns only developers, not (most) users.

Well, debug output is the first thing people will check when something
goes wrong with ASan DSO will stop working for them so I think it
makes sense to give some hints.

> Second, what does this output mean?
> I suspect that this means the same thing as with -static-libstdc++ -- asan
> failed to intercept one of the functions is wants to intercept.

Not really, it's the idiosyncrasy of Linux ld.so which resolves
intercepted symbols to PLT entries in main executable instead of ASan
interceptors. These warnings are really spurious and misleading
(please refer to http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58680 for
more details).

-Y

[Konstantin Serebryany]

On Fri, Feb 7, 2014 at 3:44 PM, Yuri Gribov <tetr...@gmail.com> wrote:

>> I don't think that REAL() currently checks anything:
>
> REAL doesn't check anything. Some interceptors do the checks

That's what I meant to say - instead or relying on interceptors (I'm
pretty sure we missed something out or may miss in future), just check
it directly in REAL().

That's an overkill. Doing a check in REAL() will require  disruptive changes 

and may negatively affect performance, while if REAL(foo) is NULL when it's called 

we'll get a SEGV with a descriptive stack strace. 

 

>> Are you sure? I don't see any ctors in compiler-rt trunk:
>
> The compiler eimits a CTOR function that call __asan_init

Is this one coming from asan_preinit? What about unsanitized
executable with preloaded ASan runtime? My proposal was to add ctor
directly to ASan runtime.

Do you have a test case where it would help?

 

> First, debug output concerns only developers, not (most) users.

Well, debug output is the first thing people will check when something
goes wrong with ASan DSO will stop working for them so I think it
makes sense to give some hints.

> Second, what does this output mean?
> I suspect that this means the same thing as with -static-libstdc++ -- asan
> failed to intercept one of the functions is wants to intercept.

Not really, it's the idiosyncrasy of Linux ld.so which resolves
intercepted symbols to PLT entries in main executable instead of ASan
interceptors. These warnings are really spurious and misleading
(please refer to http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58680 for
more details).

Ah. Now I remember this. Yes, we may want to improve the warning to handle the PLT case.

No idea how, and the change would need to be testable in LLVM tree, which in turn requires that 

we build and support asan-dso in llvm tree as a non-default alternative to the static run-time. 

We may eventually get to it, but no promises.

--kcc

[Yuri Gribov]

> That's an overkill. Doing a check in REAL() will require disruptive changes
> and may negatively affect performance, while if REAL(foo) is NULL when it's
> called we'll get a SEGV with a descriptive stack strace.

Np, as long as we give a hint about this on wiki.

>> My proposal was to add ctor
>> directly to ASan runtime.
>
> Do you have a test case where it would help?

Here is one example which currently works by pure luck:

INTERCEPTOR(int, sigaction, int signum, const struct sigaction *act,
struct sigaction *oldact) {
if (!AsanInterceptsSignal(signum) ||
common_flags()->allow_user_segv_handler) {
return REAL(sigaction)(signum, act, oldact);
}

Note the lack of ENSURE_ASAN_INITED - this may easily result in call
to NULL function if we preload libasan to unsanitized executable which
calls signal() at start of main. The only reason why it manages to
work is because libstdc++ calls __cxa_atexit at start and this calls
ENSURE_ASAN_INITED. But relying on __cxa_atexit is surely messy.

Having an __attribute__((constructor)) for __asan_init would clean all
this stuff up.

> Ah. Now I remember this. Yes, we may want to improve the warning to handle
> the PLT case.
> No idea how, and the change would need to be testable in LLVM tree, which in
> turn requires that
> we build and support asan-dso in llvm tree as a non-default alternative to
> the static run-time.
> We may eventually get to it, but no promises.

Well, I'll keep hoping then)

-Y

[Konstantin Serebryany]

Here is one example which currently works by pure luck:

INTERCEPTOR(int, sigaction, int signum, const struct sigaction *act,
                            struct sigaction *oldact) {
  if (!AsanInterceptsSignal(signum) ||
      common_flags()->allow_user_segv_handler) {
    return REAL(sigaction)(signum, act, oldact);
}

Agree, this may need to be fixed. 

Note the lack of ENSURE_ASAN_INITED - this may easily result in call
to NULL function if we preload libasan to unsanitized executable which
calls signal() at start of main. The only reason why it manages to
work is because libstdc++ calls __cxa_atexit at start and this calls
ENSURE_ASAN_INITED. But relying on __cxa_atexit is surely messy.

Having an __attribute__((constructor)) for __asan_init would clean all
this stuff up.

> Ah. Now I remember this. Yes, we may want to improve the warning to handle
> the PLT case.
> No idea how, and the change would need to be testable in LLVM tree, which in
> turn requires that
> we build and support asan-dso in llvm tree as a non-default alternative to
> the static run-time.
> We may eventually get to it, but no promises.

Well, I'll keep hoping then)

Patches are welcome, as usual. :) :) :) 

We'll need to hack both cmake and configure to produce a DSO, make all tests run with dso 

(except where they are broken), add separate tests to run with LD_PRELOAD,

add a driver flag to clang to use dso, make sure all TLS vars in all sanitizers are initial-exec,

do something to prohibit dlopen for this dso (and add a test).

Extend the wiki explaining why one should not use DSO if at all possible and what troubles await him if he does. :) 

Perhaps something else. 

I am not trying to say that if you don't make these patches we will never get to them. 

We may eventually. But this is nowhere at the top of our current list.