ThreadSanitizer can not mmap the shadow memory

[bhupe...@gmail.com]

Hi All,

OS: RHEL7  (x86_64 GNU/Linux)
Kernal: 3.10.0-123.8.1.el7.x86_64
Compiler: gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC)
RAM: 24 GB

Our product contains shared libraries, archive libraries, java programs (jars) and many executable (c, c++, java).
Our aim is to evaluate the value of Tsan (thread sanitizer) for our product and find concurrency issues.

Therefore, we have built all shared libraries and executable (c, c++) with tsan enabled as follows:
Compile flags: -fsanitize=thread -g
Shared library linking flags: -fsanitize=thread -shared -lpthread -ldl
Executable linking flags: -fsanitize=thread -pie -lpthread -ldl
Note: We have also linked "-ltsan" wherever required.

In addition, we have built object files associated with archive library with "-fPIC" to fix following type of compilation error.
/bin/ld: something.a(something.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC

After compiling all necessary modules with tsan enabled:
When we execute tests (executable) that uses other executable (c, c++) and shared libraries, tsan reports gets generated without any issue.
However, when we execute "main" product which uses shared libraries, archive libraries, java programs (jars) and many executable (c, c++, java) then it shows following "well-known" error message.
FATAL: ThreadSanitizer can not mmap the shadow memory (something is mapped at 0x400000 < 0x7cf000000000)
FATAL: Make sure to compile with -fPIE and to link with -pie.

We have further analyzed but unable to find any root cause.
However, our product have several shared libraries which it access through JNI.
We feel that shared libraries compiled with tsan and access through JNI might cause this issue.

We have also found issue (not similar but close enough) https://github.com/google/sanitizers/issues/386 related to address sanitizer.
Please let us know your suggestion and some pointers to resolve shadow memory issue.
Kindly let us know if you need any further information.

[Dmitry Vyukov]

Hi,

Main binary is usually mapped at 0x400000 if not compiled with -pie
-fPIE. Recompile it with -pie -fPIE.
Also, latest tsan in clang does support non-pie binaries, so you can
just build clang and use it. Instructions on building clang are here:
https://github.com/google/sanitizers/wiki/AddressSanitizerHowToBuild

[Bhupendra Thosare]

Thanks for your quick response.

After further analysis, we found that there is "one" binary that called by "main" binary.
This "one" binary is compiled with "-fsanitize=thread -pie".
Also, "one" binary internally uses two shared library that linked with "-fsanitize=thread -pie" and "another" relocatable library whose only object files are compiled with "-fsanitize=thread".

It seems "another" relocatable library causing the shadow memory issue.
However, we were unable to build this "another" relocatable library (ld -r) with "-fsanitize=thread -pie" due to following type of compilation errors:
/usr/bin/ld: -r and -shared may not be used together

Please let us know your suggestion.
In the meantime, we will try to reproduce the shadow memory issue with dummy sample program.

Thank you very much for your help.

[Dmitry Vyukov]

On Tue, Oct 20, 2015 at 8:05 PM, Bhupendra Thosare <bhupe...@gmail.com> wrote:
> Thanks for your quick response.
>
> After further analysis, we found that there is "one" binary that called by
> "main" binary.
> This "one" binary is compiled with "-fsanitize=thread -pie".
> Also, "one" binary internally uses two shared library that linked with
> "-fsanitize=thread -pie" and "another" relocatable library whose only object
> files are compiled with "-fsanitize=thread".
>
> It seems "another" relocatable library causing the shadow memory issue.
> However, we were unable to build this "another" relocatable library (ld -r)
> with "-fsanitize=thread -pie" due to following type of compilation errors:
> /usr/bin/ld: -r and -shared may not be used together

Who does add -shared in this case?
Do you actually need a relocatable library (-r)?
Also for dynamic libraries (.so) you must not use -pie (pie is
"position-independent executable", so it is only for executables, not
libraries). When you are compiling a library, you need to pass -fPIC,
this flag should be already in your Makefile. When you link a shared
library, you need to use -shared, this flag should also already be in
your Makefile.
If you build the library by adding just "-fsanitize=thread", does it help?

> --
> You received this message because you are subscribed to the Google Groups
> "thread-sanitizer" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to thread-sanitiz...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[bcm...@google.com]

I'm seeing this with gcc7 even with -pie -fPIE.

Per https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual#faq, I've also checked that /proc/sys/kernel/randomize_va_space is 2.

Is there some additional flag required for GCC 7?

bcmills$ cat /proc/sys/kernel/randomize_va_space

2

bcmills$ cat return0.c

int main() {

  return 0;

}

bcmills$ $CC -fsanitize=thread -pie -fPIE return0.c

bcmills$ ./a.out

FATAL: ThreadSanitizer can not mmap the shadow memory (something is mapped at 0x560b4693d000 < 0x7cf000000000)

FATAL: Make sure to compile with -fPIE and to link with -pie.

bcmills✘ $CC --version

gcc (GCC) 7.1.0

[Dmitry Vyukov]

Hi,

Support for 4.1+ kernels that map binaries at 0x55 is added to tsan
long time ago. But it probably still not in gcc7.
As far as I remember the workaround was to run the program under
'setarch x86_64 -R ./a.out'. Or use newer compiler, or older kernel.

[bcm...@google.com]

Doesn't seem to work with setarch:

bcmills$ $CC -fsanitize=thread -fPIE -pie return0.c

bcmills$ setarch x86_64 -R ./a.out

FATAL: ThreadSanitizer can not mmap the shadow memory (something is mapped at 0x555555554000 < 0x7cf000000000)

FATAL: Make sure to compile with -fPIE and to link with -pie.

GCC 7.1 was the newest GCC build I could find.

(I'm trying to test a fix for http://golang.org/issue/21196, and the tests already pass with Clang, but I'm having trouble finding a gcc with a working TSAN at all.)

[Dmitry Vyukov]

gcc (GCC) 7.1.1 20170620 works for me with/without pie.
Make sure that you use the right libtsan.so. Gcc links tsan
dynamically by default, as opposed to clang which links statically.
You may need to use LD_LIBRARY_PATH or -static-libtsan compiler flag.

On Wed, Aug 2, 2017 at 11:19 PM, bcmills via thread-sanitizer

[Bryan C. Mills]

-static-libtsan did the trick. Thanks!

> email to thread-sanitizer+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "thread-sanitizer" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/thread-sanitizer/jzqvvEzC8EU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to thread-sanitizer+unsubscribe@googlegroups.com.