TSAN doesn't sybmolize dynamically linked SO

[Alex Maximov]

Hi All,

I've got the following problem with TSAN (libtsan-8.2.1-3.el7.x86_64) on RH 7.3 - it doesn't symbolize symbols from dynamically loaded SOs (while does for the executed binary itself):

Previous write of size 8 at 0x7b9c00015b28 by main thread (mutexes: write M1058763297353980672):
#0 Mutex_unlock ../../iocore/eventsystem/I_Lock.h:567 (traffic_server+0x126402)
#1 MutexLock::unlock() ../../iocore/eventsystem/I_Lock.h:644 (traffic_server+0x126402)
#2 MutexLock::~MutexLock() ../../iocore/eventsystem/I_Lock.h:655 (traffic_server+0x126402)
#3 TSContScheduleOn /nfs-bfs/workspace/odc/azolotar/p4/sdmce/core/2.X/third-party/TrafficServer/trafficserver-3.2.0/proxy/InkAPI.cc:4689 (traffic_server+0x120358)
#4 <null> <null> (libLogQueue.so+0x54662)
#5 <null> <null> (libLogQueue.so+0x5472b)
#6 <null> <null> (libLogQueue.so+0x55ea5)
#7 <null> <null> (libLogQueue.so+0x377ec)
#8 <null> <null> (libUdNotificationsSP.so+0x24f94)
#9 <null> <null> (libUdNotificationsSP.so+0x1c8c8)
#10 <null> <null> (libNGPSPI_BASE4_6_SPA4_8_HTTP4_5_TCP4_4_IP6_0_IPC4_2_TXN4_3_AML1_0_DS1_0_WRAPPER1_12.so+0x3a625)
#11 <null> <null> (libUdNotificationsSP.so+0x1adba)
#12 <null> <null> (libspf.so+0x62e77)
#13 <null> <null> (libspf.so+0x404a5)
#14 <null> <null> (libspf.so+0x47743)
#15 <null> <null> (libspf.so+0x7c010)
#16 <null> <null> (libspf.so+0x48aba)
#17 <null> <null> (libspf.so+0x3f628)
#18 <null> <null> (libspf.so+0x47743)
#19 <null> <null> (libspf.so+0x7c010)
#20 <null> <null> (libspf.so+0x282c6)
#21 <null> <null> (libSPA.so+0x311e8)
#22 <null> <null> (libts_opwv.so+0xb2a61)
#23 <null> <null> (libts_opwv.so+0xb29f4)
#24 <null> <null> (libts_opwv.so+0xb29ee)
#25 <null> <null> (libts_opwv.so+0xb6aba)
#26 plugin_load /nfs-bfs/workspace/odc/azolotar/p4/sdmce/core/2.X/third-party/TrafficServer/trafficserver-3.2.0/proxy/Plugin.cc:166 (traffic_server+0x147d3b)
#27 plugin_init(char const*, bool) /nfs-bfs/workspace/odc/azolotar/p4/sdmce/core/2.X/third-party/TrafficServer/trafficserver-3.2.0/proxy/Plugin.cc:367 (traffic_server+0x147d3b)
#28 main /nfs-bfs/workspace/odc/azolotar/p4/sdmce/core/2.X/third-party/TrafficServer/trafficserver-3.2.0/proxy/Main.cc:1791 (traffic_server+0xbaa7b)

I was able to print symbols using addr2line:

addr2line -C -f -p -e /opt/opwv/sdmce/SystemActivePath/protocolhandlers/libLogQueue.so 0x54662
TLogQueueReader::scheduleReadNextMessage() at /nfs-bfs/workspace/odc/azolotar/p4/sdmce/core/2.X/http/src/logqueue/LogQueueReader.cpp:213

addr2line -C -f -p -e /opt/opwv/sdmce/SystemActivePath/plugins/libUdNotificationsSP.so 0x24f94
TUdNotificationsClientConfig::LoadConfig(TChglogConfig const&, TUdNotificationsClientConfig*) at /nfs-bfs/workspace/odc/azolotar/p4/sdmce/core/2.X/plugins/src/UdNotifications/UdNotificationsClientConfig.cpp:357 (discriminator 1)

So was wondering what might be the problem here.

I've found a post - wondering if this might be related:
https://stackoverflow.com/questions/30092516/threadsanitizer-tsan-meaningful-information-from-shared-library

Note that ASAN libasan-4.8.5-36.el7.x86_64 prints symbols on the same system:

==32699== ERROR: AddressSanitizer: heap-use-after-free on address 0x60380005f7e8 at pc 0x7f987ad4d6a8 bp 0x7ffe23f83c10 sp 0x7ffe23f83c00
READ of size 8 at 0x60380005f7e8 thread T0 ([ET_NET 0])
    #0 0x7f987ad4d6a7 in SpiGetRqstExecutionStatus_4_0 /nfs-bfs/workspace/odc/azolotar/p4/sdmce/core/2.X/spa/src/spf/SpfSpi.cpp:210
    #1 0x7f987609e82e in NGP_SPI_NS_BASE4_6_SPA4_8_HTTP4_5_TCP4_4_IP6_0_IPC4_2_TXN4_3_DDM1_0_WRAPPER1_12::TPluginRequest::getExecutionStatus(TSpiExecutionStatus&, void*) const /nfs-bfs/workspace/odc/azolotar/p4/sdmce/core/2.X/spa/src/spi/PluginRequest.cpp:148
    #2 0x7f985e150a73 in TLockDistributionSE::processAMLLockList(NGP_SPI_NS_BASE4_6_SPA4_8_HTTP4_5_TCP4_4_IP6_0_IPC4_2_TXN4_3_DDM1_0_WRAPPER1_12::TPluginRequest&, Locking::TAMLLockList const*&) /nfs-bfs/workspace/odc/azolotar/p4/sdmce/core/2.X/plugins/src/Locking/LockDistributionSE.cpp:316

Wondering what might be the problem here.

Thanks,
Alexander

[Alexander Potapenko]

There's some "workaround" proposed at
https://github.com/google/sanitizers/issues/89
It changes the program behavior, but may help in most cases.

> --
> You received this message because you are subscribed to the Google Groups "thread-sanitizer" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to thread-sanitiz...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/thread-sanitizer/1562239678.512639930%40f505.i.mail.ru.



--
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

[Alex Maximov]

Thanks for the suggestion but unfortunatelly the proposed workaround doesn't seem to work.

Interestingly TSAN prints symbols if I run my small test program with either statically or dynamically loaded test SO

WARNING: ThreadSanitizer: data race (pid=13175)
Read of size 4 at 0x7ff664bff078 by thread T2:
#0 Thread2 /home/almax/MyFiles/test/tsan/answer.c:19 (libtestshared.so+0xd0f)
#1 <null> <null> (libtsan.so.0+0x29c8d)

Previous write of size 4 at 0x7ff664bff078 by thread T1:
#0 Thread1 /home/almax/MyFiles/test/tsan/answer.c:14 (libtestshared.so+0xcca)
#1 <null> <null> (libtsan.so.0+0x29c8d)

Location is global 'Global' of size 4 at 0x7ff664bff078 (libtestshared.so+0x000000202078)

Thread T2 (tid=13178, running) created by main thread at:
#0 pthread_create <null> (libtsan.so.0+0x2c2ce)
#1 create_race /home/almax/MyFiles/test/tsan/answer.c:26 (libtestshared.so+0xd99)
#2 main /home/almax/MyFiles/test/tsan/main_dl.c:65 (main_shared_dl+0xc73)

Thread T1 (tid=13177, finished) created by main thread at:
#0 pthread_create <null> (libtsan.so.0+0x2c2ce)
#1 create_race /home/almax/MyFiles/test/tsan/answer.c:25 (libtestshared.so+0xd78)
#2 main /home/almax/MyFiles/test/tsan/main_dl.c:65 (main_shared_dl+0xc73)

SUMMARY: ThreadSanitizer: data race /home/almax/MyFiles/test/tsan/answer.c:19 in Thread2
==================
ThreadSanitizer: reported 1 warnings

But by some reasons this doesn't work for the product I mentioned below. Wondering if this might happen because of different build/linker flags.

Четверг, 4 июля 2019, 15:32 +03:00 от Alexander Potapenko <gli...@google.com>:

--
Alex Maximov

[Dmitry Vyukov]

On Thu, Jul 4, 2019 at 3:30 PM 'Alex Maximov' via thread-sanitizer

Hi Alex,

Unfortunately nothing come to mind right away.
FWIW TSAN and ASAN should use the same symbolization logic, so they
both should either succeed or both fail. At least in theory and for
the same version of runtimes.
I see only one way to move forward with this: re-reproduce this issue
with the latest TSAN runtime built from source and then extending
debug output in TSAN runtime to figure out where exactly things go
wrong (e.g. was it able to find the right full path to the library?
was it able to start addr2line? what did addr2line replied, etc). Such
kind of verbose debug output would probably be useful for other cases
too (when running with TSAN_OPTIONS=verbosity=3).

[Alex Maximov]

Hi Dmitry,

Thanks for your suggestion - will probably try to build TSAN by myself.
So far I've found https://github.com/google/sanitizers/wiki/ThreadSanitizerDevelopment#gcc-development page describing how to build TSAN - was wondering if there is a way to build sanitizers library (libsanitizer sub-project) only but not the whole gcc project? - I've tried but so far compilation failed.

BTW, one more interesting thing I've noticed with TSAN symbols are printed OK for libraries missing symbols below (e.g. libLogQueue.so) if they are loaded from unit-test harness, so it seems like something wrong happens with the symbolizer when running traffic_server process - I tend to think there are some issues with loaded SO paths.

WRT why symbolizer works for ASAN - quite old version 4.8.5 of ASAN is used and llvm-symbolizer tool is used to print symbols (ASAN_SYMBOLIZER_PATH EVN is set for this) so perhaps different mechanics is used to print symbols.

Thanks,
Alex

Четверг, 4 июля 2019, 20:21 +03:00 от Dmitry Vyukov <dvy...@google.com>:

--
Alex Maximov

[Dmitry Vyukov]

On Fri, Jul 5, 2019 at 10:56 AM Alex Maximov <a_...@mail.ru> wrote:
>
> Hi Dmitry,
>
> Thanks for your suggestion - will probably try to build TSAN by myself.
> So far I've found https://github.com/google/sanitizers/wiki/ThreadSanitizerDevelopment#gcc-development page describing how to build TSAN - was wondering if there is a way to build sanitizers library (libsanitizer sub-project) only but not the whole gcc project? - I've tried but so far compilation failed.

I am not expert on gcc compilation. I usually just do make && make
install to build everything. But I assume it should some kind of
incremental build, so if you change just the sanitizers, it should
rebuild only them.

Sanitizers upstream is location in llvm, gcc only imports runtimes
periodically. I usually use these instructions for llvm:
https://github.com/google/sanitizers/wiki/AddressSanitizerHowToBuild
For llvm incremental build works well (esp with ninja generator), so I
never bothers to build only parts.

[xfan]

Hi Alexander,

Have you found a solution? 

I got the same problem with TSAN on gcc 8.3 RedHat Linux. Some call stack show symbols some not (like below)

    #3 <null> <null> (libMHDocSrv.so+0x36c7e8)

    #4 <null> <null> (libMHDocSrv.so+0x373c0c)

    #5 <null> <null> (libMHDocSrv.so+0x374b00)

    #6 <null> <null> (libMHDocSrv.so+0x374f16)

It works for Asan also.  

Thanks,

Xingyan

[alex...@gmail.com]

Hi Xingyan,

Unfortunatelly by this moment I still have not found the solution. As a workaround addr2line utility might be used to print symbols

addr2line -C -f -p -e libMHDocSrv.so 0x36c7e8

Note that you binaries should be compiled with -g to include simbols. Further you could create a script using addr2line to symbolize complete stack trace.

Regards,

Alexander

четверг, 11 июля 2019 г., 18:07:25 UTC+3 пользователь xfan написал:

[xfan]

Hi Alexander, 

Thanks for the reply. That's what I did too.  But we have to use a filter script to filter some false positives because suppression list doesn't work for those symbols. 

Regards,

Xingyan