Fatal check failure from sanitizer_allocator.h

[Ilya Mirsky]

Hi,

I'm successfully running a toy program built with TSan and getting reports and everything, but getting the following error when trying to run a more complex and heavily multi-threaded application:

FATAL: ThreadSanitizer CHECK failed: /build/llvm-3.3/projects/compiler-rt/lib/sanitizer_common/sanitizer_allocator.h:996 "((IsAligned(reinterpret_cast<uptr>(p), page_size_))) != (0)" (0x0, 0x0)

    #0 __tsan::PrintCurrentStackSlow() ??:0 (exe+0x000000249290)

    #1 __tsan::TsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) ??:0 (exe+0x000000249263)

    #2 __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) ??:0 (exe+0x000000212e11)

    #3 __sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator64<137438953472000ul, 1099511627776ul, 16ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __tsan::MapUnmapCallback>, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<137438953472000ul, 1099511627776ul, 16ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __tsan::MapUnmapCallback> >, __sanitizer::LargeMmapAllocator<__tsan::MapUnmapCallback> >::GetMetaData(void*) ??:0 (exe+0x00000023ecd1)

    #4 __tsan::user_free(__tsan::ThreadState*, unsigned long, void*) ??:0 (exe+0x00000023e1a9)

    #5 operator delete(void*) ??:0 (exe+0x00000021d4db)

I'm running a VM of Ubuntu Precise with 3.5.0-36 kernel, and using Clang 3.3.

Any ideas what might be cause for the error?

Thanks,

Ilya

[Dmitry Vyukov]

Hi,

I don't remember any bugs with such failure mode.
My first guess would be that a bad pointer is passed to free. What is
pointer value? Was it allocated with new?

3.3 release is quite old. Is it reproducible with clang 3.6 or tip?

[Ilya Mirsky]

Hi Dmitry,

Thanks for the prompt reply!

I switched to Clang 3.6 and now I'm getting a slightly different error (see below), stemming from a different location in the code. An important detail I didn't mention before- we are using a custom allocator in the application. Can TSan be used at all with a custom allocator?

FATAL: ThreadSanitizer CHECK failed: /tmp/buildd/llvm-toolchain-3.6-3.6~svn230431/projects/compiler-rt/lib/sanitizer_common/sanitizer_allocator.h:1226 "((IsAligned(p, page_size_))) != (0)" (0x0, 0x0)

    #0 __tsan::TsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) <null> (nsfw+0x0000005d0bb2)

    #1 __tsan::TsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) <null> (nsfw+0x0000005d0bba)

    #2 __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) <null> (nsfw+0x00000055adc1)

    #3 __sanitizer::LargeMmapAllocator<__tsan::MapUnmapCallback>::Deallocate(__sanitizer::AllocatorStats*, void*) <null> (nsfw+0x0000005c4244)

    #4 __tsan::user_free(__tsan::ThreadState*, unsigned long, void*, bool) <null> (nsfw+0x0000005c2755)

    #5 operator delete(void*) <null> (nsfw+0x000000570bfb)

Cheers,

Ilya

[Dmitry Vyukov]

Here is that I think happens. Your allocator redefines a set of
allocation/free function, but tsan redefines more of them. As the
result, an allocation request goes to your allocator, while free
request goes to tsan allocator. Without tsan it works because the free
function forwards to one of the free functions that you intercept;
e.g. you intercept free() but not operator delete[], standard
implementation of operator delete[] simply calls free(), so the
execution goes to your allocator anyway. Tsan intercepts operator
delete[], and so execution do not go into your allocator.

As a side note, the crash can mean you have alloc/dealoc mismatch.
E.g. you allocate memory with operator new[] and then free it with
operator delete. This is a bug. It should be detectable by asan.

If possible, disable your custom allocator under tsan. We do this both
in Chromium and our internal codebase. By default they use tcmalloc,
but under sanitizers we turn off tcmalloc. This will eliminate the
crash and also give better race detection and reports.

[Ilya Mirsky]

Dmitry, many thanks! 

Your assumption was correct and disabling the custom allocator resolved the problem!

One last question- where can I find the changes to TSan between clang/gcc releases?

Is there something like release notes for TSan/ASan and the rest of the tools in the family?

Thanks in advance,

Ilya 

--
You received this message because you are subscribed to the Google Groups "thread-sanitizer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to thread-sanitiz...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dmitry Vyukov]

On Mon, Mar 9, 2015 at 3:49 PM, Ilya Mirsky <ilya....@gmail.com> wrote:
> Dmitry, many thanks!
> Your assumption was correct and disabling the custom allocator resolved the
> problem!
>
> One last question- where can I find the changes to TSan between clang/gcc
> releases?
> Is there something like release notes for TSan/ASan and the rest of the
> tools in the family?

We don't have release notes. Sorry.
What kind of changes are you interested in?