Asan initialization broken on Android?

[Dmitry Vyukov]

The error reported by buildbot is:

=================================================================
==26030==ERROR: AddressSanitizer: SEGV on unknown address 0xeaf50000
(pc 0xeadd3514 bp 0xc83e6c68 sp 0xc83e6c00 T-1)
==26030==The signal is caused by a READ memory access.
#0 0xeadd3514 in GetThreadLocked
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_thread_registry.h:103:41
#1 0xeadd3514 in GetThreadContextByTidLocked
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/asan/asan_thread.cpp:72:28
#2 0xeadd3514 in __asan::GetCurrentThread()
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/asan/asan_thread.cpp:433:33
#3 0xead771b4 in GetFakeStack
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/asan/asan_fake_stack.cpp:188:19
#4 0xead771b4 in GetFakeStackFastAlways
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/asan/asan_fake_stack.cpp:204:10
#5 0xead771b4 in OnMallocAlways
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/asan/asan_fake_stack.cpp:220:19
#6 0xead771b4 in __asan_stack_malloc_always_0
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/asan/asan_fake_stack.cpp:256:1
#7 0xad341ab0 in SignalHandler(int, siginfo*, void*)
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/test/asan/TestCases/Linux/uar_signals.cpp:19
#8 0xeab4174c (/apex/com.android.runtime/lib/bionic/libc.so+0x3374c)
#9 0xead51da8 in __internal_syscall
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_syscall_linux_arm.inc:109:3
#10 0xead51da8 in __sanitizer::internal_mmap(void*, unsigned long,
int, int, int, unsigned long long)
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp:176:10
#11 0xead53d18 in __sanitizer::MmapNamed(void*, unsigned long,
int, int, char const*)
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_posix.cpp:391:14
#12 0xead5b89c in MmapFixed
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp:304:7
#13 0xead5b89c in MmapFixedNoReserve
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp:318:10
#14 0xead5b89c in __sanitizer::MmapFixedSuperNoReserve(unsigned
long, unsigned long, char const*)
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp:328:12
#15 0xead59bb8 in __sanitizer::ReserveShadowMemoryRange(unsigned
long, unsigned long, char const*, bool)
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_common_libcdep.cpp:151:25
#16 0xead76ca0 in PoisonAll
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/asan/asan_fake_stack.cpp:80:3
#17 0xead76ca0 in __asan::FakeStack::Destroy(int)
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/asan/asan_fake_stack.cpp:66:3
#18 0xeadd2e68 in DeleteFakeStack
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/asan/asan_thread.h:98:8
#19 0xeadd2e68 in __asan::AsanThread::Destroy()
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/asan/asan_thread.cpp:116:5
#20 0xeab8ea7e in pthread_key_clean_all() pthread_key.cpp
#21 0xeab8e6ca in pthread_exit
(/apex/com.android.runtime/lib/bionic/libc.so+0x806ca)
#22 0xeab8e620 in __pthread_start(void*) pthread_create.cpp
#23 0xeab47902 in __start_thread clone.cpp

The change dfd9808b6cea59ff075498ee7e6e57f2b5b3a798 did this:

// Should be guarded by ThreadRegistryLock.
ThreadContextBase *GetThreadLocked(u32 tid) {
- DCHECK_LT(tid, n_contexts_);
- return threads_[tid];
+ return threads_.empty() ? nullptr : threads_[tid];
}

It started CHECK failing on Android with accessing threads_[0] when it
has 0 size.
Then we added this "threads_.empty() ? nullptr", but I think it only
masked the real problem.
The real problem is who/why is accessing threads_[0] before we
initialized the main thread. And I think this new crash answers this
-- a second thread calls GetThreadLocked(kMainThread) before we
initialized asan runtime and created the main thread.
I think we need to remove this "threads_.empty() ? nullptr" part and
instead ensure that runtime is initialized by the time of the call.





On Tue, 20 Jul 2021 at 09:23, <llvm.bui...@lab.llvm.org> wrote:
>
> The Buildbot has detected a failed build on builder sanitizer-x86_64-linux-android while building compiler-rt.
>
> Full details are available at:
> https://lab.llvm.org/buildbot#builders/77/builds/7937
>
> Worker for this Build: sanitizer-buildbot6
> Blamelist:
> Dmitry Vyukov <dvy...@google.com>,
> Johannes Doerfert <joha...@jdoerfert.de>
>
> BUILD FAILED: failed 'python ../sanitizer_buildbot/sanitizers/zorg/buildbot/builders/sanitizers/buildbot_selector.py'
>
> Step 2 (annotate) failure: 'python ../sanitizer_buildbot/sanitizers/zorg/buildbot/builders/sanitizers/buildbot_selector.py'
> python ../sanitizer_buildbot/sanitizers/zorg/buildbot/builders/sanitizers/buildbot_selector.py
> in dir /var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build (timeout 1200 secs)
> watching logfiles {}
> argv: [b'python', b'../sanitizer_buildbot/sanitizers/zorg/buildbot/builders/sanitizers/buildbot_selector.py']
> environment:
> BUILDBOT_BLAMELIST=[]
> BUILDBOT_BRANCH=main
> BUILDBOT_BUILDERNAME=sanitizer-x86_64-linux-android
> BUILDBOT_BUILDNUMBER=7937
> BUILDBOT_CLOBBER=
> BUILDBOT_GOT_REVISION=b899cd8edcb824c4e4f999ef254209060d1ab646
> BUILDBOT_REVISION=adb55d7c326559e99f1eeb3234cdfaaf3c9010de
> BUILDBOT_SCHEDULER=main:clang,compiler-rt,libcxx,libcxxabi,libunwind,lld,llvm
> HOME=/var/lib/buildbot
> INVOCATION_ID=c05ecc968f624671b2e2ffd0f558eeaa
> JOURNAL_STREAM=9:2763
> LANG=en_US.UTF-8
> LOGNAME=buildbot
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> PWD=/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build
> SHELL=/bin/bash
> TERM=dumb
> USER=buildbot
> using PTY: False
> builder name: sanitizer-x86_64-linux-android
> sanitizer-x86_64-linux-android runs: bash ../sanitizer_buildbot/sanitizers/zorg/buildbot/builders/sanitizers/buildbot_android.sh
> + set -e
> + set -u
> +++ dirname ../sanitizer_buildbot/sanitizers/zorg/buildbot/builders/sanitizers/buildbot_android.sh
> ++ cd ../sanitizer_buildbot/sanitizers/zorg/buildbot/builders/sanitizers
> ++ pwd
> + HERE=/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/sanitizer_buildbot/sanitizers/zorg/buildbot/builders/sanitizers
> + . /var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/sanitizer_buildbot/sanitizers/zorg/buildbot/builders/sanitizers/buildbot_functions.sh
> ++ echo @@@BUILD_STEP Info@@@
> Step 21 (run lit tests [arm/crosshatch-userdebug/RP1A.201105.002]) failure: run lit tests [arm/crosshatch-userdebug/RP1A.201105.002] (failure)
> ...
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/asan_preload_test-2.cpp (44 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/asan_preload_test-3.cpp (45 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/coverage-missing.cpp (46 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/cuda_test.cpp (47 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/dlopen-mixed-c-cxx.c (48 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/auto_memory_profile_test.cpp (49 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/bzero.cpp (50 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/calloc-preload.c (51 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/function-sections-are-bad.cpp (52 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/init-order-dlopen.cpp (53 of 1460)
> XFAIL: AddressSanitizer-arm-android :: TestCases/Linux/globals-gc-sections-lld.cpp (54 of 1460)
> XFAIL: AddressSanitizer-arm-android :: TestCases/Linux/asan_dlopen_test.cpp (55 of 1460)
> PASS: AddressSanitizer-arm-android :: TestCases/Linux/init_fini_sections.cpp (56 of 1460)
> PASS: AddressSanitizer-arm-android :: TestCases/Linux/asan_default_suppressions.cpp (57 of 1460)
> PASS: AddressSanitizer-arm-android :: TestCases/Linux/global-overflow-bfd.cpp (58 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/interface_symbols_linux.cpp (59 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/kernel-area.cpp (60 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/leak.cpp (61 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/leak_check_segv.cpp (62 of 1460)
> PASS: AddressSanitizer-arm-android :: TestCases/Linux/global-overflow-lld.cpp (63 of 1460)
> XFAIL: AddressSanitizer-arm-android :: TestCases/Linux/asan_rt_confict_test-1.cpp (64 of 1460)
> XFAIL: AddressSanitizer-arm-android :: TestCases/Linux/asan_rt_confict_test-2.cpp (65 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/malloc-in-qsort.cpp (66 of 1460)
> PASS: AddressSanitizer-arm-android :: TestCases/Linux/long-object-path.cpp (67 of 1460)
> PASS: AddressSanitizer-arm-android :: TestCases/Linux/longjmp_chk.c (68 of 1460)
> PASS: AddressSanitizer-arm-android :: TestCases/Linux/local_alias.cpp (69 of 1460)
> PASS: AddressSanitizer-arm-android :: TestCases/Linux/aligned_delete_test.cpp (70 of 1460)
> XFAIL: AddressSanitizer-arm-android :: TestCases/Linux/interception_readdir_r_test.cpp (71 of 1460)
> PASS: AddressSanitizer-arm-android :: TestCases/Linux/initialization-bug-any-order.cpp (72 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/nohugepage_test.cpp (73 of 1460)
> PASS: AddressSanitizer-arm-android :: TestCases/Linux/mincore.cpp (74 of 1460)
> XFAIL: AddressSanitizer-arm-android :: TestCases/Linux/activation-options.cpp (75 of 1460)
> PASS: AddressSanitizer-arm-android :: TestCases/Linux/clone_test.cpp (76 of 1460)
> PASS: AddressSanitizer-arm-android :: TestCases/Linux/new_delete_mismatch_global.cpp (77 of 1460)
> PASS: AddressSanitizer-arm-android :: TestCases/Linux/memmem_test.cpp (78 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/overflow-in-qsort.cpp (79 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/preinstalled_signal.cpp (80 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/print_memory_profile_test.cpp (81 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/printf-fortify-1.c (82 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/printf-fortify-2.c (83 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/printf-fortify-3.c (84 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/printf-fortify-4.c (85 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/printf-fortify-5.c (86 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/pthread_create_from_constructor.cpp (87 of 1460)
> PASS: AddressSanitizer-arm-android :: TestCases/Linux/clang_gcc_abi.cpp (88 of 1460)
> PASS: AddressSanitizer-arm-android :: TestCases/Linux/new_delete_mismatch_stack.cpp (89 of 1460)
> XFAIL: AddressSanitizer-arm-android :: TestCases/Linux/ptrace.cpp (90 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/read_binary_name_regtest.c (91 of 1460)
> UNSUPPORTED: AddressSanitizer-arm-android :: TestCases/Linux/recoverable-lsan.cpp (92 of 1460)
>
> Sincerely,
> LLVM Buildbot
>

[Evgenii Stepanov]

Sounds right. I'm not sure how there can be a secondary thread while ASan runtime is not initialized, we'll need to look into that. Can't promise a quick resolution though.

[Dmitry Vyukov]

Looking at the stack more closely, this happens in AsanThread::Destroy
after we destroyed the thread in ThreadRegistry:

void AsanThread::Destroy() {
...
(asanThreadRegistry().FinishThread(tid) == ThreadStatusRunning);
...
DeleteFakeStack(tid);


Maybe that's the problem.