unknown-crash with __asan_memset

[Dmitriy -]

Hello all.

I try using ASan for debug jvm.

All .so library in jvm instrumented with ASan.

But, I have some error here:

LD_PRELOAD=/usr/lib/libclang_rt.asan-x86_64.so LD_LIBRARY_PATH=./dist/jdk_7/debug/open/jdk/jre/lib/amd64/drlvm/ ./dist/jdk_7/debug/open/jdk/jre/bin/java -XX:-UseG1GC -version

=================================================================

==24418==ERROR: AddressSanitizer: unknown-crash on address 0x0000dfff8000 at pc 0x7f6c2e9ab3d9 bp 0x7f6c28773960 sp 0x7f6c28773110

WRITE of size 4194304 at 0x0000dfff8000 thread T1

    #0 0x7f6c2e9ab3d8 in __asan_memset /home/dbezheckov/waratek/tests/asan/sources/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:431

    #1 0x7f6c12fbd7d5 in lspace_initialize(GC*, void*, unsigned long) /home/dbezheckov/waratek/harmony-custom/drlvm/vm/gc_gen/src/los/lspace.cpp:44:5

    #2 0x7f6c12f9fc8c in gc_los_initialize(GC_Gen*, void*, unsigned long) /home/dbezheckov/waratek/harmony-custom/drlvm/vm/gc_gen/src/gen/gen.cpp:476:24

    #3 0x7f6c12f9f63f in gc_gen_initialize(GC_Gen*, unsigned long, unsigned long) /home/dbezheckov/waratek/harmony-custom/drlvm/vm/gc_gen/src/gen/gen.cpp:313:5

    #4 0x7f6c12f848eb in gc_init /home/dbezheckov/waratek/harmony-custom/drlvm/vm/gc_gen/src/common/gc_for_vm.cpp:104:5

    #5 0x7f6c29cc9623 in vm_init1(JavaVM_Internal*, JavaVMInitArgs*, JNIEnv_External**) /home/dbezheckov/waratek/harmony-custom/drlvm/vm/vmcore/src/init/vm_init.cpp:796:14

    #6 0x7f6c29b27f2c in JNI_CreateJavaVM /home/dbezheckov/waratek/harmony-custom/drlvm/vm/vmcore/src/jni/jni.cpp:436:19

    #7 0x7f6c29b286b4 in CVMI_CreateJavaVM /home/dbezheckov/waratek/harmony-custom/drlvm/vm/vmcore/src/jni/jni.cpp:526:12

    #8 0x7f6c2abd0634 in JNI_CreateJavaVM /home/dbezheckov/waratek/harmony-custom/drlvm/vm/openjdk/src/openjdk.cpp:90:12

    #9 0x7f6c2e6fdb47  (/home/dbezheckov/waratek/harmony-custom/dist/jdk_7/debug/open/jdk/jre/bin/../lib/amd64/jli/libjli.so+0x2b47)

    #10 0x7f6c2df1c181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312

    #11 0x7f6c2e43047c in clone /build/buildd/eglibc-2.19/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:111

code for lspace_initialize(GC*, void*, unsigned long):

Lspace *lspace_initialize( GC *gc, void *start, uintptr_t lspace_size ) {
 Lspace *lspace = (Lspace *)STD_MALLOC( sizeof( Lspace ) );
 assert( lspace );
 memset( lspace, 0, sizeof( Lspace ) );
 

 /* commit mspace mem */
 void *reserved_base = start;
 uintptr_t committed_size = lspace_size;
 if( !large_page_hint )
 {
 TRACE2( "gc.initialize", "lspace_initialize() ptr=" << (uintptr_t)reserved_base
 << " size=" << lspace_size );
 vm_commit_mem( reserved_base, lspace_size );
 }
 memset( reserved_base, 0, lspace_size );
 

 min_los_size_bytes -= LOS_HEAD_RESERVE_FOR_HEAP_BASE;
 lspace->committed_heap_size = committed_size - LOS_HEAD_RESERVE_FOR_HEAP_BASE;
 lspace->reserved_heap_size = gc->reserved_heap_size - min_none_los_size_bytes - LOS_HEAD_RESERVE_FOR_HEAP_BASE;
 lspace->heap_start = (void *)( (uintptr_t)reserved_base + LOS_HEAD_RESERVE_FOR_HEAP_BASE );
 lspace->heap_end = (void *)( (uintptr_t)reserved_base + committed_size );
 

 lspace->gc = gc;
 /*LOS_Shrink:*/
 lspace->move_object = 0;
 

 /*Treat with free area buddies*/
 lspace->free_pool = (Free_Area_Pool *)STD_MALLOC( sizeof( Free_Area_Pool ) );
 free_area_pool_init( lspace->free_pool );
 Free_Area *initial_fa = (Free_Area *)lspace->heap_start;
 initial_fa->size = lspace->committed_heap_size;
 free_pool_add_area( lspace->free_pool, initial_fa );
 

 lspace->num_collections = 0;
 lspace->time_collections = 0;
 lspace->survive_ratio = 0.5f;
 lspace->last_allocated_size = 0;
 lspace->accumu_alloced_size = 0;
 lspace->total_alloced_size = 0;
 lspace->last_surviving_size = 0;
 lspace->period_surviving_size = 0;
 

 p_global_lspace_move_obj = &( lspace->move_object );
 los_boundary = lspace->heap_end;
 

 return lspace;
 }

please help.

[Yuri Gribov]

On Wed, May 27, 2015 at 1:52 PM, Dmitriy - <dima...@gmail.com> wrote:

Hello all.

I try using ASan for debug jvm.

All .so library in jvm instrumented with ASan.

But, I have some error here:

LD_PRELOAD=/usr/lib/libclang_rt.asan-x86_64.so LD_LIBRARY_PATH=./dist/jdk_7/debug/open/jdk/jre/lib/amd64/drlvm/ ./dist/jdk_7/debug/open/jdk/jre/bin/java -XX:-UseG1GC -version

=================================================================

==24418==ERROR: AddressSanitizer: unknown-crash on address 0x0000dfff8000 at pc 0x7f6c2e9ab3d9 bp 0x7f6c28773960 sp 0x7f6c28773110

WRITE of size 4194304 at 0x0000dfff8000 thread T1

    #0 0x7f6c2e9ab3d8 in __asan_memset /home/dbezheckov/waratek/tests/asan/sources/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:431

    #1 0x7f6c12fbd7d5 in lspace_initialize(GC*, void*, unsigned long) /home/dbezheckov/waratek/harmony-custom/drlvm/vm/gc_gen/src/los/lspace.cpp:44:5

    #2 0x7f6c12f9fc8c in gc_los_initialize(GC_Gen*, void*, unsigned long) /home/dbezheckov/waratek/harmony-custom/drlvm/vm/gc_gen/src/gen/gen.cpp:476:24

    #3 0x7f6c12f9f63f in gc_gen_initialize(GC_Gen*, unsigned long, unsigned long) /home/dbezheckov/waratek/harmony-custom/drlvm/vm/gc_gen/src/gen/gen.cpp:313:5

    #4 0x7f6c12f848eb in gc_init /home/dbezheckov/waratek/harmony-custom/drlvm/vm/gc_gen/src/common/gc_for_vm.cpp:104:5

    #5 0x7f6c29cc9623 in vm_init1(JavaVM_Internal*, JavaVMInitArgs*, JNIEnv_External**) /home/dbezheckov/waratek/harmony-custom/drlvm/vm/vmcore/src/init/vm_init.cpp:796:14

    #6 0x7f6c29b27f2c in JNI_CreateJavaVM /home/dbezheckov/waratek/harmony-custom/drlvm/vm/vmcore/src/jni/jni.cpp:436:19

    #7 0x7f6c29b286b4 in CVMI_CreateJavaVM /home/dbezheckov/waratek/harmony-custom/drlvm/vm/vmcore/src/jni/jni.cpp:526:12

    #8 0x7f6c2abd0634 in JNI_CreateJavaVM /home/dbezheckov/waratek/harmony-custom/drlvm/vm/openjdk/src/openjdk.cpp:90:12

    #9 0x7f6c2e6fdb47  (/home/dbezheckov/waratek/harmony-custom/dist/jdk_7/debug/open/jdk/jre/bin/../lib/amd64/jli/libjli.so+0x2b47)

    #10 0x7f6c2df1c181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312

    #11 0x7f6c2e43047c in clone /build/buildd/eglibc-2.19/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:111

I don't think this rings any bells. Do you could try to analyze backtrace with gdb? You could use sleep_before_dying to attach to process on error.

-Y

[Dmitriy -]

I tried to debug it with gdb.

Result:

in this line:

memset( reserved_base, 0, lspace_size );

I want to fill reserved_base.

reserved_base = 0xdfff8000

lspace_size = 4194304 (400000 in hex)

my process memory map:

...

0008fff7000 1310692K -----   [ anon ]

000dfff0000     32K -----   [ anon ]

000dfff8000   4096K rw---   [ anon ]

000e03f8000 520224K -----   [ anon ]

00100000000 2145648604K -----   [ anon ]

2008fff7000 15032123396K rw---   [ anon ]

...

java heap start this -> 000dfff8000   4096K rw---   [ anon ]

java heap size = 4194304 bytes

All ok.

I simply want to fill it with zero.

next my step by step gdb ouput:

__asan_memset (block=0xdfff8000, c=0, size=4194304)

    at /home/dbezheckov/waratek/tests/asan/sources/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:431

431   ASAN_MEMSET_IMPL(nullptr, block, c, size);

(gdb) step

QuickCheckForUnpoisonedRegion (size=4194304, beg=3758063616)

    at /home/dbezheckov/waratek/tests/asan/sources/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:34

34   if (size == 0) return true;

(gdb) step

__asan_memset (block=0xdfff8000, c=0, size=4194304)

    at /home/dbezheckov/waratek/tests/asan/sources/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:431

431   ASAN_MEMSET_IMPL(nullptr, block, c, size);

(gdb) step

QuickCheckForUnpoisonedRegion (size=4194304, beg=3758063616)

    at /home/dbezheckov/waratek/tests/asan/sources/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:35

35   if (size <= 32)

(gdb) p size

$5 = 4194304

(gdb) step 

__asan_memset (block=0xdfff8000, c=0, size=4194304)

    at /home/dbezheckov/waratek/tests/asan/sources/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:431

431   ASAN_MEMSET_IMPL(nullptr, block, c, size);

(gdb) step

__asan_region_is_poisoned (beg=4194304, size=4194304)

    at /home/dbezheckov/waratek/tests/asan/sources/llvm/projects/compiler-rt/lib/asan/asan_poisoning.cc:186

186   if (!size) return 0;

(gdb) p size

$6 = 4194304

My question is - why __asan_region_is_poisoned be called with beg = 4194304, why not beg=0x000dfff8000?

[Alexey Samsonov]

Looks like the position of Java heap (0xdfff8000) interferes with ASan shadow memory mappings.

See memory layout in projects/compiler-rt/lib/asan/asan_mapping.h.

__asan_region_is_poisoned arguments seem to be a red herring - it's possible that debug info is incorrect.

--
You received this message because you are subscribed to the Google Groups "address-sanitizer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to address-saniti...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Alexey Samsonov, Mountain View, CA

[Yuri Gribov]

On Fri, Jun 5, 2015 at 9:12 PM, 'Alexey Samsonov' via address-sanitizer <address-...@googlegroups.com> wrote:

Looks like the position of Java heap (0xdfff8000) interferes with ASan shadow memory mappings.

See memory layout in projects/compiler-rt/lib/asan/asan_mapping.h.

__asan_region_is_poisoned arguments seem to be a red herring - it's possible that debug info is incorrect.

Indeed, they would cause a segfault if they were real, contrary to the original error report (unknown-crash at 0x0000dfff8000). It seems that someone (JVM?) spoils ASan shadow memory which later drives ASan crazy.

@Dima: ASan should have printed shadow memory contents after the backtrace, could you post it as well?

-Y

[Yuri Gribov]

If ASan intecepted mmap, it could detect errors like this.

-Y

[Dmitriy -]

How to force ASan to intercept mmap? Or how to understand that it intercepts it?

[Yury Gribov]

On 06/08/2015 09:58 AM, Dmitriy - wrote:
> How to force ASan to intercept mmap? Or how to understand that it
> intercepts it?

You can add a new interceptor in
compiler-rt/lib/asan/asan_interceptors.cc (similar to existing TSan mmap
interceptor in tsan_interceptors.cc). If you are not interested in
hacking compiler-rt, you could simply run your app under strace and look
for mmap syscalls which overlap ASan shadow.

-Y

[Dmitriy -]

Full asan log:

=================================================================

==8250==ERROR: AddressSanitizer: unknown-crash on address 0x0000dfff8000 at pc 0x7ffff6e743d9 bp 0x7ffff0c74960 sp 0x7ffff0c74110

WRITE of size 4194304 at 0x0000dfff8000 thread T1

    #0 0x7ffff6e743d8 in __asan_memset /home/dbezheckov/waratek/tests/asan/sources/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:431

    #1 0x7fffdb4be7d5 in lspace_initialize(GC*, void*, unsigned long) /home/dbezheckov/waratek/harmony-custom/drlvm/vm/gc_gen/src/los/lspace.cpp:44:5

    #2 0x7fffdb4a0c8c in gc_los_initialize(GC_Gen*, void*, unsigned long) /home/dbezheckov/waratek/harmony-custom/drlvm/vm/gc_gen/src/gen/gen.cpp:476:24

    #3 0x7fffdb4a063f in gc_gen_initialize(GC_Gen*, unsigned long, unsigned long) /home/dbezheckov/waratek/harmony-custom/drlvm/vm/gc_gen/src/gen/gen.cpp:313:5

    #4 0x7fffdb4858eb in gc_init /home/dbezheckov/waratek/harmony-custom/drlvm/vm/gc_gen/src/common/gc_for_vm.cpp:104:5

    #5 0x7ffff21ca433 in vm_init1(JavaVM_Internal*, JavaVMInitArgs*, JNIEnv_External**) /home/dbezheckov/waratek/harmony-custom/drlvm/vm/vmcore/src/init/vm_init.cpp:796:14

    #6 0x7ffff2028e4c in JNI_CreateJavaVM /home/dbezheckov/waratek/harmony-custom/drlvm/vm/vmcore/src/jni/jni.cpp:436:19

    #7 0x7ffff20295d4 in CVMI_CreateJavaVM /home/dbezheckov/waratek/harmony-custom/drlvm/vm/vmcore/src/jni/jni.cpp:526:12

    #8 0x7ffff30d0634 in JNI_CreateJavaVM /home/dbezheckov/waratek/harmony-custom/drlvm/vm/openjdk/src/openjdk.cpp:90:12

    #9 0x7ffff6bc6b47  (/home/dbezheckov/waratek/harmony-custom/dist/jdk_7/debug/open/jdk/jre/bin/../lib/amd64/jli/libjli.so+0x2b47)

    #10 0x7ffff63e5181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312

    #11 0x7ffff68f947c in clone /build/buildd/eglibc-2.19/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Address 0x0000dfff8000 is located in the shadow gap area.

SUMMARY: AddressSanitizer: unknown-crash /home/dbezheckov/waratek/tests/asan/sources/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:431 in __asan_memset

Thread T1 created by T0 here:

    #0 0x7ffff6e0257d in __interceptor_pthread_create /home/dbezheckov/waratek/tests/asan/sources/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:225

    #1 0x7ffff6bcd98e  (/home/dbezheckov/waratek/harmony-custom/dist/jdk_7/debug/open/jdk/jre/bin/../lib/amd64/jli/libjli.so+0x998e)

    #2 0x7ffff6bc84f6  (/home/dbezheckov/waratek/harmony-custom/dist/jdk_7/debug/open/jdk/jre/bin/../lib/amd64/jli/libjli.so+0x44f6)

    #3 0x7ffff6bc9043 in JLI_Launch (/home/dbezheckov/waratek/harmony-custom/dist/jdk_7/debug/open/jdk/jre/bin/../lib/amd64/jli/libjli.so+0x5043)

    #4 0x4006b4  (/home/dbezheckov/waratek/harmony-custom/dist/jdk_7/debug/open/jdk/jre/bin/java+0x4006b4)

    #5 0x7ffff6820ec4 in __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287

==8250==ABORTING

"Address 0x0000dfff8000 is located in the shadow gap area"

(gdb) print gc_heap_base_address()

$1 = (void *) 0xdfff8000

That is the address precisely indicates the protected memory segment (heap base in jvm protected by a read/write/execution).

That is, if I use in program mmap function, ASan will not work? Unless of course it is not hacked.

@Yury: nothing shadow memory contents after the backtrace(

[Evgeniy Stepanov]

mmap(... MAP_FIXED ...) will not work if the address is outside of the
ASan application memory area.
Non-fixed mmap is supposed to work.
Is this heap base address a hardcoded constant?