Issue 246 in address-sanitizer: Porting to a new target (AArch64)

[address-...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 246 by christop...@linaro.org: Porting to a new target (AArch64)
http://code.google.com/p/address-sanitizer/issues/detail?id=246

Hello,

I am currently working on enabling address-sanitizer in GCC for AArch64.

I am not sure if this is the right channel to ask for support?

I have made some changes necessary to have it build for this new target
(GetPcSpBp, syscalls, ...) but now I am facing runtime problems:
==15518==ERROR: AddressSanitizer failed to allocate 0xdfff0001000
(15392894357504) bytes at address 0x02008fff7000 (12)
==15518==ReserveShadowMemoryRange failed while trying to map 0xdfff0001000
bytes. Perhaps you're using ulimit -v

In my case, in asan_rtl.cc, full_shadow_is_available is false, and I guess
I have to give sensible values to kMidMemBeg and kMidMemEnd.

How do I compute the right values, and will this fix the error message
above?

Thanks


--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[address-...@googlecode.com]

Updates:
Cc: euge...@google.com ramosia...@gmail.com

Comment #1 on issue 246 by konstant...@gmail.com: Porting to a new target
(AArch64)
http://code.google.com/p/address-sanitizer/issues/detail?id=246

This is *the* right channel.
I've never touched AArch64 before, so I have no idea.
You can start from posting the contents of /proc/self/maps here.

Also, please note, that all work on the asan run-time should be done in llvm
repository first and then it will get merged into gcc.

[address-...@googlecode.com]

Comment #2 on issue 246 by christop...@linaro.org: Porting to a new target
(AArch64)
http://code.google.com/p/address-sanitizer/issues/detail?id=246

Here is the contents of /proc/self/maps (where self == /bin/cat)
# cat /proc/self/maps
00400000-0040b000 r-xp 00000000 fe:02 93
/bin/cat.coreutils
0041a000-0041b000 rwxp 0000a000 fe:02 93
/bin/cat.coreutils
3b799000-3b7ba000 rwxp 00000000 00:00 0
[heap]
7f7ca25000-7f7cb52000 r-xp 00000000 fe:02 611
/lib/libc-2.17-2013.07-2.so
7f7cb52000-7f7cb61000 ---p 0012d000 fe:02 611
/lib/libc-2.17-2013.07-2.so
7f7cb61000-7f7cb65000 r-xp 0012c000 fe:02 611
/lib/libc-2.17-2013.07-2.so
7f7cb65000-7f7cb67000 rwxp 00130000 fe:02 611
/lib/libc-2.17-2013.07-2.so
7f7cb67000-7f7cb6b000 rwxp 00000000 00:00 0
7f7cb6b000-7f7cb86000 r-xp 00000000 fe:02 652
/lib/ld-2.17-2013.07-2.so
7f7cb8b000-7f7cb8c000 rwxp 00000000 00:00 0
7f7cb93000-7f7cb94000 rwxp 00000000 00:00 0
7f7cb94000-7f7cb96000 r-xp 00000000 00:00 0
[vdso]
7f7cb96000-7f7cb97000 r-xp 0001b000 fe:02 652
/lib/ld-2.17-2013.07-2.so
7f7cb97000-7f7cb99000 rwxp 0001c000 fe:02 652
/lib/ld-2.17-2013.07-2.so
7fc44c3000-7fc44e4000 rwxp 00000000 00:00 0
[stack]

I don't work on LLVM, but I can make progress on the GCC side, and forward
you the results when I think it's ready, so that it can be discussed with
the LLVM team, or I'll ask Renato Golin.

Thanks

[address-...@googlecode.com]

Comment #3 on issue 246 by konstant...@gmail.com: Porting to a new target
(AArch64)
http://code.google.com/p/address-sanitizer/issues/detail?id=246

So, the address space is 0..2^39 and otherwise it looks pretty simple.
You need ShadowOffset=2^36 and kHighMemEnd=(2^39)-1 -- please try that.

>> but I can make progress on the GCC side,

Sure. But please don't commit changes tolibsanitizer directly to gcc tree.
They should go via LLVM.

It would be nice if someone can set up a buildbot for LLVM+asan on AArch64

[address-...@googlecode.com]

Comment #4 on issue 246 by christop...@linaro.org: Porting to a new target
(AArch64)
http://code.google.com/p/address-sanitizer/issues/detail?id=246

How do you know that ShadowOffset should be 2^36? It's not obvious to me,
sorry.

As a quick hack, I have forced kHighMemEnd = 0xfffffffffULL in
InitializeHighMemEnd(), and I get a new error:
libsanitizer/sanitizer_common/sanitizer_allocator.h:314 "((kSpaceBeg)) ==
((reinterpret_cast<uptr>(Mprotect(kSpaceBeg, kSpaceSize))))"
(0x600000000000, 0xffffffffffffffff)

(and I set shadow offset in aarch64_asan_shadow_offset to HOST_WIDE_INT_1
<< 36)

[address-...@googlecode.com]

Comment #5 on issue 246 by konstant...@gmail.com: Porting to a new target
(AArch64)
http://code.google.com/p/address-sanitizer/issues/detail?id=246

2^36 is what you get if you divide 2^39 by 8 (the shadow scale factor)
it could be some other smaller value too. (e.g. in x86_64 we use 0x7FFF8000)

>> "((kSpaceBeg)) == ((reinterpret_cast<uptr>(Mprotect(kSpaceBeg,
>> kSpaceSize))))" (0x600000000000, 0xffffffffffffffff)

That's the next failure.
play with kAllocatorSpace and kAllocatorSize in asan/asan_allocator2.cc
It should be somewhere in the upper half of the address space.

[address-...@googlecode.com]

Comment #6 on issue 246 by christop...@linaro.org: Porting to a new target
(AArch64)
http://code.google.com/p/address-sanitizer/issues/detail?id=246

Hmmm..
I have set
const uptr kAllocatorSpace = 0x4000000000ULL;
const uptr kAllocatorSize = 0x4000000000ULL;

and now the programs all end with segfault :-(

I tried to use gdb but it doesn't work, not sure why. It says:
Cannot find new threads: debugger service failed

However I am using a simulator, so it may be causing some problems; in
particular when running (or trying to run) the GCC testsuite, the simulator
always crashes at some point.

Not sure how to make more progress :-(

Thanks.

[address-...@googlecode.com]

Comment #7 on issue 246 by konstant...@gmail.com: Porting to a new target
(AArch64)
http://code.google.com/p/address-sanitizer/issues/detail?id=246

0x7fffffffff -- end of address space
0x4000000000 -- beginning of allocator space
0x8000000000 -- end of allocator space.
This is not going to work :)

try smaller kAllocatorSize first (e.g. 0x2000000000 or 0x1000000000)
Make sure you understand how the allocator works
(sanitizer_common/sanitizer_allocator.h)

If nothing works, switch to using SizeClassAllocator32 instead of
SizeClassAllocator64

[address-...@googlecode.com]

Comment #8 on issue 246 by christop...@linaro.org: Porting to a new target
(AArch64)
http://code.google.com/p/address-sanitizer/issues/detail?id=246

Why do you say this is not going to work? Need to have free space after the
allocator space?

It seems I can't reduce kAllocatorSize; from what I understood I have
kNumClassesRounded=64, and since kRegionSize must be >= 2^32, this means
kAllocatorSize >= 2^38 = 0x4000000000ULL

Since I must have kSpaceBeg % kSpaceSize == 0, this implies kAllocatorSpace
= 0x4000000000ULL.

As of switching to SizeClassAllocator32, I did a quick hack in
sanitizer_internal_defs.h:
#ifdef __aarch64__
#undef SANITIZER_WORDSIZE
#define SANITIZER_WORDSIZE 32
#endif

but then in sanitizer_linux.cc, I had to make sure that
SANITIZER_LINUX_USES_64BIT_SYSCALLS == 1
and in sanitizer_placement_new.h, make sure that
typedef uptr operator_new_ptr_type
for aarch64.

With this, the lib finally compiles, but at runtime:
==1399==AddressSanitizer CHECK failed:
/XXX/libsanitizer/asan/asan_thread.cc:136 "((AddrIsInMem(stack_bottom_))) !=
(0)" (0x0, 0x0)
=================================================================
==1399==ERROR: AddressSanitizer: unknown-crash on address 0x7fc63948d0 at
pc 0x7faa929588 bp 0x7fc6393a70 sp 0x7fc6393aa8
WRITE of size 960 at 0x7fc63948d0 thread T0
==1399==AddressSanitizer CHECK failed:
/XXX/libsanitizer/asan/asan_report.cc:239 "((0 && "Address is not in memory
and not in shadow?")) != (0)" (0x0, 0x0)

[address-...@googlecode.com]

Comment #9 on issue 246 by konstant...@gmail.com: Porting to a new target

(AArch64)
http://code.google.com/p/address-sanitizer/issues/detail?id=246

>> Why do you say this is not going to work? Need to have free space after
>> the allocator space?

Because the addressed like 7fc44c3000 are used for stack and DSOs.

Well, perhaps the 39-bit address space is too small to use our 64-bit
allocator.
See how we use SizeClassAllocator32 in
sanitizer_common/sanitizer_allocator_internal.h on 64-bit.
You'll need similar changes in asan/asan_allocator2.cc
Make sure you check the fresh llvm code base, not gcc.

>> #define SANITIZER_WORDSIZE 32
That's definitely not what you want.

[address-...@googlecode.com]

Updates:
Labels: -Type-Defect Type-Enhancement

Comment #10 on issue 246 by konstant...@gmail.com: Porting to a new target
(AArch64)
http://code.google.com/p/address-sanitizer/issues/detail?id=246

http://llvm.org/viewvc/llvm-project?view=revision&revision=198044
should give you what you need for asan's allocator.
Just define SANITIZER_CAN_USE_ALLOCATOR64 to 0

[christo...@linaro.org]

Hello,

I have finally a working patch for GCC, for which I had to merged LLVM changes from rev 198044 as you suggested.

I've had someone run tests on actual AArch64 hardware (I don't have access myself), as it's not currently practical to validate under simulators. The Foundation Model has some time out problems as well as give random errors; QEMU goes in infinite loop when unwinding the null-deref-1 test (which does work in the Foundation Model and on hardware).

Some of the ubsan tests cause the compiler to crash, but I think that should be addressed separately.

Not sure how I can attach my patch here, so that you can look at the libsanitizer part?

Thanks.

[Konstantin Serebryany]

Hi Christophe, 

Great to hear you have a working patch. 

All libsanitizer code should go through the LLVM tree first. 

Please follow https://code.google.com/p/address-sanitizer/wiki/HowToContribute

Once the patch is in LLVM, it will eventually reach GCC. 

Please note: we can not guarantee that AArch64 support will remain in working conditions

unless you set up a public build bot -- we'd appreciate if you can do this too, 

once the machines are more widely available. 

http://llvm.org/docs/HowToAddABuilder.html

On Fri, Feb 7, 2014 at 7:21 PM, <christo...@linaro.org> wrote:

Hello,

I have finally a working patch for GCC, for which I had to merged LLVM changes from rev 198044 as you suggested.

I've had someone run tests on actual AArch64 hardware (I don't have access myself), as it's not currently practical to validate under simulators. The Foundation Model has some time out problems as well as give random errors; QEMU goes in infinite loop when unwinding the null-deref-1 test (which does work in the Foundation Model and on hardware).

Some of the ubsan tests cause the compiler to crash, but I think that should be addressed separately.

Correct. 

For ubsan run-time please contact llv...@cs.uiuc.edu

--kcc 

 

Not sure how I can attach my patch here, so that you can look at the libsanitizer part?

Thanks.

--
You received this message because you are subscribed to the Google Groups "address-sanitizer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to address-saniti...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[address-...@googlecode.com]

Comment #11 on issue 246 by svha...@gmail.com: Porting to a new target
(AArch64)
https://code.google.com/p/address-sanitizer/issues/detail?id=246

Please see http://gcc.gnu.org/PR64435
Aarch64 doesn't have always 39-bit address space, but configurably either
39, 42 or 47 bit address space.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64435#c19 contains a partial
patch for that, which lets at least the
LowMem/LowShadow/ShadowGap/HighShadow/HighMem division be runtime decided
to support all 3. Unfortunately the 39-bit address space is quite small
and it is hard to stick in the allocator64 at a fixed address, and 32-bit
allocator doesn't really work with 42-bit addres space (assertion failure).

[address-...@googlecode.com]

Comment #12 on issue 246 by konstant...@gmail.com: Porting to a new target
(AArch64)
https://code.google.com/p/address-sanitizer/issues/detail?id=246

svhavel@,
Let's fix the assertion in Allocator32.
All patches *must* be sent upstream, we can not accept patches from gcc
mailing lists.
https://code.google.com/p/address-sanitizer/wiki/HowToContribute

Alternatively, we can decide that we support sanitizers only with 47-bit AS.
This will simplify lots of things.

[address-...@googlecode.com]

Comment #13 on issue 246 by svha...@gmail.com: Porting to a new target
(AArch64)
https://code.google.com/p/address-sanitizer/issues/detail?id=246

I can send the current patch there, sure.
I don't think anybody ships with 47-bit AS right now, so requiring only
that would be likely equal to not supporting it at all.

[address-...@googlecode.com]

Comment #14 on issue 246 by svha...@gmail.com: Porting to a new target
(AArch64)
https://code.google.com/p/address-sanitizer/issues/detail?id=246

So, if we keep using Allocator32, that would supposedly mean we have to
revert the
// The range of addresses which can be returned my mmap.
// FIXME: this value should be different on different platforms,
// e.g. on AArch64 it is most likely (1ULL << 39). Larger values will still
work
// but will consume more memory for TwoLevelByteMap.
#if defined(__aarch64__)
# define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL <<
39)
#else
# define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL <<
47)
#endif
change and use 1ULL << 47 even on aarch64? What implications does it have
exactly (i.e. how much more memory it will consume?).

Or, what is the minimum usable size for Allocator64? In case of 39-bit
address space it is tight and we definitely can't find terrabytes of memory
anywhere. 42-bit and 47-bit AS has space for it, but the question is if
the space will be always at the same base.

[address-...@googlecode.com]

Comment #15 on issue 246 by christop...@linaro.org: Porting to a new target
(AArch64)
https://code.google.com/p/address-sanitizer/issues/detail?id=246

For the record, the asan tests passed on a recent GCC trunk, on a AArch64
juno board, running with 39-bits address space. That's the only HW I have
access to at the moment.

[Venkataramanan Kumar]

The machine I have running 42 bit address space. The test are failing

FAIL: c-c++-common/asan/misalign-2.c -O1 output pattern test, is
==13754==AddressSanitizer CHECK failed:
../../../../gcc/libsanitizer/asan/asan_poisoning.cc:24
"((AddrIsInMem(addr))) != (0)" (0x0, 0x0)
,

[root@localhost libsanitizer]# cat /proc/self/maps
00400000-00410000 r-xp 00000000 08:04 16881937
/usr/bin/cat
00410000-00420000 r--p 00000000 08:04 16881937
/usr/bin/cat
00420000-00430000 rw-p 00010000 08:04 16881937
/usr/bin/cat
00430000-00460000 rw-p 00000000 00:00 0 [heap]
3ffaf940000-3ffb61e0000 r--p 00000000 08:04 33597753
/usr/lib/locale/locale-archive
3ffb61e0000-3ffb6340000 r-xp 00000000 08:04 33597006
/usr/lib64/libc-2.17.so
3ffb6340000-3ffb6350000 r--p 00150000 08:04 33597006
/usr/lib64/libc-2.17.so
3ffb6350000-3ffb6360000 rw-p 00160000 08:04 33597006
/usr/lib64/libc-2.17.so
3ffb6370000-3ffb6380000 r--p 00000000 00:00 0 [vvar]
3ffb6380000-3ffb6390000 r-xp 00000000 00:00 0 [vdso]
3ffb6390000-3ffb63b0000 r-xp 00000000 08:04 33596999
/usr/lib64/ld-2.17.so
3ffb63b0000-3ffb63c0000 r--p 00010000 08:04 33596999
/usr/lib64/ld-2.17.so
3ffb63c0000-3ffb63d0000 rw-p 00020000 08:04 33596999
/usr/lib64/ld-2.17.so
3ffea3b0000-3ffea3e0000 rw-p 00000000 00:00 0 [stack]

> You received this message because you are subscribed to the Google Groups
> "address-sanitizer" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to address-saniti...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

[address-...@googlecode.com]

Comment #16 on issue 246 by svha...@gmail.com: Porting to a new target
(AArch64)
https://code.google.com/p/address-sanitizer/issues/detail?id=246

FYI, following patch makes ASAN work fine on 42-bit AS, but will break the
39-bit AS and won't fix 48-bit one. So we really need something more
dynamic.

--- libsanitizer/asan/asan_allocator.h (revision 219833)
+++ libsanitizer/asan/asan_allocator.h (working copy)
@@ -100,6 +100,10 @@
# if defined(__powerpc64__)
const uptr kAllocatorSpace = 0xa0000000000ULL;
const uptr kAllocatorSize = 0x20000000000ULL; // 2T.
+# elif defined(__aarch64__)
+// Valid only for 42-bit VA
+const uptr kAllocatorSpace = 0x10000000000ULL;
+const uptr kAllocatorSize = 0x10000000000ULL; // 1T.
# else
const uptr kAllocatorSpace = 0x600000000000ULL;
const uptr kAllocatorSize = 0x40000000000ULL; // 4T.
--- libsanitizer/sanitizer_common/sanitizer_platform.h (revision 219833)
+++ libsanitizer/sanitizer_common/sanitizer_platform.h (working copy)
@@ -79,7 +79,7 @@
// For such platforms build this code with
-DSANITIZER_CAN_USE_ALLOCATOR64=0 or
// change the definition of SANITIZER_CAN_USE_ALLOCATOR64 here.
#ifndef SANITIZER_CAN_USE_ALLOCATOR64
-# if defined(__aarch64__) || defined(__mips64)
+# if defined(__mips64)
# define SANITIZER_CAN_USE_ALLOCATOR64 0
# else
# define SANITIZER_CAN_USE_ALLOCATOR64 (SANITIZER_WORDSIZE == 64)
@@ -88,10 +88,10 @@

// The range of addresses which can be returned my mmap.
// FIXME: this value should be different on different platforms,

-// e.g. on AArch64 it is most likely (1ULL << 39). Larger values will
still work
+// e.g. on AArch64 it is most likely (1ULL << 42). Larger values will

still work
// but will consume more memory for TwoLevelByteMap.
#if defined(__aarch64__)

-# define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL <<
39)
+# define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL <<
42)

#else
# define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL <<
47)
#endif

[Venkataramanan Kumar]

Hi I tried using patch suggested on aarch64 machine which is 42 VA.

In addition, I set shadow offset as below in asan_mapping.h
static const u64 kAArch64_ShadowOffset64 = 1ULL << 39;

ASAN builds but when I do make check-asan, I get lot of failures.

Expected Passes : 40
Expected Failures : 6
Unsupported Tests : 129
Unexpected Failures: 135

The errors are looking like this

Command Output (stderr):
--
==32641==AddressSanitizer: failed to intercept '__isoc99_printf'
==32641==AddressSanitizer: failed to intercept '__isoc99_sprintf'
==32641==AddressSanitizer: failed to intercept '__isoc99_snprintf'
==32641==AddressSanitizer: failed to intercept '__isoc99_fprintf'
==32641==AddressSanitizer: failed to intercept '__isoc99_vprintf'
==32641==AddressSanitizer: failed to intercept '__isoc99_vsprintf'
==32641==AddressSanitizer: failed to intercept '__isoc99_vsnprintf'
==32641==AddressSanitizer: failed to intercept '__isoc99_vfprintf'
==32641==AddressSanitizer: libc interceptors initialized
|| `[0x010000000000, 0x03ffffffffff]` || HighMem ||
|| `[0x00a000000000, 0x00ffffffffff]` || HighShadow ||
|| `[0x009000000000, 0x009fffffffff]` || ShadowGap ||
|| `[0x008000000000, 0x008fffffffff]` || LowShadow ||
|| `[0x000000000000, 0x007fffffffff]` || LowMem ||
MemToShadow(shadow): 0x009000000000 0x0091ffffffff 0x009400000000 0x009fffffffff
redzone=16
max_redzone=2048
quarantine_size_mb=256M
malloc_context_size=30
SHADOW_SCALE: 3
SHADOW_GRANULARITY: 8
SHADOW_OFFSET: 8000000000
==32641==Installed the sigaction for signal 11
==32641==Installed the sigaction for signal 7
==32641==T0: stack [0x03ffdd280000,0x03ffdda80000) size 0x800000;
local=0x03ffdda75ef0
==32641==AddressSanitizer Init done
ASAN:SIGSEGV
=================================================================
==32641==ERROR: AddressSanitizer: SEGV on unknown address
0x007ffbb4ebc0 (pc 0x0000004c1e1c bp 0x03ffdda75ec0 sp 0x03ffdda75e00
T0)
#0 0x4c1e1b in main
/home/venkataramanan.kumar/LLVM/TSAN_port/compiler-rt/test/asan/TestCases/Linux/coverage-disabled.cc:18
#1 0x3ffaa1b0c93 in __libc_start_main (/lib64/libc.so.6+0x20c93)
#2 0x41b1f3 in _start
(/home/venkataramanan.kumar/LLVM/TSAN_port/Build/projects/compiler-rt/test/asan/AARCH64LinuxConfig/TestCases/Linux/Output/coverage-disabled.cc.tmp+0x41b1f3)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV
/home/venkataramanan.kumar/LLVM/TSAN_port/compiler-rt/test/asan/TestCases/Linux/coverage-disabled.cc:18
main
==32641==ABORTING

No idea what is going wrong? Am I missing something here?

regards,
Venkat,

[Konstantin Serebryany]

Hard to tell w/o being able to reproduce. I can work as a remote
debugger for you, if you wish.
Let's start from something more trivial than coverage-disabled.cc
(e.g. start from a "hello world") and run it under gdb.

[Venkataramanan Kumar]

Hi Konstantin,


On 19 March 2015 at 23:25, Konstantin Serebryany

OK. Let me start with this test case.

#include <stdio.h>

int main()
{
printf("Hello World\n");
return 0;
}

/home/venkataramanan.kumar/
LLVM/TSAN_port/Build/./bin/clang
--driver-mode=g++ -fsanitize=address test.c

[venkataramanan.kumar@amd-01 ~]$ ./a.out
ASAN:SIGSEGV
=================================================================
==25303==ERROR: AddressSanitizer: SEGV on unknown address
0x007fff96387c (pc 0x0000004c1dfc bp 0x03fffcb1c460 sp 0x03fffcb1c3e0
T0)
#0 0x4c1dfb (/home/venkataramanan.kumar/a.out+0x4c1dfb)
#1 0x3ff9c790c93 (/lib64/libc.so.6+0x20c93)
#2 0x41b1f3 (/home/venkataramanan.kumar/a.out+0x41b1f3)

AddressSanitizer can not provide additional info.

==25303==ABORTING


Now I am running under gdb

[venkataramanan.kumar@amd-01 ~]$ gdb ./a.out
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-62.sa1.4
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/venkataramanan.kumar/a.out...done.
(gdb) r
Starting program: /home/venkataramanan.kumar/./a.out
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Hello World
[Inferior 1 (process 25306) exited normally]
Missing separate debuginfos, use: debuginfo-install
glibc-2.17-68.sa1.4.aarch64 libgcc-4.8.3-8.sa1.4.aarch64
libstdc++-4.8.3-8.sa1.4.aarch64

It ran to completion.


ASAN:SIGSEGV
==============================
===================================
==6221==ERROR: AddressSanitizer: SEGV on unknown address
0x007ffd1329f4 (pc 0x0000004c1e64 bp 0x03ffe8995090 sp 0x03ffe8994fa0
T0)
#0 0x4c1e63 in main
/home/venkataramanan.kumar/LLVM/TSAN_port/compiler-rt/test/asan/TestCases/Linux/coverage-direct-large.cc:52
#1 0x3ffb2730c93 in __libc_start_main (/lib64/libc.so.6+0x20c93)

What this error mean?
Stack address 0x03ffe8994fa0 when I do 0x03ffe8994fa0 >> 0x3, I get
this address 0x007ffd1329f4,

"address 0x007ffd1329f4 is illegal. Why this address is accessed?

Between my cat /proc/self/maps look like that.

venataramanan.kumar@amd-01 ~]$ cat /proc/self/maps
00400000-00410000 r-xp 00000000 08:04 16890529
/usr/bin/cat
00410000-00420000 r--p 00000000 08:04 16890529
/usr/bin/cat
00420000-00430000 rw-p 00010000 08:04 16890529
/usr/bin/cat
07a30000-07a60000 rw-p 00000000 00:00 0 [heap]
3ff85ca0000-3ff8c540000 r--p 00000000 08:04 34089040
/usr/lib/locale/locale-archive
3ff8c540000-3ff8c6a0000 r-xp 00000000 08:04 33642757
/usr/lib64/libc-2.17.so
3ff8c6a0000-3ff8c6b0000 r--p 00150000 08:04 33642757
/usr/lib64/libc-2.17.so
3ff8c6b0000-3ff8c6c0000 rw-p 00160000 08:04 33642757
/usr/lib64/libc-2.17.so
3ff8c6d0000-3ff8c6e0000 r--p 00000000 00:00 0 [vvar]
3ff8c6e0000-3ff8c6f0000 r-xp 00000000 00:00 0 [vdso]
3ff8c6f0000-3ff8c710000 r-xp 00000000 08:04 33642750
/usr/lib64/ld-2.17.so
3ff8c710000-3ff8c720000 r--p 00010000 08:04 33642750
/usr/lib64/ld-2.17.so
3ff8c720000-3ff8c730000 rw-p 00020000 08:04 33642750
/usr/lib64/ld-2.17.so
3ffc0d80000-3ffc0db0000 rw-p 00000000 00:00 0 [stack]


regards,
Venkat.

[Konstantin Serebryany]

Interesting. So, w/ gdb the test passes, w/o gdb it fails.
Try few things:

Run w/o gdb but with ASLR off (is there ASLR on AArch64?). Use setarch for that.

Run w/ gdb but with ASLR on (disable-randomization off). If you get a
crash with gdb, type 'dis' and send the result here.

build the following code with -S and with '-S -o - -emit-llvm' and
send the results:
void foo(long *x) { *x = 0; }

--kcc

[Venkataramanan Kumar]

Hi Konstantin,

The machine I was working is down now. I am waiting for someone in
linaro lab to turn it on.

Now I used another machine and with this one I am not getting segfault
for the simple "hello world" test case.

However ASAN tests continued to fail with same errors.

An example
<stdin>:3:1: note: possible intended match here
==4793==ERROR: AddressSanitizer: SEGV on unknown address
0x007ffd9552d8 (pc 0x0000004c41e8 bp 0x03ffecaa9760 sp 0x03ffecaa96c0
T0)
^


Below is the LLVM IR for the test case you gave.

(------Snip----)
; ModuleID = 'try.c'
target datalayout = "e-m:e-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-unknown-linux-gnu"

@llvm.global_ctors = appending global [1 x { i32, void ()* }] [{ i32,
void ()* } { i32 1, void ()* @asan.module_ctor }]

; Function Attrs: nounwind sanitize_address uwtable
define void @_Z3fooPl(i64* %x) #0 {
entry:
%x.addr = alloca i64*, align 8
store i64* %x, i64** %x.addr, align 8
%0 = load i64*, i64** %x.addr, align 8
%1 = ptrtoint i64* %0 to i64
%2 = lshr i64 %1, 3
%3 = or i64 %2, 68719476736
%4 = inttoptr i64 %3 to i8*
%5 = load i8, i8* %4
%6 = icmp ne i8 %5, 0
br i1 %6, label %7, label %8

; <label>:7 ; preds = %entry
call void @__asan_report_store8(i64 %1)
call void asm sideeffect "", ""()
unreachable

; <label>:8 ; preds = %entry
store i64 0, i64* %0, align 8
ret void
}

define internal void @asan.module_ctor() {
call void @__asan_init_v5()
ret void
}

declare void @__asan_init_v5()

declare void @__asan_report_load_n(i64, i64)

declare void @__asan_loadN(i64, i64)

declare void @__asan_report_load1(i64)

declare void @__asan_load1(i64)

declare void @__asan_report_load2(i64)

declare void @__asan_load2(i64)

declare void @__asan_report_load4(i64)

declare void @__asan_load4(i64)

declare void @__asan_report_load8(i64)

declare void @__asan_load8(i64)

declare void @__asan_report_load16(i64)

declare void @__asan_load16(i64)

declare void @__asan_report_store_n(i64, i64)

declare void @__asan_storeN(i64, i64)

declare void @__asan_report_store1(i64)

declare void @__asan_store1(i64)

declare void @__asan_report_store2(i64)

declare void @__asan_store2(i64)

declare void @__asan_report_store4(i64)

declare void @__asan_store4(i64)

declare void @__asan_report_store8(i64)

declare void @__asan_store8(i64)

declare void @__asan_report_store16(i64)

declare void @__asan_store16(i64)

declare void @__asan_report_exp_load_n(i64, i64, i32)

declare void @__asan_exp_loadN(i64, i64, i32)

declare void @__asan_report_exp_load1(i64, i32)

declare void @__asan_exp_load1(i64, i32)

declare void @__asan_report_exp_load2(i64, i32)

declare void @__asan_exp_load2(i64, i32)

declare void @__asan_report_exp_load4(i64, i32)

declare void @__asan_exp_load4(i64, i32)

declare void @__asan_report_exp_load8(i64, i32)

declare void @__asan_exp_load8(i64, i32)

declare void @__asan_report_exp_load16(i64, i32)

declare void @__asan_exp_load16(i64, i32)

declare void @__asan_report_exp_store_n(i64, i64, i32)

declare void @__asan_exp_storeN(i64, i64, i32)

declare void @__asan_report_exp_store1(i64, i32)

declare void @__asan_exp_store1(i64, i32)

declare void @__asan_report_exp_store2(i64, i32)

declare void @__asan_exp_store2(i64, i32)

declare void @__asan_report_exp_store4(i64, i32)

declare void @__asan_exp_store4(i64, i32)

declare void @__asan_report_exp_store8(i64, i32)

declare void @__asan_exp_store8(i64, i32)

declare void @__asan_report_exp_store16(i64, i32)

declare void @__asan_exp_store16(i64, i32)

declare i8* @__asan_memmove(i8*, i8*, i64)

declare i8* @__asan_memcpy(i8*, i8*, i64)

declare i8* @__asan_memset(i8*, i32, i64)

declare void @__asan_handle_no_return()

declare void @__sanitizer_ptr_cmp(i64, i64)

declare void @__sanitizer_ptr_sub(i64, i64)

declare void @__asan_before_dynamic_init(i64)

declare void @__asan_after_dynamic_init()

declare void @__asan_register_globals(i64, i64)

declare void @__asan_unregister_globals(i64, i64)

attributes #0 = { nounwind sanitize_address uwtable
"less-precise-fpmad"="false" "no-frame-pointer-elim"="true"
"no-frame-pointer-elim-non-leaf" "no-infs-fp-math"="false"
"no-nans-fp-math"="false" "stack-protector-buffer-size"="8"
"target-cpu"="generic" "target-features"="+neon"
"unsafe-fp-math"="false" "use-soft-float"="false" }

!llvm.ident = !{!0}

!0 = !{!"clang version 3.7.0 (http://llvm.org/git/clang.git
f3f115679db522ca44f75e3dadf3cdef8679b928)
(http://llvm.org/git/llvm.git
048ca17f6ea4386f608eea62ca716466bf9f674c)"}
(-----Snip------)

%2 = lshr i64 %1, 3
%3 = or i64 %2, 68719476736
%4 = inttoptr i64 %3 to i8*

I am trying to understand why 68719476736 (0x1000000000) has been
ored with stack here.


On 20 March 2015 at 21:27, Konstantin Serebryany

regards,
Venkat.

[Venkataramanan Kumar]

Hi,

Slightly changed the hello world test case to have a local variable

#include <stdio.h>
int main()
{
int a[10]={0};

printf("Hello World\n");

return a[1];
}


With ASLR "on" I get

ASAN:SIGSEGV
=================================================================
==12464==ERROR: AddressSanitizer: SEGV on unknown address
0x007ff92af4ac (pc 0x0000004c41f4 bp 0x03ffc957a650 sp 0x03ffc957a560
T0)
#0 0x4c41f3 (/root/work/vekumar/ASAN/build-asan-64bit/a.out+0x4c41f3)
#1 0x3ffaacd0c93 (/lib64/libc.so.6+0x20c93)
#2 0x41b7ab (/root/work/vekumar/ASAN/build-asan-64bit/a.out+0x41b7ab)

AddressSanitizer can not provide additional info.

SUMMARY: AddressSanitizer: SEGV
(/root/work/vekumar/ASAN/build-asan-64bit/a.out+0x4c41f3)

setarch linux64 -R ./a.out
Hello world


On 22 March 2015 at 19:59, Venkataramanan Kumar

[Venkataramanan Kumar]

Hi Konstantin,

Another case for heap reuse.

int main() {
char *x = new char[20];
delete[] x;
return x[0];
}

ASAN:SIGSEGV
=================================================================
==6524==ERROR: AddressSanitizer: SEGV on unknown address
0x003180001dfc (pc 0x0000004c4188 bp 0x03fffffff4c0 sp 0x03fffffff490
T0)
#0 0x4c4187 (/root/work/vekumar/ASAN/build-asan-64bit/a.out+0x4c4187)
#1 0x3ffb7b90c93 (/lib64/libc.so.6+0x20c93)

#2 0x41b7ab (/root/work/vekumar/ASAN/build-asan-64bit/a.out+0x41b7ab)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV

(/root/work/vekumar/ASAN/build-asan-64bit/a.out+0x4c4187)
==6524==ABORTING

Snip from assembly

(--snip--)
.LBB0_2: // %delete.end
ldur x8, [x29, #-16]
lsr x9, x8, #3
orr x9, x9, #0x1000000000
ldrb w10, [x9]
(--snip--)

I manually changed it to

(--snip--)
.LBB0_2: // %delete.end
ldur x8, [x29, #-16]
lsr x9, x8, #3
orr x9, x9, #0x8000000000 <== shadow offset
ldrb w10, [x9]
(--snip--)

Now I am able to get ASAN report.

[root@SQA-Overdrive02-010236011075 build-asan-64bit]# ./a.out
=================================================================
==6548==ERROR: AddressSanitizer: heap-use-after-free on address
0x010c0000efe0 at pc 0x0000004c41c8 bp 0x03fffffff470 sp
0x03fffffff488
READ of size 1 at 0x010c0000efe0 thread T0
#0 0x4c41c7 (/root/work/vekumar/ASAN/build-asan-64bit/a.out+0x4c41c7)
#1 0x3ffb7b90c93 (/lib64/libc.so.6+0x20c93)
#2 0x41b7ab (/root/work/vekumar/ASAN/build-asan-64bit/a.out+0x41b7ab)

0x010c0000efe0 is located 0 bytes inside of 20-byte region
[0x010c0000efe0,0x010c0000eff4)
freed by thread T0 here:
#0 0x4c3b63 (/root/work/vekumar/ASAN/build-asan-64bit/a.out+0x4c3b63)
#1 0x4c417b (/root/work/vekumar/ASAN/build-asan-64bit/a.out+0x4c417b)
#2 0x3ffb7b90c93 (/lib64/libc.so.6+0x20c93)
#3 0x41b7ab (/root/work/vekumar/ASAN/build-asan-64bit/a.out+0x41b7ab)


Looks like shadow offset is still set as 1<<36.

This is my patch

[root@SQA-Overdrive02-010236011075 compiler-rt]# git diff
diff --git a/cmake/config-ix.cmake b/cmake/config-ix.cmake
index 1207f4c..71e0646 100644
--- a/cmake/config-ix.cmake
+++ b/cmake/config-ix.cmake
@@ -222,7 +222,7 @@ endfunction()
filter_available_targets(SANITIZER_COMMON_SUPPORTED_ARCH
x86_64 i386 i686 powerpc64 powerpc64le arm aarch64 mips mips64 mipsel mips64e
filter_available_targets(ASAN_SUPPORTED_ARCH
- x86_64 i386 i686 powerpc64 powerpc64le arm mips mipsel mips64 mips64el)
+ x86_64 i386 i686 powerpc64 powerpc64le arm aarch64 mips mipsel mips64 mips64e
filter_available_targets(DFSAN_SUPPORTED_ARCH x86_64 mips64 mips64el)
filter_available_targets(LSAN_SUPPORTED_ARCH x86_64 mips64 mips64el)
# LSan common files should be available on all architectures supported
diff --git a/lib/asan/asan_allocator.h b/lib/asan/asan_allocator.h
index 3208d1f..b46d4d2 100644
--- a/lib/asan/asan_allocator.h
+++ b/lib/asan/asan_allocator.h
@@ -115,6 +115,9 @@ struct AsanMapUnmapCallback {

# if defined(__powerpc64__)
const uptr kAllocatorSpace = 0xa0000000000ULL;
const uptr kAllocatorSize = 0x20000000000ULL; // 2T.
+# elif defined(__aarch64__)

+const uptr kAllocatorSpace = 0x10000000000ULL;

+const uptr kAllocatorSize = 0x10000000000ULL; // 2T.

# else
const uptr kAllocatorSpace = 0x600000000000ULL;
const uptr kAllocatorSize = 0x40000000000ULL; // 4T.

diff --git a/lib/asan/asan_mapping.h b/lib/asan/asan_mapping.h
index 5cb011d..87f99d8 100644
--- a/lib/asan/asan_mapping.h
+++ b/lib/asan/asan_mapping.h
@@ -101,7 +101,7 @@ static const u64 kDefaultShadowOffset32 = 1ULL << 29; // 0x
static const u64 kIosShadowOffset32 = 1ULL << 30; // 0x40000000
static const u64 kDefaultShadowOffset64 = 1ULL << 44;
static const u64 kDefaultShort64bitShadowOffset = 0x7FFF8000; // < 2G.
-static const u64 kAArch64_ShadowOffset64 = 1ULL << 36;
+static const u64 kAArch64_ShadowOffset64 = 1ULL << 39;
static const u64 kMIPS32_ShadowOffset32 = 0x0aaa0000;
static const u64 kMIPS64_ShadowOffset64 = 1ULL << 37;
static const u64 kPPC64_ShadowOffset64 = 1ULL << 41;
diff --git a/lib/sanitizer_common/sanitizer_platform.h b/lib/sanitizer_common/sa
index fef5a5b..7dbd956 100644
--- a/lib/sanitizer_common/sanitizer_platform.h
+++ b/lib/sanitizer_common/sanitizer_platform.h
@@ -81,7 +81,7 @@

// For such platforms build this code with -DSANITIZER_CAN_USE_ALLOCATOR64=0 or
// change the definition of SANITIZER_CAN_USE_ALLOCATOR64 here.
#ifndef SANITIZER_CAN_USE_ALLOCATOR64
-# if defined(__aarch64__) || defined(__mips64)
+# if defined(__mips64)
# define SANITIZER_CAN_USE_ALLOCATOR64 0
# else
# define SANITIZER_CAN_USE_ALLOCATOR64 (SANITIZER_WORDSIZE == 64)

@@ -93,7 +93,7 @@

// e.g. on AArch64 it is most likely (1ULL << 39). Larger values will still wor

// but will consume more memory for TwoLevelByteMap.
#if defined(__aarch64__)
-# define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL << 39)
+# define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL << 42)

#elif defined(__mips__)
# define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL << 40)
#else

Am I missing something?

regards,
Venkat.

On 22 March 2015 at 20:33, Venkataramanan Kumar

[Venkataramanan Kumar]

Ok I missed to add here.

diff --git a/lib/Transforms/Instrumentation/AddressSanitizer.cpp b/lib/Transform
index 6dfc3fe..4465e49 100644
--- a/lib/Transforms/Instrumentation/AddressSanitizer.cpp
+++ b/lib/Transforms/Instrumentation/AddressSanitizer.cpp
@@ -69,7 +69,7 @@ static const uint64_t kSmallX86_64ShadowOffset = 0x7FFF8000;
static const uint64_t kPPC64_ShadowOffset64 = 1ULL << 41;
static const uint64_t kMIPS32_ShadowOffset32 = 0x0aaa0000;
static const uint64_t kMIPS64_ShadowOffset64 = 1ULL << 37;
-static const uint64_t kAArch64_ShadowOffset64 = 1ULL << 36;
+static const uint64_t kAArch64_ShadowOffset64 = 1ULL << 39;
static const uint64_t kFreeBSD_ShadowOffset32 = 1ULL << 30;
static const uint64_t kFreeBSD_ShadowOffset64 = 1ULL << 46;
static const uint64_t kWindowsShadowOffset32 = 3ULL << 28;


Now with that I have only 1 test case failure.

-- Testing: 557 tests, 4 threads --
Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.
FAIL: AddressSanitizer-aarch64-linux :: TestCases/throw_invoke_test.cc
(542 of 557)
******************** TEST 'AddressSanitizer-aarch64-linux ::
TestCases/throw_invoke_test.cc' FAILED ********************
Script:
--
/root/work/vekumar/ASAN/build-asan-64bit/./bin/clang --driver-mode=g++
-fsanitize=address -mno-omit-leaf-frame-pointer
-fno-omit-frame-pointer -fno-optimize-sibling-calls -gline-tables-only
/root/work/vekumar/ASAN/source/llvm/projects/compiler-rt/test/asan/TestCases/throw_invoke_test.cc
-o /root/work/vekumar/ASAN/build-asan-64bit/projects/compiler-rt/test/asan/AARCH64LinuxConfig/TestCases/Output/throw_invoke_test.cc.tmp
&& /root/work/vekumar/ASAN/build-asan-64bit/projects/compiler-rt/test/asan/AARCH64LinuxConfig/TestCases/Output/throw_invoke_test.cc.tmp
/root/work/vekumar/ASAN/build-asan-64bit/./bin/clang --driver-mode=g++
-fsanitize=address -mno-omit-leaf-frame-pointer
-fno-omit-frame-pointer -fno-optimize-sibling-calls -gline-tables-only
/root/work/vekumar/ASAN/source/llvm/projects/compiler-rt/test/asan/TestCases/throw_invoke_test.cc
-o /root/work/vekumar/ASAN/build-asan-64bit/projects/compiler-rt/test/asan/AARCH64LinuxConfig/TestCases/Output/throw_invoke_test.cc.tmp
-static-libstdc++ &&
/root/work/vekumar/ASAN/build-asan-64bit/projects/compiler-rt/test/asan/AARCH64LinuxConfig/TestCases/Output/throw_invoke_test.cc.tmp
--
Exit Code: 1

Command Output (stderr):
--
Throw stack = 0x3fffffff900
ReallyThrow
a = 42
CheckStack stack = 0x3fffffff7a0, 0x3fffffff930
/usr/bin/ld: cannot find -lstdc++
clang-3.7: error: linker command failed with exit code 1 (use -v to
see invocation)

--

********************
Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
Testing Time: 564.64s
********************
Failing Tests (1):
AddressSanitizer-aarch64-linux :: TestCases/throw_invoke_test.cc

Expected Passes : 427
Expected Failures : 6
Unsupported Tests : 123
Unexpected Failures: 1
make[3]: *** [projects/compiler-rt/test/asan/CMakeFiles/check-asan] Error 1
make[2]: *** [projects/compiler-rt/test/asan/CMakeFiles/check-asan.dir/all]
Error 2
make[1]: *** [projects/compiler-rt/test/asan/CMakeFiles/check-asan.dir/rule]
Error 2

regards,
Venkat,

On 23 March 2015 at 14:28, Venkataramanan Kumar

[Venkataramanan Kumar]

Hi konstantin,


is there a way to initialize ASAN mapping for Aarch64 during runtime ?

I see a code in ASAN which initializes "kHighMemEnd" at runtime.

static void InitializeHighMemEnd() {
#if !ASAN_FIXED_MAPPING
kHighMemEnd = GetMaxVirtualAddress();
// Increase kHighMemEnd to make sure it's properly
// aligned together with kHighMemBeg:
kHighMemEnd |= SHADOW_GRANULARITY * GetPageSizeCached() - 1;
#endif // !ASAN_FIXED_MAPPING
CHECK_EQ((kHighMemBeg % GetPageSizeCached()), 0);
}

is it possible to extend this for low memory, shadow offset etc for
aarch64 42 bit VA case so that 39 bit VA and 42 bit VA can coexist.


Also these value in asan_allocator.h has to be set at runtime.

const uptr kAllocatorSpace = 0x10000000000ULL;

const uptr kAllocatorSize = 0x10000000000ULL; // 2T.

Another method would be to set AARCH64_VA_SIZE. This will be detected
at configure time and based on the add the required mappings.

are there any better approach ?

please advise.

regards,
Venkat.




On 23 March 2015 at 16:29, Venkataramanan Kumar

[Konstantin Serebryany]

On Tue, Mar 24, 2015 at 5:14 AM, Venkataramanan Kumar
<venkataram...@linaro.org> wrote:
> Hi konstantin,
>
>
> is there a way to initialize ASAN mapping for Aarch64 during runtime ?

This *is* happening at run-time.
Did you talk to others from Linaro who already reported to have
working asan on aarch64?
Renato?

[Konstantin Serebryany]

On Wed, Mar 25, 2015 at 4:33 AM, Renato Golin <renato...@linaro.org> wrote:
> On 24 March 2015 at 15:46, Konstantin Serebryany

> <konstantin....@gmail.com> wrote:
>>> is there a way to initialize ASAN mapping for Aarch64 during runtime ?
>>
>> This *is* happening at run-time.
>> Did you talk to others from Linaro who already reported to have
>> working asan on aarch64?
>> Renato?
>

> Hi Kostya,
>
> There are multiple issues here...
>
> Venkat was working on TSAN for AArch64 and got caught in a virtual
> address problem. Not only AArch64 has less virtual address bits (39 or
> 42, as opposed to 48 in x86_64),

Is it only 39 and 42, or is there a 3-rd option?


> but they're different and there is no
> generic solution. That's when he started working on ASAN to get the
> 64-bit allocator working, and the virtual address space became an
> issue once again. In the end, for the sanitizers to work well on all
> AArch64 variations, we need to solve the virtual address issue.
>
> IIRC, ASAN has some macros to define the address ranges, so at least
> part of the ranges logic is not runtime based, right?

The values that define the location of the shadow are initialized at startup,
so it should not be a problem.

The values for Allocator64 have to be compile-time constants for
better performance,
but Allocator64 is not going to work on 39 (and probably on 42) bits anyway.
We'll need to use Allocator32.


> I think the most
> sane way to proceed is to:
>
> 1. Detect this at runtime and behave accordingly. This way we can use
> the same binaries on all AArch64 machines.
> 2. Detect this at configure time and hard-code the ranges. This way we
> can not penalise the architectures where that is constant, ex. x86_64.
> 3. Make into a cmake/configure option. This is especially important
> for cross-compilation.
>
> Does that sound sensible?
>
> cheers,
> --renato

[Venkataramanan Kumar]

Hi Konstantin/Renato,

>> The values that define the location of the shadow are initialized at startup,
>> so it should not be a problem.
>

> Excellent!
>

The Shadow offset, I need to set "1<<39" for Aarch64 (42 bit VA) and
1<<36 for Aarch64 (39 bit VA).

In asan_mapping.h, it is defined as static const u64
kAArch64_ShadowOffset64 = 1ULL << 36.

If I understand correctly, I need to initialize this at startup in
both places, compiler-rt/lib/asan and LLVM 's instrumentation place
lib/Transforms/Instrumentation/AddressSanitizer.cpp ?

>> The values for Allocator64 have to be compile-time constants for
>> better performance,
>> but Allocator64 is not going to work on 39 (and probably on 42) bits anyway.
>> We'll need to use Allocator32.

For Aarch64 with 42 VA bit I need to turn on Allocator64 and for 39
bit VA I need to turn on Allocator32.
ASAN/TSAN does not work when allocator32 is on for 42bit VA.

So I also need to detect and switch allocator types based on VA.


On 25 March 2015 at 22:21, Renato Golin <renato...@linaro.org> wrote:
> On 25 March 2015 at 16:41, Konstantin Serebryany

> <konstantin....@gmail.com> wrote:
>> Is it only 39 and 42, or is there a 3-rd option?
>

> As far as I know, only these two.

>
>
>> The values that define the location of the shadow are initialized at startup,
>> so it should not be a problem.
>

> Excellent!

>
>
>> The values for Allocator64 have to be compile-time constants for
>> better performance,
>> but Allocator64 is not going to work on 39 (and probably on 42) bits anyway.
>> We'll need to use Allocator32.
>

> That's our conclusion, too.
>
> I think the next step is to implement the run-time check for AArch64.
>
> Thanks for the help!
>
> cheers,
> --renato

[Christophe Lyon]

On 25 March 2015 at 17:41, Konstantin Serebryany

<konstantin....@gmail.com> wrote:
> On Wed, Mar 25, 2015 at 4:33 AM, Renato Golin <renato...@linaro.org> wrote:
>> On 24 March 2015 at 15:46, Konstantin Serebryany
>> <konstantin....@gmail.com> wrote:
>>>> is there a way to initialize ASAN mapping for Aarch64 during runtime ?
>>>
>>> This *is* happening at run-time.
>>> Did you talk to others from Linaro who already reported to have
>>> working asan on aarch64?
>>> Renato?
>>
>> Hi Kostya,
>>
>> There are multiple issues here...
>>
>> Venkat was working on TSAN for AArch64 and got caught in a virtual
>> address problem. Not only AArch64 has less virtual address bits (39 or
>> 42, as opposed to 48 in x86_64),
>
> Is it only 39 and 42, or is there a 3-rd option?
>

I believe there is support for 48, but no HW available at present.

>
>> but they're different and there is no
>> generic solution. That's when he started working on ASAN to get the
>> 64-bit allocator working, and the virtual address space became an
>> issue once again. In the end, for the sanitizers to work well on all
>> AArch64 variations, we need to solve the virtual address issue.
>>
>> IIRC, ASAN has some macros to define the address ranges, so at least
>> part of the ranges logic is not runtime based, right?
>
> The values that define the location of the shadow are initialized at startup,
> so it should not be a problem.
>
> The values for Allocator64 have to be compile-time constants for
> better performance,
> but Allocator64 is not going to work on 39 (and probably on 42) bits anyway.
> We'll need to use Allocator32.
>

Could you explain why it is not going to work? Should we think about fixing it?

>
>> I think the most
>> sane way to proceed is to:
>>
>> 1. Detect this at runtime and behave accordingly. This way we can use
>> the same binaries on all AArch64 machines.
>> 2. Detect this at configure time and hard-code the ranges. This way we
>> can not penalise the architectures where that is constant, ex. x86_64.
>> 3. Make into a cmake/configure option. This is especially important
>> for cross-compilation.
>>
>> Does that sound sensible?
>>
>> cheers,
>> --renato
>

[Konstantin Serebryany]

On Wed, Mar 25, 2015 at 10:23 AM, Venkataramanan Kumar
<venkataram...@linaro.org> wrote:
> Hi Konstantin/Renato,
>
>>> The values that define the location of the shadow are initialized at startup,
>>> so it should not be a problem.
>>
>> Excellent!
>>
>
> The Shadow offset, I need to set "1<<39" for Aarch64 (42 bit VA) and
> 1<<36 for Aarch64 (39 bit VA).
>
> In asan_mapping.h, it is defined as static const u64
> kAArch64_ShadowOffset64 = 1ULL << 36.

Why 1ULL << 36 or some other smaller value will not work for both?

>
> If I understand correctly, I need to initialize this at startup in
> both places, compiler-rt/lib/asan and LLVM 's instrumentation place
> lib/Transforms/Instrumentation/AddressSanitizer.cpp ?
>
>>> The values for Allocator64 have to be compile-time constants for
>>> better performance,
>>> but Allocator64 is not going to work on 39 (and probably on 42) bits anyway.
>>> We'll need to use Allocator32.
>
> For Aarch64 with 42 VA bit I need to turn on Allocator64

Why?

> and for 39
> bit VA I need to turn on Allocator32.
> ASAN/TSAN does not work when allocator32 is on for 42bit VA.

Let's not mix asan and tsan.
What's wrong with asan when "allocator32 is on for 42bit VA"?

[Konstantin Serebryany]

On Wed, Mar 25, 2015 at 1:27 PM, Christophe Lyon
<christo...@linaro.org> wrote:
> On 25 March 2015 at 17:41, Konstantin Serebryany
> <konstantin....@gmail.com> wrote:
>> On Wed, Mar 25, 2015 at 4:33 AM, Renato Golin <renato...@linaro.org> wrote:
>>> On 24 March 2015 at 15:46, Konstantin Serebryany
>>> <konstantin....@gmail.com> wrote:
>>>>> is there a way to initialize ASAN mapping for Aarch64 during runtime ?
>>>>
>>>> This *is* happening at run-time.
>>>> Did you talk to others from Linaro who already reported to have
>>>> working asan on aarch64?
>>>> Renato?
>>>
>>> Hi Kostya,
>>>
>>> There are multiple issues here...
>>>
>>> Venkat was working on TSAN for AArch64 and got caught in a virtual
>>> address problem. Not only AArch64 has less virtual address bits (39 or
>>> 42, as opposed to 48 in x86_64),
>>
>> Is it only 39 and 42, or is there a 3-rd option?
>>
> I believe there is support for 48, but no HW available at present.

Any publicly known roadmap here?
If 48 will appear soon, I don't want to hear about tsan on 39 and 42 --
this is not a trivial task to solve and tsan will function well only
if there is plenty of VA *and* physical memory.
Or we are stuck with 39 and 42 for a long time?

This is not related to asan. We can make asan work on 16 bit AS, if we need to.

>
>>
>>> but they're different and there is no
>>> generic solution. That's when he started working on ASAN to get the
>>> 64-bit allocator working, and the virtual address space became an
>>> issue once again. In the end, for the sanitizers to work well on all
>>> AArch64 variations, we need to solve the virtual address issue.
>>>
>>> IIRC, ASAN has some macros to define the address ranges, so at least
>>> part of the ranges logic is not runtime based, right?
>>
>> The values that define the location of the shadow are initialized at startup,
>> so it should not be a problem.
>>
>> The values for Allocator64 have to be compile-time constants for
>> better performance,
>> but Allocator64 is not going to work on 39 (and probably on 42) bits anyway.
>> We'll need to use Allocator32.
>>
> Could you explain why it is not going to work? Should we think about fixing it?

"fixing it" means using Allocator32.
Allocator64 is designed in assumption that there is a huge chunk of AS
available to the allocator
to place it's 50+ size classes in fixed (constant) address ranges.
Every region dedicated to a given size class should be at least the
size of the physical memory available on the system.
Suppose we have may have up to 32Gb available on an AArch device.
(even if not today, in near future we will have that much, right?)
It means we need 64x32Gb of address space, which is 2^41 -- we can not
fit this into 42-bit AS.
If we try hard and limit ourselves with smaller amount of physical RAM
in a device, we may be able to squeeze Allocator64 into 42 AS,
but not into 39 AS.

[Konstantin Serebryany]

On Wed, Mar 25, 2015 at 2:53 PM, Renato Golin <renato...@linaro.org> wrote:
> On 25 March 2015 at 21:40, Konstantin Serebryany

> <konstantin....@gmail.com> wrote:
>> Or we are stuck with 39 and 42 for a long time?
>

> Yes.
>
> There are a lot of production hardware using 39/42, we won't see them
> disappear any time soon. I'd risk to say, never. After all, ARM7TDMI
> is still one of the most popular ARM chips year on year, and it's
> already 17 years old.
>
> If we want it to run on current and near future hardware, we must support 39/42.

Then forget about Allocator64.
We need to either use Allocator32 (which is fine, imho. it works
pretty well on ARM, x86_32, MIPS, etc)
or invent something else (but I can't see why we should do that).

With tsan, I am quite reluctant to support 39, even 42 may be a challenge.

>
> cheers,
> --renato

[Renato Golin]

On 25 March 2015 at 21:40, Konstantin Serebryany
<konstantin....@gmail.com> wrote:

> Or we are stuck with 39 and 42 for a long time?

Yes.

There are a lot of production hardware using 39/42, we won't see them
disappear any time soon. I'd risk to say, never. After all, ARM7TDMI
is still one of the most popular ARM chips year on year, and it's
already 17 years old.

If we want it to run on current and near future hardware, we must support 39/42.

cheers,
--renato

[Venkataramanan Kumar]

Hi Konstantin,

On 26 March 2015 at 03:01, Konstantin Serebryany

<konstantin....@gmail.com> wrote:
> On Wed, Mar 25, 2015 at 10:23 AM, Venkataramanan Kumar
> <venkataram...@linaro.org> wrote:
>> Hi Konstantin/Renato,
>>
>>>> The values that define the location of the shadow are initialized at startup,
>>>> so it should not be a problem.
>>>
>>> Excellent!
>>>
>>
>> The Shadow offset, I need to set "1<<39" for Aarch64 (42 bit VA) and
>> 1<<36 for Aarch64 (39 bit VA).
>>
>> In asan_mapping.h, it is defined as static const u64
>> kAArch64_ShadowOffset64 = 1ULL << 36.
>
>
> Why 1ULL << 36 or some other smaller value will not work for both?
>

On 42 bit VA, I was trying to use 64 bit allocator.

I have to fix the allocator start and size to 0x10000000000. Other
wise I got the below an assert failures.

COMPILER_CHECK(kSpaceBeg % kSpaceSize == 0)
COMPILER_CHECK((kRegionSize) >= (1ULL << (SANITIZER_WORDSIZE / 2)))

Putting shadow offset at 2^39 I got HighMemBeg starting at
0x10000000000 which also will not interfere with application,
libraries and the stack.

>>
>> If I understand correctly, I need to initialize this at startup in
>> both places, compiler-rt/lib/asan and LLVM 's instrumentation place
>> lib/Transforms/Instrumentation/AddressSanitizer.cpp ?
>>
>>>> The values for Allocator64 have to be compile-time constants for
>>>> better performance,
>>>> but Allocator64 is not going to work on 39 (and probably on 42) bits anyway.
>>>> We'll need to use Allocator32.
>>
>> For Aarch64 with 42 VA bit I need to turn on Allocator64
> Why?

ASAN tests passed with 64 bit allocator and with changes I mentioned above.

So I was assuming we can switch 64 bit allocators when we detect 42
bit VA. But I agree, as we discussed it is not a good thing to do. It
should be uniform across all VA and hence 32 bit allocators should be
default.

>
>> and for 39
>> bit VA I need to turn on Allocator32.
>> ASAN/TSAN does not work when allocator32 is on for 42bit VA.
>
> Let's not mix asan and tsan.
> What's wrong with asan when "allocator32 is on for 42bit VA"?

On 42 bit VA, running a simple stack resuse test case. I got this
assert failure.

==24573==AddressSanitizer CHECK failed:
/home/venkataramanan.kumar/LLVM/TSAN_port/Source/projects/compiler-rt/lib/sanitizer_common/sanitizer_allocator.h:841
"((res)) < ((kNumPossibleRegions))" (0x3ffb56, 0x80000)

I am looking it into now,

>>
>> So I also need to detect and switch allocator types based on VA.
>>
>>
>> On 25 March 2015 at 22:21, Renato Golin <renato...@linaro.org> wrote:
>>> On 25 March 2015 at 16:41, Konstantin Serebryany
>>> <konstantin....@gmail.com> wrote:
>>>> Is it only 39 and 42, or is there a 3-rd option?
>>>
>>> As far as I know, only these two.
>>>
>>>
>>>> The values that define the location of the shadow are initialized at startup,
>>>> so it should not be a problem.
>>>
>>> Excellent!
>>>
>>>
>>>> The values for Allocator64 have to be compile-time constants for
>>>> better performance,
>>>> but Allocator64 is not going to work on 39 (and probably on 42) bits anyway.
>>>> We'll need to use Allocator32.
>>>
>>> That's our conclusion, too.
>>>
>>> I think the next step is to implement the run-time check for AArch64.
>>>
>>> Thanks for the help!
>>>
>>> cheers,
>>> --renato

regards,
Venkat,

[Venkataramanan Kumar]

Hi Konstantin,

On 26 March 2015 at 18:10, Venkataramanan Kumar

The "kNumPossibleRegions" is defined as per 39 bit VA machine (juno)
as 0x8000000000 (max 39 bit VA) / (2^20) = 0x80000.

On 42 bit I get stack memory reference at 0x3ffb5600000. The region id
computation for the memory is 0x3ffb5600000/(2^20) is "0x3ffb56"

Hence the check "((res)) < ((kNumPossibleRegions))" fails.

I have to change the SANITIZER_MMAP_RANGE_SIZE to 1 <<42.

0x40000000000 (max 42 bit VA) / (2^20) = 0x3FFFFF

Now I still find 76 cases still failing and looking at them.

>
>
>>>
>>> So I also need to detect and switch allocator types based on VA.
>>>
>>>
>>> On 25 March 2015 at 22:21, Renato Golin <renato...@linaro.org> wrote:
>>>> On 25 March 2015 at 16:41, Konstantin Serebryany
>>>> <konstantin....@gmail.com> wrote:
>>>>> Is it only 39 and 42, or is there a 3-rd option?
>>>>
>>>> As far as I know, only these two.
>>>>
>>>>
>>>>> The values that define the location of the shadow are initialized at startup,
>>>>> so it should not be a problem.
>>>>
>>>> Excellent!
>>>>
>>>>
>>>>> The values for Allocator64 have to be compile-time constants for
>>>>> better performance,
>>>>> but Allocator64 is not going to work on 39 (and probably on 42) bits anyway.
>>>>> We'll need to use Allocator32.
>>>>
>>>> That's our conclusion, too.
>>>>
>>>> I think the next step is to implement the run-time check for AArch64.
>>>>
>>>> Thanks for the help!
>>>>
>>>> cheers,
>>>> --renato
>
> regards,
> Venkat,

regards,
Venkat.

[Venkataramanan Kumar]

Hi Konstantin/Renato

Remaining failures were due to the way compiler (LLVM) generates code
for accessing shadow memory.
It optimizes the code by Oring shadow offset instead of adding it
thinking that shadow offset is at 1/8 of address space.

On 42 bit VA, default value of 1<<36 is not at 1/8 of address space.

I changed it to

// OR-ing shadow offset if more efficient (at least on x86) if the offset
// is a power of two, but on ppc64 we have to use add since the shadow
// offset is not necessary 1/8-th of the address space.
Mapping.OrShadowOffset = !IsAArch64 && !IsPPC64
&& !(Mapping.Offset & (Mapping.Offset - 1));

Now

Failing Tests (1):
AddressSanitizer-aarch64-linux :: TestCases/throw_invoke_test.cc

Expected Passes : 414
Expected Failures : 6
Unsupported Tests : 129
Unexpected Failures: 1

Failure is due to missing stdc++, I dont have root permission on that machine.

/usr/bin/ld: cannot find -lstdc++
clang-3.7: error: linker command failed with exit code 1 (use -v to
see invocation)

regards,
Venkat.




On 30 March 2015 at 17:23, Venkataramanan Kumar

[Venkataramanan Kumar]

Hi Konstantin/Renato,

I did make check all on LLVM on 42 bit VA for the ASAN changes.

Testing Time: 845.30s
********************
Failing Tests (1):
SanitizerCommon-asan :: Linux/getpwnam_r_invalid_user.cc

Expected Passes : 22090
Expected Failures : 129
Unsupported Tests : 420
Unexpected Failures: 1
make[3]: *** [CMakeFiles/check-all] Error 1
make[2]: *** [CMakeFiles/check-all.dir/all] Error 2
make[1]: *** [CMakeFiles/check-all.dir/rule] Error 2
make: *** [check-all] Error 2
'make check-all' failed, bailing out


The test getpwnam_r_invalid_user.cc fails at assert(res == 0);

#include <assert.h>
#include <pwd.h>
#include <signal.h>
#include <stdio.h>
#include <sys/types.h>
#include <unistd.h>

int main(void) {
struct passwd pwd;
struct passwd *pwdres;
char buf[10000];
int res = getpwnam_r("no-such-user", &pwd, buf, sizeof(buf), &pwdres);
assert(res == 0);
assert(pwdres == 0);
return 0;
}

getpwnam_r, here is expected to return 0 but in the Aarch64 test
machine it returns 2 (ENOENT)

man getpwnam_r says

(Snip)
On success, getpwnam_r() and getpwuid_r() return zero, and set
*result to pwd. If no matching password record was found, these
func‐tions return 0 and store NULL in *result. In case of error, an
error number is returned, and NULL is stored in *result.

ERRORS
0 or ENOENT or ESRCH or EBADF or EPERM or ...
The given name or uid was not found
(Snip)

So Planning to drop the assert "assert(res == 0)" from that test case.

does that look good to you?

regards,
Venkat.



On 31 March 2015 at 19:34, Venkataramanan Kumar

[Renato Golin]

On 2 April 2015 at 13:50, Venkataramanan Kumar

<venkataram...@linaro.org> wrote:
> So Planning to drop the assert "assert(res == 0)" from that test case.
>
> does that look good to you?

No. A non-zero return doesn't mean the name wasn't found, but that the
function failed. Name-not-found situation still returns zero. There is
something wrong there...

--renato

[Venkataramanan Kumar]

It also says under ERRORS
(snip)

0 or ENOENT or ESRCH or EBADF or EPERM or ...
The given name or uid was not found

(snip)

Looks like we can remove the assert.

regards,
Venkat.

[Renato Golin]

On 2 April 2015 at 17:14, Venkataramanan Kumar

<venkataram...@linaro.org> wrote:
> It also says under ERRORS
> (snip)
> 0 or ENOENT or ESRCH or EBADF or EPERM or ...
> The given name or uid was not found
> (snip)
>
> Looks like we can remove the assert.

I think the bigger question is: what is this testing...

Kostya,

This test seems very fragile and possibly {platform+libc+OS}-specific.
Is there any other way of testing whatever it was without calling libc
functions? If that was testing the specific function, I'm not sure
those asserts will give you any indication of success, other than libc
execution tests, which ASAN is not the place to have.

If the test is trying not to have any sanitizer crashes, than the
asserts are indeed useless.

cheers,
--renato

[Evgeniy Stepanov]

This assert verifies that the test has tested the situation it was
supposed to test. It is a regression test in a situation where a user
is not found, which used to cause a crash in sanitizer interceptor
code.

Could you please check if this test passes without sanitizer? It's
either getpwname_r not following the spec, or ASan changing it's
behaviour.

[Venkataramanan Kumar]

Hi

I checked without sanitizers also, the return value is ENOENT in the
Aarch64 machine and its fails at assert (res==0).

regards,
Venkat.

[Evgeniy Stepanov]

Well, then the man page is wrong.
Please change the assert to allow ENOENT in addition to 0.

[Venkataramanan Kumar]

Hi Evgeniy and Konstantin,

Ok adding ENOENT to the assert fixes that and all test passes.
Now ASAN on 42 bit VA works with the 32 bit allocate code.

I have couple of questions here.

"asan/asan_allocator.h:typedef SizeClassAllocator32<0,
SANITIZER_MMAP_RANGE_SIZE, 16,"


(1) Can I post the patch to LLVM by increasing
SANITIZER_MMAP_RANGE_SIZE to 1<<42?

For 39 bit VA, TwoLevelByteMap uses more dimensions now.
If someone wants to support 48 bit VA later, we may need to
increase it by 1<<48.

(2) SANITIZER_MMAP_RANGE_SIZE is used in template parameter and
compile time known.
Still there a way to initialize it dynamically ?

[Renato Golin]

On 5 April 2015 at 19:18, Venkataramanan Kumar

<venkataram...@linaro.org> wrote:
> (1) Can I post the patch to LLVM by increasing
> SANITIZER_MMAP_RANGE_SIZE to 1<<42?

Venkat,

Let's wait and make sure we can cleanly apply Jakub's patch [1] to
LLVM without any license issues.

> For 39 bit VA, TwoLevelByteMap uses more dimensions now.
> If someone wants to support 48 bit VA later, we may need to
> increase it by 1<<48.

I don't like increasing the memory footprint on the most common
AArch64 platforms just because there will be 48-bit VA in the future.

For now, the reference platform is still Juno, which is 39-bit. It
also works in the APMs, which have just joined with Gigabyte for
server grade ATX boards. If we want to make it work in the AMD (which
is the only 42-bit so far, and it's not even public), we'll need a way
to pick the mmap range at run time. I remember Kostya saying there was
a way, we should at least do that. For 48-bit we must absolutely do
that, and only when someone has real hardware to test on.

But that's all after we're sure that the patch is safe for LLVM.

cheers,
--renato

[1] http://pkgs.fedoraproject.org/cgit/gcc.git/tree/gcc5-libsanitize-aarch64-va42.patch

[Evgeniy Stepanov]

On Sun, Apr 5, 2015 at 9:33 PM, Renato Golin <renato...@linaro.org> wrote:
> On 5 April 2015 at 19:18, Venkataramanan Kumar
> <venkataram...@linaro.org> wrote:
>> (1) Can I post the patch to LLVM by increasing
>> SANITIZER_MMAP_RANGE_SIZE to 1<<42?
>
> Venkat,
>
> Let's wait and make sure we can cleanly apply Jakub's patch [1] to
> LLVM without any license issues.
>
>
>> For 39 bit VA, TwoLevelByteMap uses more dimensions now.
>> If someone wants to support 48 bit VA later, we may need to
>> increase it by 1<<48.
>
> I don't like increasing the memory footprint on the most common
> AArch64 platforms just because there will be 48-bit VA in the future.

It's 8KB for 42-bit VA, and 512KB for 48-bit. And that's .bss space
that is never written to, so no extra physical memory use at all.

I'm not sure we can make bytemap allocation dynamic without
sacrificing performance.

[Renato Golin]

On 6 April 2015 at 10:04, Evgeniy Stepanov <eug...@google.com> wrote:
> It's 8KB for 42-bit VA, and 512KB for 48-bit. And that's .bss space
> that is never written to, so no extra physical memory use at all.

Oh, ok, that sounds good. I thought it was per-object.

But I still think this submission should come from Jakub. I don't know
what the legal implications are, this being GCC.

cheers,
--renato