Stacktrace offsets that can't be mapped to sources

[Miha]

Hi!

I am using NDK r8 and SDK r18 to compile our app. The app is java/c++, interactions going through JNI.

Sometimes, the app would crash somewhere and I would get a stacktrace. This stacktrace would include offsets where it happened and I could go on and use addr2line to find out where in source did it crash.

Sometimes, I would get addres in the lower range (0x00014f0b), and sometimes in the upper range (0x819a031b). The addresses in the upper range would need to be calculated - subtract 0x8180000 from the actual address to recieve the real offset. But lately, I am getting address like this: 

be8872fc  511990b4  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

And I don't know how to map 511990b4 to the location of my source files. Below is the header of the stack trace, with relevant registers. I'm not sure if this is related to newer sdk/ndk versions or newer firmware/OS on the device itself.

If I run the app in the emulator (API level 10 - Android 2.3.3), I get different offset numbers (still not usable with addr2line). Note that I am using NDK's arm-linux-androideabi-addr2line.

 1817                  DEBUG  I  Build fingerprint: 'samsung/GT-I9100/GT-I9100:4.0.3/IML74K/XWLPD:user/release-keys'

  1817                  DEBUG  I  pid: 25420, tid: 25420  >>> com.islonline.isllight.android <<<

  1817                  DEBUG  I  signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr deadbaad

  1817                  DEBUG  I   r0 deadbaad  r1 00000001  r2 40000000  r3 00000000

  1817                  DEBUG  I   r4 00000000  r5 00000027  r6 00000004  r7 4be5bca0

  1817                  DEBUG  I   r8 be887690  r9 4be5bc98  10 4cc8b2d3  fp be88768c

  1817                  DEBUG  I   ip ffffffff  sp be8872f8  lr 40080179  pc 4007c8d8  cpsr 60000030

  1817                  DEBUG  I   d0  0000000000c5c100  d1  0000000000000000

  1817                  DEBUG  I   d2  0000000000000000  d3  439e0000431d0000

  1817                  DEBUG  I   d4  8000000000000000  d5  419000003f800000

  1817                  DEBUG  I   d6  c0400000c1900000  d7  00c5c10000000000

  1817                  DEBUG  I   d8  0000000043470000  d9  431d000042c80000

  1817                  DEBUG  I   d10 0000000000000000  d11 0000000000000000

  1817                  DEBUG  I   d12 0000000000000000  d13 0000000000000000

  1817                  DEBUG  I   d14 0000000000000000  d15 0000000000000000

  1817                  DEBUG  I   d16 4168b82010000000  d17 3fe0000000000000

  1817                  DEBUG  I   d18 4008000000000000  d19 0000000000000000

  1817                  DEBUG  I   d20 3ff0000000000000  d21 8000000000000000

  1817                  DEBUG  I   d22 c008000000000000  d23 0000000000000000

  1817                  DEBUG  I   d24 0000000000000000  d25 0000000000000000

  1817                  DEBUG  I   d26 0000000000000000  d27 e6e6e6e6e6e6e6e6

  1817                  DEBUG  I   d28 bffe0d01b7ee0434  d29 3ff0000000000000

  1817                  DEBUG  I   d30 0000000000000000  d31 3ff0000000000000

  1817                  DEBUG  I   scr 80000012

  1817                  DEBUG  I  

  1817                  DEBUG  I           #00  pc 000178d8  /system/lib/libc.so

  1817                  DEBUG  I           #01  pc 0001ea74  /system/lib/libc.so (__assert2)

Thanks,

 Miha.

[David Turner]

FYI, fault address deadbaad is a magic code that indicates that malloc() or free() detected that your heap is corrupted and forcibly aborted your process.

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To view this discussion on the web visit https://groups.google.com/d/msg/android-ndk/-/LG8YfkBzbg4J.
To post to this group, send email to andro...@googlegroups.com.
To unsubscribe from this group, send email to android-ndk...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/android-ndk?hl=en.

[Miha Valencic]

Thanks David. So you think this is happening? This is actaully happening on a simple assert... it seems.

Does that also mean that no addr2line magic for me when heap gets corrupted?

Thanks,

 Miha.

[David Turner]

On Wed, May 30, 2012 at 4:11 PM, Miha Valencic <miha.v...@gmail.com> wrote:

Thanks David. So you think this is happening? This is actaully happening on a simple assert... it seems.

Does that also mean that no addr2line magic for me when heap gets corrupted?

It means that the call site won't help you much because the corruption certainly occured before.

Normally, the stack trace will automatically remap the addresses into library offsets for you.

Recent versions of Android implement address-space virtualization, which is why the libraries are not loaded at the same address on each iteration.

To know the real load address, you will have to look at the content of /proc/$PID/maps and find where exactly the library you're looking for is mapped.

But again, this will not help you in this exact case.

 

Thanks,

 Miha.

On Wed, May 30, 2012 at 4:09 PM, David Turner <di...@android.com> wrote:

FYI, fault address deadbaad is a magic code that indicates that malloc() or free() detected that your heap is corrupted and forcibly aborted your process.

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.

[Martin Storsjö]

On Wed, 30 May 2012, Miha Valencic wrote:

> Thanks David. So you think this is happening? This is actaully happening on
> a simple assert... it seems.

Hmm, doesn't read errors for 0xdeadbaad also appear on abort() from
failing asserts? So that the "error" in this case simply would be that you
have an assert for a condition which isn't true.

// Martin

[Miha Valencic]

On Wed, May 30, 2012 at 4:14 PM, David Turner <di...@android.com> wrote:

It means that the call site won't help you much because the corruption certainly occured before.

Normally, the stack trace will automatically remap the addresses into library offsets for you.

Recent versions of Android implement address-space virtualization, which is why the libraries are not loaded at the same address on each iteration.

To know the real load address, you will have to look at the content of /proc/$PID/maps and find where exactly the library you're looking for is mapped.

I understand that won't help me in this case. In other cases, though, which number is the base address? The first one (0x80800000)? Because it is 'not writable' block of memory?

80800000-80ce5000 r-xp 00000000 1f:01 570        /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

80ce5000-80cf1000 rwxp 004e4000 1f:01 570        /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

This is what is contained in the proc/PID/maps. 

Re: Martin: I guess assert() could call abort(). I'm trying to catch an exception but it is not an exception that is being thrown... I'm just issuing some tests and the app explodes on the second run.

Miha

[David Turner]

On Wed, May 30, 2012 at 4:24 PM, Miha Valencic <miha.v...@gmail.com> wrote:

On Wed, May 30, 2012 at 4:14 PM, David Turner <di...@android.com> wrote:

It means that the call site won't help you much because the corruption certainly occured before.

Normally, the stack trace will automatically remap the addresses into library offsets for you.

Recent versions of Android implement address-space virtualization, which is why the libraries are not loaded at the same address on each iteration.

To know the real load address, you will have to look at the content of /proc/$PID/maps and find where exactly the library you're looking for is mapped.

I understand that won't help me in this case. In other cases, though, which number is the base address? The first one (0x80800000)? Because it is 'not writable' block of memory?

80800000-80ce5000 r-xp 00000000 1f:01 570        /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

80ce5000-80cf1000 rwxp 004e4000 1f:01 570        /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

The first line, most of the library is mapped read-only. Only a smaller number of pages need to be writable.

 

This is what is contained in the proc/PID/maps. 

Re: Martin: I guess assert() could call abort(). I'm trying to catch an exception but it is not an exception that is being thrown... I'm just issuing some tests and the app explodes on the second run.

Miha

--

[Miha]

Hey!

On Wednesday, May 30, 2012 5:17:34 PM UTC+2, Digit wrote:

80800000-80ce5000 r-xp 00000000 1f:01 570        /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

80ce5000-80cf1000 rwxp 004e4000 1f:01 570        /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

This all still does not produce the correct offsets. For instance, I have a stack trace: (removed some lines for brevity) 

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***

Build fingerprint: 'generic/sdk/generic:2.3.3/GRI34/101070:eng/test-keys'

pid: 1964, tid: 2005  >>> com.islonline.isllight.android <<<

signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 00000034

 r0 49161a38  r1 00000000  r2 00000000  r3 00000000

 r4 80cf427c  r5 00389bec  r6 00000000  r7 00000000

 r8 00000000  r9 00000000  10 00000000  fp 49161a7c

 ip 80cf44c4  sp 49161a50  lr afd10f08  pc 809696c0  cpsr 20000010

         #00  pc 809696c0  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

         #01  lr afd10f08  /system/lib/libc.so

code around pc:

809696a0 e59f4178 e08f4004 e50b0028 e3a0000f 

809696b0 e3a01000 ebff9802 e51b3028 e3a02000 

809696c0 e5c32034 e24b3020 e1a00003 e59f3150 

809696d0 e08f3003 e1a01003 ebffe411 e24b3014 

809696e0 e1a00003 eb0a9707 e24b3018 e24b2014 

code around lr:

afd10ee8 e2506000 0a000019 e5964000 e5967000 

afd10ef8 e2144903 1a00000e e5965000 ebffede4 

afd10f08 e2055a02 e3853001 e1500003 0a000006 

afd10f18 e5865000 e1a00006 e1a01005 e3a02001 

afd10f28 ebfffeac e1a00004 e8bd87f0 e3a00000 

stack:

    49161a1c  8094fc28  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161a38  80cf88dc  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161a40  80cf427c  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161a50  80c66f38  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161a64  afd14769  /system/lib/libc.so

    49161a6c  80c35a40  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161a70  80cf427c  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161a7c  80972b74  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161a8c  80b7f104  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161aac  80b7cf18  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161abc  80c6729c  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161acc  80c4aa08  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161b14  afd1386d  /system/lib/libc.so

    49161b24  80c4aa08  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161b2c  80cf427c  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161b3c  80aa70b0  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161b44  80cf427c  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161b54  809714e0  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161b8c  80b7dbe0  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161b94  80c35170  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161ba0  80c672e4  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161ba4  afd10a01  /system/lib/libc.so

    49161bb0  80c672e4  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161bbc  afd10fb8  /system/lib/libc.so

    49161bdc  80ad46e8  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161bfc  80ad4910  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161c0c  80a1cd04  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

    49161c2c  80970a94  /data/data/com.islonline.isllight.android/lib/libandroid-bridge.so

The /proc/pid/maps tells me the library is loaded at 0x80800000, so I subtract 0x80800000 from say 0x809696c0. addr2line resolves this to a location in source which could not have made this crash, as there is just a simple member boolean assignment. And for instance, address 0x80cf427c can not be resolved (relative offset =  0x004f427c).

What am I doing wrong? It is really frustrating to not be able to troubleshoot and debug android crashes.

[mic _]

It seems like the instruction in question is a store and that you've got a NULL pointer issue:

-- Disassembly around PC (809696c0) --
--------------------------------------
809696a0:    e59f4178     ldr    r4, [pc, #376]    ; 80969820 <pc+0x130>
809696a4:    e08f4004     add    r4, pc, r4
809696a8:    e50b0028     str    r0, [fp, #-40]    ; 0x28
809696ac:    e3a0000f     mov    r0, #15
809696b0:    e3a01000     mov    r1, #0
809696b4:    ebff9802     bl    8094f6c4 <_binary_distrace_tmp_size+0x8094f674>
809696b8:    e51b3028     ldr    r3, [fp, #-40]    ; 0x28
809696bc:    e3a02000     mov    r2, #0
-->> 809696c0:    e5c32034     strb    r2, [r3, #52]    ; 0x34  ; r3 = 0x00000000
809696c4:    e24b3020     sub    r3, fp, #32
809696c8:    e1a00003     mov    r0, r3


/Michael

--

You received this message because you are subscribed to the Google Groups "android-ndk" group.

To view this discussion on the web visit https://groups.google.com/d/msg/android-ndk/-/RUn3ePHqi8QJ.

[Miha Valencic]

Thanks Michael! What tool did you use to convert those hex number to assembler?

On Thu, Jun 14, 2012 at 1:51 PM, mic _ <mico...@gmail.com> wrote:

It seems like the instruction in question is a store and that you've got a NULL pointer issue:

[mic _]

>>What tool did you use to convert those hex number to assembler?

A small Python script that I wrote for quick initial analysis of native tombstones. Since I wrote it at work I don't think I can put it on the net, but the steps it performs are just as follows:

* Go through a log file and look for a tombstone.
* Locate the "code around pc/lr" hex string blobs.
* Convert each string to a 32-bit number and write the numbers to a binary file.
* Slap an ELF header on the binary file using objcopy.
* Disassemble the resulting ELF file using objdump.

/Michael

--

You received this message because you are subscribed to the Google Groups "android-ndk" group.

[Miha Valencic]

On Thu, Jun 14, 2012 at 2:07 PM, mic _ <mico...@gmail.com> wrote:

>>What tool did you use to convert those hex number to assembler?

A small Python script that I wrote for quick initial analysis of native tombstones. Since I wrote it at work I don't think I can put it on the net, but the steps it performs are just as follows:

I understand.

 

* Go through a log file and look for a tombstone.
* Locate the "code around pc/lr" hex string blobs.
* Convert each string to a 32-bit number and write the numbers to a binary file.
* Slap an ELF header on the binary file using objcopy.
* Disassemble the resulting ELF file using objdump.

Thanks! That's very helpful!

Regards,

 Miha. 

[Thomas Martitz]

Am 14.06.2012 14:07, schrieb mic _:
> >>What tool did you use to convert those hex number to assembler?
>
> A small Python script that I wrote for quick initial analysis of
> native tombstones. Since I wrote it at work I don't think I can put it
> on the net, but the steps it performs are just as follows:
>
> * Go through a log file and look for a tombstone.
> * Locate the "code around pc/lr" hex string blobs.
> * Convert each string to a 32-bit number and write the numbers to a
> binary file.
> * Slap an ELF header on the binary file using objcopy.
> * Disassemble the resulting ELF file using objdump.

You don't need the objcopy step. objdump can disassemble raw binary
files if you tell it the ABI (however it cannot separate code and data)
(arm-linux-androideabi-objdump -D $FILE -b binary -m arm [-M force-thumb]).

Best regards.