Best way to get a readable stacktrace with line numbers

[Felipe Monteiro de Carvalho]

Hello,

Does anyone have an example of which tool and how to call it to
convert the ndk stack trace into a more readable form?

For example, having:

I/DEBUG ( 70): thread: .pascal.lcltest
I/DEBUG ( 70): signal 4 (SIGILL), fault addr a8b41314
I/DEBUG ( 70): r0 429fd8f8 r1 42a83460 r2 8158ee70 r3
8157b9b4
I/DEBUG ( 70): r4 42a83300 r5 a8b41311 r6 813c781c r7
00000001
I/DEBUG ( 70): r8 81340860 r9 814d254c 10 00000002 fp
00000000
I/DEBUG ( 70): ip befa9d60 sp befaa198 lr 813abbc8 pc
a8b41314 cpsr 20000010
I/DEBUG ( 70): #00 pc 00041314 /system/lib/
libsqlite.so
I/DEBUG ( 70): #01 pc 0001057c /system/lib/libc.so
I/DEBUG ( 70):
I/DEBUG ( 70): code around pc:
I/DEBUG ( 70): a8b412f4 a029a697 ffffb60e ffffb613 ffff9d18
I/DEBUG ( 70): a8b41304 4b771290 f7ffb510 bd10fe71 2206b510
I/DEBUG ( 70): a8b41314 f7ff2300 bd10fe6b 4c23b5f0 2300b085
I/DEBUG ( 70): a8b41324 600b9003 1c0d447c fcfef7ce d1381e06
I/DEBUG ( 70): a8b41334 f7c92000 491dfeb1 18601c07 78043064
I/DEBUG ( 70):
I/DEBUG ( 70): code around lr:
I/DEBUG ( 70): 813abba8 e51b002c e3500000 059f2048 01a00002
I/DEBUG ( 70): 813abbb8 e59f2044 e5925000 e1a0e00f e1a0f005
I/DEBUG ( 70): 813abbc8 e1a01000 e1a00004 eb000022 ebf56215
I/DEBUG ( 70): 813abbd8 e24b002c ebf53ad4 e3a00000 e50b002c
I/DEBUG ( 70): 813abbe8 e51b0064 e3500000 1bf5628a e91ba830
I/DEBUG ( 70):
I/DEBUG ( 70): stack:
I/DEBUG ( 70): befaa158 ab2494dc /system/lib/libskia.so
I/DEBUG ( 70): befaa15c befaa488 [stack]
I/DEBUG ( 70): befaa160 00000001
I/DEBUG ( 70): befaa164 befaa488 [stack]
I/DEBUG ( 70): befaa168 00000001
I/DEBUG ( 70): befaa16c 422c0000 /system/framework/
framework.odex
I/DEBUG ( 70): befaa170 42180000 /system/framework/
framework.odex
I/DEBUG ( 70): befaa174 00000085
I/DEBUG ( 70): befaa178 00000000
I/DEBUG ( 70): befaa17c 00004051
I/DEBUG ( 70): befaa180 0000002a
I/DEBUG ( 70): befaa184 42a83300
I/DEBUG ( 70): befaa188 813abb1c /data/data/
com.pascal.lcltest/lib/liblclapp.so
I/DEBUG ( 70): befaa18c 813c781c /data/data/
com.pascal.lcltest/lib/liblclapp.so
I/DEBUG ( 70): befaa190 42a83300
I/DEBUG ( 70): befaa194 813abb94 /data/data/
com.pascal.lcltest/lib/liblclapp.so
I/DEBUG ( 70): #00 befaa198 ab249d8c /system/lib/libskia.so
I/DEBUG ( 70): befaa19c afd10580 /system/lib/libc.so
I/DEBUG ( 70): #01 befaa1a0 00146b00 [heap]
I/DEBUG ( 70): befaa1a4 ab249d8c /system/lib/libskia.so
I/DEBUG ( 70): befaa1a8 00000000
I/DEBUG ( 70): befaa1ac afd10460 /system/lib/libc.so
I/DEBUG ( 70): befaa1b0 00000000
I/DEBUG ( 70): befaa1b4 42a83300
I/DEBUG ( 70): befaa1b8 813abb1c /data/data/
com.pascal.lcltest/lib/liblclapp.so
I/DEBUG ( 70): befaa1bc 813c781c /data/data/
com.pascal.lcltest/lib/liblclapp.so
I/DEBUG ( 70): befaa1c0 00000001
I/DEBUG ( 70): befaa1c4 81340860 /data/data/
com.pascal.lcltest/lib/liblclapp.so
I/DEBUG ( 70): befaa1c8 814d254c /data/data/
com.pascal.lcltest/lib/liblclapp.so
I/DEBUG ( 70): befaa1cc 00000002
I/DEBUG ( 70): befaa1d0 befaa214 [stack]
I/DEBUG ( 70): befaa1d4 befaa198 [stack]
I/DEBUG ( 70): befaa1d8 813abb4c /data/data/
com.pascal.lcltest/lib/liblclapp.so
I/DEBUG ( 70): befaa1dc befaa1b4 [stack]
I/DEBUG ( 70): befaa1e0 befaa2c0 [stack]
I/DEBUG ( 70): befaa1e4 00000001
I/ActivityManager( 99): Process com.pascal.lcltest (pid 645) has
died.
I/WindowManager( 99): WIN DEATH: Window{44d9b7c0
com.pascal.lcltest/
com.pascal.lcltest.LCLActivity paused=false}

thanks =)

Felipe Monteiro de Carvalho

[Felipe Monteiro de Carvalho]

Hello,

Doing some tryes on my own:

[felipe@localhost android]$ ~/Programas/android-ndk-r7/toolchains/arm-
linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-androideabi-
addr2line -e libs/armeabi/liblclapp.so 0x813ad160
??:0

It seams that for addresses which are very big like 813xxxxx nothing
is found =(

[felipe@localhost android]$ ~/Programas/android-ndk-r7/toolchains/arm-
linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-androideabi-
addr2line -e libs/armeabi/liblclapp.so 0x00041314
/home/felipe/Programas/lazarus/lcl/maskedit.pp:0

For smaller addresses it finds something, finds the correct unit, but
not the correct line.

thanks,

Felipe

[Felipe Monteiro de Carvalho]

This brings an interresting question: Which debug info format is the
best to use in NDK projects?

FPC has the following options:
*stabs
*dwarf2
*dwarf3

thanks,

Felipe

[mic _]

>>It seams that for addresses which are very big like 813xxxxx nothing
>>is found =(

That address is the virtual address where the library is loaded. You need to convert it to a file offset. Try clearing the top bits (e.g. use 000ad160).


>>addr2line -e libs/armeabi/liblclapp.so 0x00041314
>>/home/felipe/Programas/

>>lazarus/lcl/maskedit.pp:0

Wasn't 0x41314 inside libsqlite.so according to the stack trace ?

>>For smaller addresses it finds something, finds the correct unit, but
>>not the correct line.

Does the library contain any debug symbols / a line number section ?


/Michael

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To post to this group, send email to andro...@googlegroups.com.
To unsubscribe from this group, send email to android-ndk...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/android-ndk?hl=en.

[Felipe Monteiro de Carvalho]

On Apr 3, 2:11 pm, mic _ <micol...@gmail.com> wrote:
> That address is the virtual address where the library is loaded. You need
> to convert it to a file offset. Try clearing the top bits (e.g. use
> 000ad160).

Didn't help much:

[felipe@localhost androidlcl]$ ~/Programas/android-ndk-r7/toolchains/
arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-
androideabi-addr2line -e android/libs/armeabi/liblclapp.so 0x000ad160
/home/felipe/Programas/lazarus/lcl/maskedit.pp:0

> Wasn't 0x41314 inside libsqlite.so according to the stack trace ?

ops, you are correct!

> Does the library contain any debug symbols / a line number section ?

Yes, although in a bit convoluted fashion. Some units are compiled
with stabs, other units with dwarf2 and yet other ones without debug
info ...

I was thinking of changing this and building everything with one
single format, but then it came the question from my previous e-mail:
Which one is preferible for NDK development?

Felipe

[mic _]

>>Didn't help much

If you can reproduce the crash you can try this tip from another thread:

>>Do you know how do I check the base address of the library?

>
>If you can get in there while it's still running, it's probably easier to look at /proc/pid ##/maps

/Michael

Felipe

[Riasat Abir]

You may look at this following link how to read crash dump.

http://kobablog.wordpress.com/2011/05/14/how-to-read-crash-dump-of-android/ 

When porting Android system, sometimes boot fails animating Android logo forever. At the case some critical part failed by some reason. Look at log from logcat. There is crash dump.
Let’s check the crash dump with me. (Japanese version of this page)

The debuggerd generated this crash dump. I mentioned before.

I/DEBUG   ( 3037): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
I/DEBUG   ( 3037): Build fingerprint: 'emxx/kmc_kzm9d/kzm9d/:2.2/FRF91/eng.koba.20110221.133144:eng/test-keys'
I/DEBUG   ( 3037): pid: 3233, tid: 3234  >>> /system/bin/netd <<<
I/DEBUG   ( 3037): signal 11 (SIGSEGV), fault addr deadbaad
I/DEBUG   ( 3037):  r0 00001728  r1 100ffd7c  r2 00000027  r3 00000000
I/DEBUG   ( 3037):  r4 afd42328  r5 00000000  r6 00000000  r7 00011000
I/DEBUG   ( 3037):  r8 00100000  r9 aef01cad  10 10000000  fp 40008008
I/DEBUG   ( 3037):  ip ffffffff  sp 100ffd68  lr deadbaad  pc afd11ca4  cpsr 40000030
I/DEBUG   ( 3037):  d0  3d454d414e564544  d1  6164732f7665642f
I/DEBUG   ( 3037):  d2  65766972642d693d  d3  3a302d6273752d64
I/DEBUG   ( 3037):  d4  652d6d726f667461  d5  696368652d78786d
I/DEBUG   ( 3037):  d6  2d7265766972642d  d7  2e313a302d627375
I/DEBUG   ( 3037):  d8  0000000000000000  d9  0000000000000000
I/DEBUG   ( 3037):  d10 0000000000000000  d11 0000000000000000
I/DEBUG   ( 3037):  d12 0000000000000000  d13 0000000000000000
I/DEBUG   ( 3037):  d14 0000000000000000  d15 0000000000000000
I/DEBUG   ( 3037):  d16 0000000000000000  d17 0000000000000000
I/DEBUG   ( 3037):  d18 0000000000000000  d19 0000000000000000
I/DEBUG   ( 3037):  d20 0000000000000000  d21 0000000000000000
I/DEBUG   ( 3037):  d22 0000000000000000  d23 0000000000000000
I/DEBUG   ( 3037):  d24 0000000000000000  d25 0000000000000000
I/DEBUG   ( 3037):  d26 0000000000000000  d27 0000000000000000
I/DEBUG   ( 3037):  d28 0000000000000000  d29 0000000000000000
I/DEBUG   ( 3037):  d30 0000000000000000  d31 0000000000000000
I/DEBUG   ( 3037):  scr 00000000
I/DEBUG   ( 3037):
I/DEBUG   ( 3037):          #00  pc 00011ca4  /system/lib/libc.so
I/DEBUG   ( 3037):          #01  pc 0000bdf2  /system/lib/libc.so
I/DEBUG   ( 3037):          #02  pc 0000cd52  /system/lib/libc.so
I/DEBUG   ( 3037):          #03  pc 00002770  /system/lib/libsysutils.so
I/DEBUG   ( 3037):          #04  pc 0000279c  /system/lib/libsysutils.so
I/DEBUG   ( 3037):          #05  pc 0000253a  /system/lib/libsysutils.so
I/DEBUG   ( 3037):          #06  pc 00001b16  /system/lib/libsysutils.so
I/DEBUG   ( 3037):          #07  pc 00001cae  /system/lib/libsysutils.so
I/DEBUG   ( 3037):          #08  pc 00010ee0  /system/lib/libc.so
I/DEBUG   ( 3037):          #09  pc 000109d0  /system/lib/libc.so
I/DEBUG   ( 3037):
I/DEBUG   ( 3037): code around pc:
I/DEBUG   ( 3037): afd11c84 2d00682d e029d1fb b12b68db c05cf8df
I/DEBUG   ( 3037): afd11c94 f8442001 4798000c e054f8df 26002227
I/DEBUG   ( 3037): afd11ca4 2000f88e eecaf7fb f7fc2106 f04feff8
I/DEBUG   ( 3037): afd11cb4 91035180 460aa901 96012006 f7fc9602
I/DEBUG   ( 3037): afd11cc4 a905eb6e 20024632 eb78f7fc eeb6f7fb
I/DEBUG   ( 3037):
I/DEBUG   ( 3037): code around lr:
I/DEBUG   ( 3037): deadba8c ffffffff ffffffff ffffffff ffffffff
I/DEBUG   ( 3037): deadba9c ffffffff ffffffff ffffffff ffffffff
I/DEBUG   ( 3037): deadbaac ffffffff ffffffff ffffffff ffffffff
I/DEBUG   ( 3037): deadbabc ffffffff ffffffff ffffffff ffffffff
I/DEBUG   ( 3037): deadbacc ffffffff ffffffff ffffffff ffffffff
I/DEBUG   ( 3037):
I/DEBUG   ( 3037): stack:
I/DEBUG   ( 3037):     100ffd28  00000000
I/DEBUG   ( 3037):     100ffd2c  00000000
I/DEBUG   ( 3037):     100ffd30  00000000
I/DEBUG   ( 3037):     100ffd34  00000000
I/DEBUG   ( 3037):     100ffd38  00000000
I/DEBUG   ( 3037):     100ffd3c  afd10290  /system/lib/libc.so
I/DEBUG   ( 3037):     100ffd40  afd43724  /system/lib/libc.so
I/DEBUG   ( 3037):     100ffd44  0000001b
I/DEBUG   ( 3037):     100ffd48  00012850  [heap]
I/DEBUG   ( 3037):     100ffd4c  4000805e
I/DEBUG   ( 3037):     100ffd50  100ffd7c
I/DEBUG   ( 3037):     100ffd54  afd42328  /system/lib/libc.so
I/DEBUG   ( 3037):     100ffd58  00000000
I/DEBUG   ( 3037):     100ffd5c  100ffd7c
I/DEBUG   ( 3037):     100ffd60  df002777
I/DEBUG   ( 3037):     100ffd64  e3a070ad
I/DEBUG   ( 3037): #00 100ffd68  afd438dc  /system/lib/libc.so
I/DEBUG   ( 3037):     100ffd6c  afd103ac  /system/lib/libc.so
I/DEBUG   ( 3037):     100ffd70  afd42328  /system/lib/libc.so
I/DEBUG   ( 3037):     100ffd74  aef04178  /system/lib/libsysutils.so
I/DEBUG   ( 3037):     100ffd78  000123e0  [heap]
I/DEBUG   ( 3037):     100ffd7c  fffffbdf
I/DEBUG   ( 3037):     100ffd80  afd42328  /system/lib/libc.so
I/DEBUG   ( 3037):     100ffd84  afd43724  /system/lib/libc.so
I/DEBUG   ( 3037):     100ffd88  000123e0  [heap]
I/DEBUG   ( 3037):     100ffd8c  afd0bdf7  /system/lib/libc.so
I/DEBUG   ( 3037): #01 100ffd90  00012850  [heap]
I/DEBUG   ( 3037):     100ffd94  afd1d595  /system/lib/libc.so
I/DEBUG   ( 3037):     100ffd98  000013fc
I/DEBUG   ( 3037):     100ffd9c  aef04178  /system/lib/libsysutils.so
I/DEBUG   ( 3037):     100ffda0  00012348  [heap]
I/DEBUG   ( 3037):     100ffda4  00012348  [heap]
I/DEBUG   ( 3037):     100ffda8  aef04178  /system/lib/libsysutils.so
I/DEBUG   ( 3037):     100ffdac  40008008
I/DEBUG   ( 3037):     100ffdb0  00000360
I/DEBUG   ( 3037):     100ffdb4  afd0cd55  /system/lib/libc.so

/system/bin/netd caused SEGV(egmentation fault). The address was 0xdeadbaad.

This is immediate analysis. Next, you want to know which route the problem come through, don’t you?

It is in the stacktrace. Look at this.

I/DEBUG   ( 3037):          #00  pc 00011ca4  /system/lib/libc.so
I/DEBUG   ( 3037):          #01  pc 0000bdf2  /system/lib/libc.so
I/DEBUG   ( 3037):          #02  pc 0000cd52  /system/lib/libc.so
I/DEBUG   ( 3037):          #03  pc 00002770  /system/lib/libsysutils.so
  ...

This means that SEGV occuerd at the instruction offseted 0x00011ca4 in /system/lib/libc.so and that is called from one offseted 0x000bdf2 in the same libc.so.
Then, called from offseted 0x0000cd52 in libc.so, called from offset 0×00002770 in /system/lib/libsysutils.so.

Next step, Let’s find where are these instructions in the source code. objdump command show us.

At first, move to the directory where object file with debug information are. In the case ofKZM-A9-Dual,

$ cd $(ANDROID_TOP)/out/target/product/kmz9d/symbols/system/lib

The object files here have information which maps addresses and source code lines.

$ arm-eabi-objdump -S libc.so |less

Put -S option to objdump command to show disasemble with source code.
Then search string “11ca4″, and investigate what was doing there.

    /* temporary, for bug hunting */
    /* seg fault seems to produce better debuggerd results than SIGABRT */
    *((char*)0xdeadbaad) = 39;
   11ca4:       f88e 2000       strb.w  r2, [lr]
    /* -- */

Oh, this code intentionaly write immediate value 39 to address 0xdeadbaad to cause SEGV. And you find this is in bionic/libc/unistd/abort.c

In the same manner, search string “bdf2″ in libc.so.

    bdec:       2300            movs    r3, #0
    bdee:       6183            str     r3, [r0, #24]
    bdf0:       e001            b.n     bdf6 <dlfree+0x532>
          check_free_chunk(fm, p);
          goto postaction;
        }
      }
    erroraction:
      USAGE_ERROR_ACTION(fm, p);
    bdf2:       f005 ff25       bl      11c40 <abort>
    postaction:
      POSTACTION(fm);
    bdf6:       4907            ldr     r1, [pc, #28]   (be14 )
    bdf8:       1864            adds    r4, r4, r1

Surely it calls abort. This is in the function dlfree.

Then search string “cd52″.

0000cd3c <free>:
void free(void* mem) {
    __libc_malloc_dispatch->free(mem);
    cd3c:       4b06            ldr     r3, [pc, #24]   (cd58 <free+0x1c>)
    cd3e:       bf00            nop
    cd40:       a100            add     r1, pc, #0      (adr r1, cd44
)
    cd42:       4a06            ldr     r2, [pc, #24]   (cd5c <free+0x20>)
    cd44:       185b            adds    r3, r3, r1
    cd46:       b510            push    {r4, lr}
    cd48:       f853 c002       ldr.w   ip, [r3, r2]
    cd4c:       f8dc 1000       ldr.w   r1, [ip]
    cd50:       684b            ldr     r3, [r1, #4]
    cd52:       4798            blx     r3
}
    cd54:       bd10            pop     {r4, pc}
    cd56:       bf00            nop
    cd58:       000355e4        .word   0x000355e4
    cd5c:       00000054        .word   0x00000054

The fuction dlfree is called from free.

Let’s sail up the river more. Do objdump libsysutils.so. This library is written in C++, put -C option to objdump to demangle C++ symboles.

$ arm-eabi-objdump -S -C libsysutils.so |less

“2770″ is not found. But found near address.
(Offset 0c00002770 is middle of instruction. I think this is bug of stack trace. 4byte instruction of Thumb2 seems to be not considered.)

NetlinkEvent::~NetlinkEvent() {
    276a:       6022            str     r2, [r4, #0]
    int i;
    if (mPath)
    276c:       b108            cbz     r0, 2772 <NetlinkEvent::~NetlinkEvent()+0x1e>
        free(mPath);
    276e:       f7ff e848       blx     1800 <SocketListener::sendBroadcast(char const*)-0xc>
    if (mSubsystem)
    2772:       6920            ldr     r0, [r4, #16]
    2774:       b108            cbz     r0, 277a <NetlinkEvent::~NetlinkEvent()+0x26>
        free(mSubsystem);
    2776:       f7ff e844       blx     1800 <SocketListener::sendBroadcast(char const*)-0xc>
    277a:       2500            movs    r5, #0

free(mPath) is used in the deconstractor of NetlinkEvent. Probabley mPath has illegal value.

So then, using these information you can find what triggered this problem.

Let’s summarize.

NetlinkEvent::~NetlinkEvent() :free(mPath);
  free
    dlfree
      abort
        (SEGV here.)

Felipe Monteiro de Carvalho

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To post to this group, send email to andro...@googlegroups.com.
To unsubscribe from this group, send email to android-ndk...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/android-ndk?hl=en.

--

HimHim

[Felipe Monteiro de Carvalho]

Thanks a lot! This extensive instructions got me going and I advanced
quite a bit and learned how to get the addresses from my code.

Indeed objdump works a lot better then addr2line and gave me fine
results for my own code and even shows the surrounding Pascal code.
And it seams to work for both stabs and dwarf2

I am using this command (without -C since my code is Pascal not C++):

[felipe@localhost android]$ ~/Programas/android-ndk-r7/toolchains/arm-
linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-androideabi-

objdump -S libs/armeabi/liblclapp.so > lclappsym2.txt

But I find myself unable to find the debug symbols from HTC Wildfire
for libc.so. If anyone knows where to download them it will be great.
I am searching Google but haven't found anything so far...

thanks,

Felipe Monteiro de Carvalho