ARM binary instrumentation on Android with DynamoRIO

Caroline Trippel

unread,

Apr 19, 2016, 12:48:57 AM4/19/16

to DynamoRIO Users

I have seemingly successfully built the cross-compiled android version of DynamoRIO on a VM that is running Ubuntu. Additionally, I am doing my testing on a Nexus 6 which has ARM cores. I began by following the installation and build steps outlined in the repo (https://github.com/DynamoRIO/dynamorio/wiki/How-To-Build and https://github.com/DynamoRIO/dynamorio/wiki/Test-Suite), specifically:

$ git clone https://github.com/DynamoRIO/dynamorio.git

$ mkdir build_android

$ cd build_android

$ cmake -DCMAKE_TOOLCHAIN_FILE=../dynamorio/make/toolchain-android.cmake -DANDROID_TOOLCHAIN=/mytooldir/android-ndk-21 -DBUILD_TESTS=ON -DDR_COPY_TO_DEVICE=ON ../dynamorio

make -j6

make test

My understanding is that the “make test” command is meant to build and run a series of provided example tests with the help of adb shell and drrun (the DynamoRIO run executable). All but the first of theses tests fail. I then created a “hello world” ARM binary (hello) and successfully ran it using adb shell as a sanity test which worked.

My next step was to adb shell into the phone and work interactively from there. I tried running the phone-local drrun executable. It seemed to run and correctly accept arguments; however, when I run something like drrun -- ls or drrun -- hello a segmentation fault results. The main questions I have right now are:

1. How can run a provided instrumentation tool (for example the inscount tool for instruction counting) on my hello binary?

2. How can I write my own instrumentation tool to inject assembly instructions of my choosing.

Thanks for any help!

Derek Bruening

unread,

Apr 19, 2016, 10:35:44 AM4/19/16

to dynamor...@googlegroups.com

On Tue, Apr 19, 2016 at 12:48 AM, Caroline Trippel <carolin...@gmail.com> wrote:

My understanding is that the “make test” command is meant to build and run a series of provided example tests with the help of adb shell and drrun (the DynamoRIO run executable). All but the first of theses tests fail.

We expect the tests to pass, and they do pass on our devices. However, we are not able to test every version of Android on every device, and need the community's help to diagnose issues that are hit in other environments. See below for debugging information.

My next step was to adb shell into the phone and work interactively from there. I tried running the phone-local drrun executable. It seemed to run and correctly accept arguments; however, when I run something like drrun -- ls or drrun -- hello a segmentation fault results.

We expect this to work. Please debug following https://github.com/DynamoRIO/dynamorio/wiki/Debugging: using debug build, run under gdb (for Android, use the gdb server) and load the symbols to see where the fault is. If the fault is in the code cache, use -loglevel 4 to see the mangled version of the block that faulted. Sharing the fault callstack and the last block here may enable us to diagnose the issue remotely.

1. How can run a provided instrumentation tool (for example the inscount tool for instruction counting) on my hello binary?

As the README and drrun help shows, pass -c and thenthe path to the tool library. But if DR with no tool is crashing then this will likely do the same thing until that's fixed.

2. How can I write my own instrumentation tool to inject assembly instructions of my choosing.

See our latest tutorial slides at https://github.com/DynamoRIO/dynamorio/wiki/Downloads.

Qin Zhao

unread,

Apr 19, 2016, 11:09:25 AM4/19/16

to dynamor...@googlegroups.com

There is no significant difference (other than architectural specific code) writing a client for x86 on Linux, for ARM on Linux, or for ARM on Android.

For easy of development, you might want to get a ARM based Linux system, e.g. a Chromebook, and develop and test your code there first.

Then you can focus on Android specific problems while running on Android system.

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To post to this group, send email to dynamor...@googlegroups.com.
Visit this group at https://groups.google.com/group/dynamorio-users.
For more options, visit https://groups.google.com/d/optout.

--

Interested in Yoga? Be careful of The Yoga Cult or The Scary Yoga Obsession.

More information from Lorie Anderson and Rick Ross.

Caroline Trippel

unread,

Apr 19, 2016, 3:12:56 PM4/19/16

to DynamoRIO Users

On Tuesday, April 19, 2016 at 10:35:44 AM UTC-4, Derek Bruening wrote:

On Tue, Apr 19, 2016 at 12:48 AM, Caroline Trippel <carolin...@gmail.com> wrote:

My understanding is that the “make test” command is meant to build and run a series of provided example tests with the help of adb shell and drrun (the DynamoRIO run executable). All but the first of theses tests fail.
We expect the tests to pass, and they do pass on our devices. However, we are not able to test every version of Android on every device, and need the community's help to diagnose issues that are hit in other environments. See below for debugging information.
My next step was to adb shell into the phone and work interactively from there. I tried running the phone-local drrun executable. It seemed to run and correctly accept arguments; however, when I run something like drrun -- ls or drrun -- hello a segmentation fault results.
We expect this to work. Please debug following https://github.com/DynamoRIO/dynamorio/wiki/Debugging: using debug build, run under gdb (for Android, use the gdb server) and load the symbols to see where the fault is. If the fault is in the code cache, use -loglevel 4 to see the mangled version of the block that faulted. Sharing the fault callstack and the last block here may enable us to diagnose the issue remotely.

Thanks for the help. When I run in debug mode, I am seeing the following:

ctrippel@ctrippel-VirtualBox:/usr/local/tools/build_android$ adb shell /data/local/tmp/build_android/bin32/drrun -debug -- /data/local/tmp/hello

<Application /data/local/tmp/build_android/lib32/debug/libdynamorio.so (5744). Internal Error: DynamoRIO debug check failure: failed to find envp @/usr/local/tools/dynamorio/core/unix/os.c:720 envp != NULL

(Error occurred @-1 frags)

version 6.1.16905, custom build

0xb6cf9808 0x00000000

0xb6ce7144 0x68636769>

The /usr/local/tools/dynamorio/core/unix/os.c path is on my host machine while the /data/local/tmp is on the Android device. Could this be causing an issue?

Derek Bruening

unread,

Apr 19, 2016, 6:57:39 PM4/19/16

to dynamor...@googlegroups.com

On Tue, Apr 19, 2016 at 3:12 PM, Caroline Trippel <carolin...@gmail.com> wrote:

Thanks for the help. When I run in debug mode, I am seeing the following:

ctrippel@ctrippel-VirtualBox:/usr/local/tools/build_android$ adb shell /data/local/tmp/build_android/bin32/drrun -debug -- /data/local/tmp/hello

<Application /data/local/tmp/build_android/lib32/debug/libdynamorio.so (5744). Internal Error: DynamoRIO debug check failure: failed to find envp @/usr/local/tools/dynamorio/core/unix/os.c:720 envp != NULL
(Error occurred @-1 frags)
version 6.1.16905, custom build

0xb6cf9808 0x00000000
0xb6ce7144 0x68636769>

Are you sure that you switched from the gold linker to the bfd linker as the wiki instructs? The _init routine should not be called, and the only reason I can think of is that the gold linker was used and it put in a PT_INTERP entry which we do not want.

What happens when you run with the 6.1.1-3 release package?

Caroline Trippel

unread,

Apr 20, 2016, 10:09:24 AM4/20/16

to DynamoRIO Users

I double checked that the I had switched the symbolic link to point to the bfd linker -- I had. I even tried rebuilding the whole thing more than once. I'm not sure where I went wrong... The good news: I didn't realize there were pre-built past releases. I was able to successfully run the 6.1.1-3 release. I should be able to write my own test from the samples and tutorial now. Thanks for your help!

Derek Bruening

unread,

Apr 20, 2016, 10:21:43 AM4/20/16

to dynamor...@googlegroups.com

It's good that the release package works. We would like to understand why your toolchain had this problem, so that we can update the wiki instructions if necessary. Can you confirm that the problem is the PT_INTERP? If you run:

% readelf -l lib32/release/libdynamorio.so

Do you see a program header labeled INTERP? Something like this:

INTERP 0x000154 0x00000154 0x00000154 0x00013 0x00013 R 0x1

[Requesting program interpreter: /system/bin/linker]

Also, what is the output from:

% /mytooldir/android-ndk-21/bin/arm-linux-androideabi-ld --version

Caroline Trippel

unread,

Apr 20, 2016, 8:57:12 PM4/20/16

to DynamoRIO Users

I do see that:

Elf file type is DYN (Shared object file)

Entry point 0xd285c

There are 7 program headers, starting at offset 52

Program Headers:

Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align

PHDR 0x000034 0x00000034 0x00000034 0x000e0 0x000e0 R 0x4

INTERP 0x000114 0x00000114 0x00000114 0x00013 0x00013 R 0x1

[Requesting program interpreter: /system/bin/linker]

LOAD 0x000000 0x00000000 0x00000000 0xfb700 0xfb700 R E 0x1000

LOAD 0x0fc52c 0x000fd52c 0x000fd52c 0x51b04 0x73a22 RW 0x1000

DYNAMIC 0x139bd4 0x0013abd4 0x0013abd4 0x000a8 0x000a8 RW 0x4

GNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RW 0

GNU_RELRO 0x0fc52c 0x000fd52c 0x000fd52c 0x3dad4 0x3dad4 RW 0x4

Section to Segment mapping:

Segment Sections...

00

01 .interp

02 .interp .dynsym .dynstr .hash .rel.dyn .text .rodata

03 .data.rel.ro.local .data.rel.ro .dynamic .got .data .fspdata .cspdata .nspdata .bss

04 .dynamic

05

06 .data.rel.ro.local .data.rel.ro .dynamic .got

Also, what is the output from:

% /mytooldir/android-ndk-21/bin/arm-linux-androideabi-ld --version

root@ctrippel-VirtualBox:/usr/local/tools# ./android-ndk-21/bin/arm-linux-androideabi-ld --version

GNU ld (binutils-2.25-48ba864) 2.25.51.20141117

This program is free software; you may redistribute it under the terms of

the GNU General Public License version 3 or (at your option) a later version.

This program has absolutely no warranty.

Also confirming it is pointing to the correct version:

root@ctrippel-VirtualBox:/usr/local/tools# ls -l ./android-ndk-21/bin/arm-linux-androideabi-ld

lrwxrwxrwx 1 ctrippel ctrippel 28 Apr 19 21:20 ./android-ndk-21/bin/arm-linux-androideabi-ld -> arm-linux-androideabi-ld.bfd

Another question:

I was able to run the test provided in samples/bin32 of the DynamoRIO-ARM-Android-EABI-6.1.1-3 release on my Android device successfully. However, I'm having a bit of trouble using cmake to compile my own instrumentation library. I found these instructions:

mkdir build
cd build
CFLAGS=-m32 CXXFLAGS=-m32 cmake -DDynamoRIO_DIR=$DYNAMORIO_HOME/cmake $DYNAMORIO_HOME/samples
make instrcalls

However, I'm wondering how to specify that I'm building for Android. This make command is giving me some issues.

Thanks!

Caroline Trippel

unread,

Apr 20, 2016, 10:41:17 PM4/20/16

to DynamoRIO Users

Actually, I believe I have this working for me now. Although, the dynamorio/make/toolchain-android.cmake file was not included in the release, just in the repo.

cmake -DCMAKE_TOOLCHAIN_FILE=/mytools/dynamorio/make/toolchain-android.cmake -DANDROID_TOOLCHAIN=/mytools/android-ndk-21 samples/

Derek Bruening

unread,

Apr 20, 2016, 11:05:31 PM4/20/16

to dynamor...@googlegroups.com

I filed https://github.com/DynamoRIO/dynamorio/issues/1927 on the ld.bfd issue.

If you'd like to send a patch adding Android information to the samples.dox documentation we would welcome the contribution (see https://github.com/DynamoRIO/dynamorio/wiki/Contributing).

--

Caroline Trippel

unread,

Apr 20, 2016, 11:18:40 PM4/20/16

to DynamoRIO Users

On Wednesday, April 20, 2016 at 11:05:31 PM UTC-4, Derek Bruening wrote:

I filed https://github.com/DynamoRIO/dynamorio/issues/1927 on the ld.bfd issue.

If you'd like to send a patch adding Android information to the samples.dox documentation we would welcome the contribution (see https://github.com/DynamoRIO/dynamorio/wiki/Contributing).

Great, I'll do that.

Caroline Trippel

unread,

Apr 25, 2016, 11:19:29 AM4/25/16

to DynamoRIO Users

Hi Derek -- I'm working on getting something running for a deadline next week and then I'll get a more detailed documentation patch up. I'm having another issue. I've successfully been able to write an instrumentation tool; however, when I try to run any instrumentation tool (including the ones provided, i.e., libinscount.so) on a multi-threaded program, I get an internal crash/program aborted. Is there something special I should be doing to instrument, say, a 2-thread program?

Qin Zhao

unread,

Apr 25, 2016, 12:02:24 PM4/25/16

to dynamor...@googlegroups.com

There might be some bug handling multi-threaded application.

Could you please file an issue with detailed information, e.g., OS, command, output, etc., ideally with a simplified program that can reproduce the issue easily?

Derek Bruening

unread,

Apr 25, 2016, 12:24:24 PM4/25/16

to dynamor...@googlegroups.com

On Mon, Apr 25, 2016 at 11:19 AM, Caroline Trippel <carolin...@gmail.com> wrote:

Hi Derek -- I'm working on getting something running for a deadline next week and then I'll get a more detailed documentation patch up. I'm having another issue. I've successfully been able to write an instrumentation tool; however, when I try to run any instrumentation tool (including the ones provided, i.e., libinscount.so) on a multi-threaded program, I get an internal crash/program aborted. Is there something special I should be doing to instrument, say, a 2-thread program?

All of the multi-threaded tests in our test suite on Android pass, including those with clients (though admittedly there are not many client+thread tests as we have not been able to port the entire test suite to ARM/Android yet: contributions appreciated :)). Details would help to diagnose the issue: callstack of the crash, debug build results, etc.

Caroline Trippel

unread,

Apr 25, 2016, 1:15:08 PM4/25/16

to DynamoRIO Users

NOTE: I'm running everything inside adb shell.

I attached a simple tool here. I'm able to run the provided multithread libraries, e.g. memtrace_simple.so, on both a single threaded ARM binary and a simple 2-thread ARM binary.

I am able to run the attached library on both single-thread and 2-thread ARM binaries when I comment out the "preinsert" line.

However, with the "preinsert" line included, I am only able to run the llfence library on a single threaded program. The output for the 2-thread program is:

255|root@shamu:/data/local/tmp # ./build_android/bin32/drrun -verbose -c libl>

INFO: client 0 path: /data/local/tmp/libllfence.so

INFO: targeting application: "/data/local/tmp/a.out"

INFO: app cmdline: "./a.out"

INFO: configuration directory is "/data/local/tmp/.dynamorio"

INFO: will exec /data/local/tmp/a.out

INFO: registering client with id=0 path=|/data/local/tmp/libllfence.so| ops=||

<Application /data/local/tmp/a.out (5459). DynamoRIO Sample Client 'llfence' internal crash at PC 0xb6f29960. Please report this at http://dynamorio.org/issues. Program aborted.

Received SIGSEGV at pc 0xb6f29960 in thread 5459

Base: 0xb6e57000

Registers: r0 =0x00000000 r1 =0xb6dcdca6 r2 =0x00000004 r3 =0x00000011

r4 =0x00000000 r5 =0x5397ef48 r6 =0x00000000 r7 =0x00000000

r8 =0

version 6.1.1, build 3

-no_dynamic_options -client_lib '/data/local/tmp/libllfence.so;0;' -code_api -stack_size 56K -max_elide_jmp 0 -max_elide_call 0 -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >

Am I missing something when it comes to inserting instructions into a multi-threaded program?

Thanks!

llfence.c

Derek Bruening

unread,

Apr 25, 2016, 4:40:15 PM4/25/16

to dynamor...@googlegroups.com

It may have nothing to do with threads: probably the 2-thread binary simply contains additional opcode sequences not present in the smaller binary. For example, DMB has no 16-bit Thumb encoding, so perhaps you're trying to insert it inside an IT block, though that should result in an encoding message. Use debug DR with logging. For small apps, run with "-loglevel 3" and you can see precisely which block it last instrumented. Attach gdb at the crash and get a callstack. Using logging and the debugger will be indispensable steps in diagnosing any issue with these types of low-level tools.

Qin Zhao

unread,

Apr 25, 2016, 6:02:13 PM4/25/16

to dynamor...@googlegroups.com

It may have nothing to do with threads: probably the 2-thread binary simply contains additional opcode sequences not present in the smaller binary. For example, DMB has no 16-bit Thumb encoding, so perhaps you're trying to insert it inside an IT block, though that should result in an encoding message.

It seems an IT block could contain 32-bit Thumb instructions, so that should not be a problem.

Use debug DR with logging. For small apps, run with "-loglevel 3" and you can see precisely which block it last instrumented. Attach gdb at the crash and get a callstack. Using logging and the debugger will be indispensable steps in diagnosing any issue with these types of low-level tools.

--

You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To post to this group, send email to dynamor...@googlegroups.com.
Visit this group at https://groups.google.com/group/dynamorio-users.
For more options, visit https://groups.google.com/d/optout.

Message has been deleted

Caroline Trippel

unread,

Apr 26, 2016, 5:51:07 PM4/26/16

to DynamoRIO Users

I just posted a reply, but it appears to have been deleted. After some more debugging, it seems I was not setting the translation field. I'm still having issues with crashing, so I wanted to see if what I'm doing makes sense. I do; however, know that the proper dmb instructions are being inserted, as several basic blocks are instrumented before failure.

On each basic block, I do the following:

static dr_emit_flags_t

event_basic_block(void *drcontext, void *tag, instrlist_t *bb,

bool for_trace, bool translating)

{

instr_t *instr, *next_instr;

dr_printf("in dynamorio_basic_block(tag="PFX")\n", tag);

instrlist_disassemble(drcontext, tag, bb, STDOUT);

for (instr = instrlist_first_app(bb); instr != NULL; instr = next_instr) {

/* grab next now so we don't go over instructions we insert */

next_instr = instr_get_next_app(instr);

if (instr_reads_memory(instr)) {

instr_t* dmb = INSTR_CREATE_dmb(drcontext, opnd_create_immed_uint(15, OPSZ_4b));

instrlist_preinsert(bb, instr, instr_set_translation(dmb, instr_get_app_pc_instr(instr));; // is this correct for the app_pc field?

}

dr_printf("Finished instrumenting dynamorio_basic_block(tag="PFX")\n", tag);

instrlist_disassemble(drcontext, tag, bb, STDOUT);

return DR_EMIT_STORE_TRANSLATIONS; // is this correct for the return statement?

//return DR_EMIT_DEFAULT;

}

I added a couple comments/questions in the code segment; however this still crashes. When I run this with debug, I'm getting:

<Application /system/bin/toolbox (19575). Internal Error: DynamoRIO debug check failure: /dynamorio_package/core/translate.c:1455 app != NULL || instr_is_meta(inst)

(Error occurred @3 frags)

version 6.1.1, build 3

This is the last basic block that is instrumented:

in dynamorio_basic_block(tag=0xb6c29008)

TAG 0xb6c29008

+0 L3 b510 stmdb %r4 %lr %sp -> -0x08(%sp)[8byte] %sp

+2 L3 4c15 ldr +0x54(%pc)[4byte] -> %r4

+4 L3 447c add %r4 %pc -> %r4

+6 L3 7823 ldrb (%r4)[1byte] -> %r3

+8 L3 bb03 cbnz $0xb6c29054 %r3

END 0xb6c29008

Finished instrumenting dynamorio_basic_block(tag=0xb6c29008)

TAG 0xb6c29008

+0 L3 b510 stmdb %r4 %lr %sp -> -0x08(%sp)[8byte] %sp

+2 L4 @0x52f866a0 f3bf 8f5f dmb $0x0f

+6 L3 4c15 ldr +0x54(%pc)[4byte] -> %r4

+8 L3 447c add %r4 %pc -> %r4

+10 L4 @0x52f86278 f3bf 8f5f dmb $0x0f

+14 L3 7823 ldrb (%r4)[1byte] -> %r3

+16 L3 bb03 cbnz $0xb6c29054 %r3

END 0xb6c29008

Do you have any suggestions? Also, is there any way to omit library code from instrumentation?

Thanks!

Caroline Trippel

unread,

Apr 26, 2016, 9:37:05 PM4/26/16

to dynamor...@googlegroups.com

For some reason, my messages are being deleted from the Google group. Here is the second message I posted:

You received this message because you are subscribed to a topic in the Google Groups "DynamoRIO Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dynamorio-users/eL1__o7m4AQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dynamorio-use...@googlegroups.com.

Derek Bruening

unread,

Apr 26, 2016, 10:44:28 PM4/26/16

to dynamor...@googlegroups.com

On Tue, Apr 26, 2016 at 5:51 PM, Caroline Trippel <carolin...@gmail.com> wrote:

I just posted a reply, but it appears to have been deleted.

There is a spam filter that responses sometimes get caught in and we have to go free them.

After some more debugging, it seems I was not setting the translation field. I'm still having issues with crashing, so I wanted to see if what I'm doing makes sense. I do; however, know that the proper dmb instructions are being inserted, as several basic blocks are instrumented before failure.

if (instr_reads_memory(instr)) {

instr_t* dmb = INSTR_CREATE_dmb(drcontext, opnd_create_immed_uint(15, OPSZ_4b));

instrlist_preinsert(bb, instr, instr_set_translation(dmb, instr_get_app_pc_instr(instr));; // is this correct for the app_pc field?

Yes, this seems reasonable. The way to think about it is, if there were a fault here or signal that arrived here, what PC should the application see? If DR wants to relocate a thread currently at this location, at which PC should it start execution?

return DR_EMIT_STORE_TRANSLATIONS; // is this correct for the return statement?

//return DR_EMIT_DEFAULT;

Since your instrumentation is deterministic, DR_EMIT_DEFAULT is fine.

I added a couple comments/questions in the code segment; however this still crashes. When I run this with debug, I'm getting:

<Application /system/bin/toolbox (19575). Internal Error: DynamoRIO debug check failure: /dynamorio_package/core/translate.c:1455 app != NULL || instr_is_meta(inst)

I assume that was before you set the translation field.

What does the debug build say now? If debug crashes, where is the crash : code cache, client code, DR code? If client or DR, what is the callstack? Does this app have a handled fault early on (explaining the problem when no translation was set)?

Caroline Trippel

unread,

Apr 27, 2016, 11:29:01 AM4/27/16

to DynamoRIO Users

Hi Derek,

So, I think I have my tool figured out. I'm printing the BB assembly both before and after instrumentation and it appears to be inserting instructions in the correct places. And if my understanding is correct, this should be dynamically inserting these new instructions into the actual app instruction stream. Is this right?

Anyways, I am able to run my tool on a simple single-threaded hello world app, the Android-ARM ls binary and another 2-thread (one main thread and one child thread) app. However, if I try to run my library (or any library for that matter) or just drrun with no client on an app that has 3 threads (main + 2 children), I get a segmentation fault. This happens with even the very simple .cpp program that I've also attached here. Is DR only able to handle 2 threads? Are there some compiler flags I'm missing? This happens with both dynamic and static linking.

Thanks!

Caroline

simple.cpp

llfence.c

Derek Bruening

unread,

Apr 27, 2016, 12:21:20 PM4/27/16

to dynamor...@googlegroups.com

On Wed, Apr 27, 2016 at 11:29 AM, Caroline Trippel <carolin...@gmail.com> wrote:

So, I think I have my tool figured out. I'm printing the BB assembly both before and after instrumentation and it appears to be inserting instructions in the correct places. And if my understanding is correct, this should be dynamically inserting these new instructions into the actual app instruction stream. Is this right?

Yes, use -loglevel 4 to see the exact code run in the code cache.

Anyways, I am able to run my tool on a simple single-threaded hello world app, the Android-ARM ls binary and another 2-thread (one main thread and one child thread) app. However, if I try to run my library (or any library for that matter) or just drrun with no client on an app that has 3 threads (main + 2 children), I get a segmentation fault. This happens with even the very simple .cpp program that I've also attached here. Is DR only able to handle 2 threads? Are there some compiler flags I'm missing? This happens with both dynamic and static linking.

There are no scalability limits, no.

Your C++11 thread app "simple.cpp" runs fine for me with plain DR or a client, so I cannot reproduce any problem. "a segmentation fault" is just not enough information to act on: please diagnose via debug build, logs, callstacks: *where* is the fault.

# /work/toolchain/android-ndk-21/bin/arm-linux-androideabi-g++ -std=c++11 -g -fpie -pie -o cppthread simple.cpp

# adb push cppthread /data/local/tmp/

# adb shell

/data/local/tmp/DynamoRIO-ARM-Android-EABI-6.1.1-3 # bin32/drrun -stderr_mask 15 -c samples/bin32/libopcodes.so -- ../cppthread

Client opcodes is running

Hello from main

Greetings from thread2

Hi from thread1

/data/local/tmp/DynamoRIO-ARM-Android-EABI-6.1.1-3 # bin32/drrun -stderr_mask 15 -c samples/bin32/libopcodes.so -- ../cppthread

Client opcodes is running

Hello from main

Greetings from thread2

Hi from thread1

Top 15 opcode execution counts in 32-bit ARM mode:

14800 : bic

15988 : strd

17084 : uxtb

17467 : ldrb

18951 : lsr

20640 : ands

21013 : mov

21618 : rev

22268 : tst

31843 : subs

38500 : ldr

41264 : sel

41264 : uadd8

52241 : ldrd

62624 : pld

125121 : cmp

179644 : b

Top 15 opcode execution counts in 32-bit Thumb mode:

62396 : cbnz

63016 : bx

97261 : ldm

97612 : stmdb

112748 : movs

144979 : eor

150978 : eors

155834 : bl

181485 : and

225147 : str

307302 : cmp

331952 : ldrb

471425 : cbz

597670 : mov

716760 : add

727564 : b

792575 : ldr

/data/local/tmp/DynamoRIO-ARM-Android-EABI-6.1.1-3 #

Caroline Trippel

unread,

Apr 27, 2016, 2:04:57 PM4/27/16

to DynamoRIO Users

So, I re-compiled my simple.cpp file the exact way you did above. I then ran it two times, both in debug mode. It completed the first time, and experienced a segmentation fault the second. The results are below. Also when trying to used gdbserver/gdb, I'm having a little trouble loading shared libraries. Should these paths in gdb be set to point locally or remotely on the phone?

139|root@shamu:/data/local/tmp # ./build_android/bin32/drrun -debug -stderr_mask 15 -c samples/bin32/libopcodes.so -- ./cppthread>

<Paste into GDB to debug DynamoRIO clients:

set confirm off

add-symbol-file '/data/local/tmp/samples/bin32/libopcodes.so' 0xb6c23700

add-symbol-file '/data/local/tmp/build_android/lib32/debug/libdynamorio.so' 0x00021508

add-symbol-file '/data/local/tmp/build_android/ext/lib32/debug/libdrx.so' 0xb6c111a4

add-symbol-file '/data/local/tmp/build_android/ext/lib32/debug/libdrreg.so' 0xb6bfe9b8

add-symbol-file '/data/local/tmp/build_android/ext/lib32/debug/libdrmgr.so' 0xb6bebd64

add-symbol-file '/system/lib/libc.so' 0xb6b84ef0

add-symbol-file '/system/lib/libdl.so' 0xb6b6f56c

>

Client opcodes is running

<get_memory_info mismatch! (can happen if os combines entries in /proc/pid/maps)

os says: 0xb6602000-0xb6700000 prot=0x00000003

cache says: 0xb6602000-0xb6900000 prot=0x00000003

>

Hi from thread1

Hello from main

Greetings from thread2

Top 15 opcode execution counts in 32-bit ARM mode:

4518 : uxtb

4757 : ldrb

4778 : bic

5218 : lsr

5358 : bx

5868 : rev

5987 : orr

6309 : ands

6762 : tst

7848 : ldrd

9181 : subs

9726 : mov

11117 : add

12060 : pld

17260 : ldr

22457 : cmp

43383 : b

Top 15 opcode execution counts in 32-bit Thumb mode:

7144 : stmdb

8061 : bl

8449 : cbnz

8729 : sub

9068 : movs

14557 : adds

15959 : and

16104 : b

21787 : ldrb

22752 : eor

26512 : cbz

33856 : str

54397 : mov

71868 : add

74465 : cmp

106325 : b

158566 : ldr

root@shamu:/data/local/tmp # ./build_android/bin32/drrun -debug -stderr_mask 15 -c samples/bin32/libopcodes.so -- ./cppthread>

<Paste into GDB to debug DynamoRIO clients:

set confirm off

add-symbol-file '/data/local/tmp/samples/bin32/libopcodes.so' 0xb6c39700

add-symbol-file '/data/local/tmp/build_android/lib32/debug/libdynamorio.so' 0x00021508

add-symbol-file '/data/local/tmp/build_android/ext/lib32/debug/libdrx.so' 0xb6c271a4

add-symbol-file '/data/local/tmp/build_android/ext/lib32/debug/libdrreg.so' 0xb6c149b8

add-symbol-file '/data/local/tmp/build_android/ext/lib32/debug/libdrmgr.so' 0xb6c01d64

add-symbol-file '/system/lib/libc.so' 0xb6b9aef0

add-symbol-file '/system/lib/libdl.so' 0xb6b8556c

>

Client opcodes is running

<get_memory_info mismatch! (can happen if os combines entries in /proc/pid/maps)

os says: 0xb6602000-0xb6700000 prot=0x00000003

cache says: 0xb6602000-0xb6900000 prot=0x00000003

>

Segmentation fault

139|root@shamu:/data/local/tmp #

Caroline Trippel

unread,

Apr 27, 2016, 2:15:59 PM4/27/16

to DynamoRIO Users

Just to confirm, which version of the toolchain are you using? I'm currently using drrun from https://github.com/DynamoRIO/dynamorio/releases/download/release_6_1_1/DynamoRIO-ARM-Android-EABI-6.1.1-3.tar.gz and arm-linux-androideabi-g++ build for Android platform 21, gcc version 4.9.

Caroline Trippel

unread,

Apr 27, 2016, 4:28:05 PM4/27/16

to DynamoRIO Users

Right now gdb is not even letting me step through instructions as it says it cannot find the bounds of the function. Also, it appears to seg fault as soon as the third thread is created.

ctrippel$ ~/android-ndk-r11c/prebuilt/darwin-x86_64/bin/gdb

GNU gdb (GDB) 7.10

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law. Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-apple-darwin14.5.0".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at:

<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".

Type "apropos word" to search for commands related to "word".

(gdb) target remote :1234

Remote debugging using :1234

warning: Architecture rejected target-supplied description

Reading /data/local/tmp/build_android/bin32/drrun from remote target...

warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.

Reading /data/local/tmp/build_android/bin32/drrun from remote target...

Reading symbols from target:/data/local/tmp/build_android/bin32/drrun...Reading /data/local/tmp/build_android/bin32/drrun.debug from remote target...

Reading /data/local/tmp/build_android/bin32/drrun.debug from remote target...

Reading symbols from target:/data/local/tmp/build_android/bin32/drrun.debug...done.

done.

0xb6ff0a18 in ?? ()

(gdb) s

Cannot find bounds of current function

(gdb) continue

Continuing.

Program received signal SIGINT, Interrupt.

0xb6ff0a18 in ?? ()

(gdb) continue

Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap.

0x2a20f5f8 in ?? ()

(gdb) cont

Continuing.

[New Thread 3910]

[New Thread 3909]

Program received signal SIGSEGV, Segmentation fault.

[Switching to Thread 3910]

0x2a21118e in ?? ()

(gdb)

Caroline Trippel

unread,

Apr 27, 2016, 5:47:52 PM4/27/16

to DynamoRIO Users

One additional observation: the segmentation fault seems to be happening when switching back to the main thread. E.g., output from another run of gdb on a different binary this time:

(gdb) cont

Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap.

0x2a20f5f8 in ?? ()

(gdb) n

Cannot find bounds of current function

(gdb) cont

Continuing.

[New Thread 4190]

[New Thread 4188]

[New Thread 4189]

Program received signal SIGSEGV, Segmentation fault.

[Switching to Thread 4190]

0x2a21118e in ?? ()

Qin Zhao

unread,

Apr 28, 2016, 11:17:30 AM4/28/16

to dynamor...@googlegroups.com

You should use the path on your desktop (locally), because gdb on your desktop loading those libraries, parsing the debug information.

<Initial options = -no_dynamic_options -client_lib '/data/local/tmp/samples/bin32/libopcodes.so;0;' -code_api -stack_size 56K -max_elide_jmp 0 -max_elide_call 0 -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >

Client opcodes is running

<get_memory_info mismatch! (can happen if os combines entries in /proc/pid/maps)

os says: 0xb6602000-0xb6700000 prot=0x00000003

cache says: 0xb6602000-0xb6900000 prot=0x00000003

>

It is common in Linux, not sure about Android.

Did you try run with -loglevel 4, which should create a lot of logging to help identify the problem?

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To post to this group, send email to dynamor...@googlegroups.com.
Visit this group at https://groups.google.com/group/dynamorio-users.
For more options, visit https://groups.google.com/d/optout.

Qin Zhao

unread,

Apr 28, 2016, 11:23:17 AM4/28/16

to dynamor...@googlegroups.com

You many want to make sure the gdb has the debug information correctly.

Try something like set a breakpoint to see if it is able to find the right place and put a breakpoint.

(gdb) s

Cannot find bounds of current function

can you try x/10i to see the instructions.

I often find si (step instruction by instruction) and ni (next instruction) is useful.

Also the debugger is not perfect, I often set a breakpoint and then continue to make execution progress.

(gdb) continue

Continuing.

Program received signal SIGINT, Interrupt.

0xb6ff0a18 in ?? ()

(gdb) continue

Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap.

0x2a20f5f8 in ?? ()

(gdb) cont

Continuing.

[New Thread 3910]

[New Thread 3909]

Program received signal SIGSEGV, Segmentation fault.

[Switching to Thread 3910]

0x2a21118e in ?? ()

(gdb)

--

You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To post to this group, send email to dynamor...@googlegroups.com.
Visit this group at https://groups.google.com/group/dynamorio-users.
For more options, visit https://groups.google.com/d/optout.

Qin Zhao

unread,

Apr 28, 2016, 11:24:00 AM4/28/16

to dynamor...@googlegroups.com

I think the most useful information would be log file produced by -loglevel 4.

What's the instruction at 0x2a21118e? what's the machine context?

--

You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To post to this group, send email to dynamor...@googlegroups.com.
Visit this group at https://groups.google.com/group/dynamorio-users.
For more options, visit https://groups.google.com/d/optout.

Caroline Trippel

unread,

Apr 28, 2016, 9:50:41 PM4/28/16

to DynamoRIO Users

I'm attaching 3 files here that were generated from running the cppthread program I posted earlier, compiled in the exact way Derek compiled it earlier, with DR release 6.1-3, running on Android Nexus 6 (platform 21). The DR command I ran was as follows:

./bin32/drrun -debug -loglevel 4 -- ./cppthread

I don's even see this address in the log files: 0x2a21118e

Where do I look to see the code cache?

cppthread.0.13428.html

Caroline Trippel

unread,

Apr 28, 2016, 9:51:22 PM4/28/16

to DynamoRIO Users

log.1.13432.html

Caroline Trippel

unread,

Apr 28, 2016, 9:52:48 PM4/28/16

to DynamoRIO Users

***The size limits are preventing me from attaching log.1.*.html

Derek Bruening

unread,

Apr 28, 2016, 11:47:03 PM4/28/16

to dynamor...@googlegroups.com

These files will compress by a large factor.

On Thu, Apr 28, 2016 at 9:52 PM, Caroline Trippel <carolin...@gmail.com> wrote:

***The size limits are preventing me from attaching log.1.*.html

--

Caroline Trippel

unread,

Apr 28, 2016, 11:52:54 PM4/28/16

to DynamoRIO Users

log.1.13432.html.zip

Derek Bruening

unread,

Apr 29, 2016, 5:39:41 PM4/29/16

to dynamor...@googlegroups.com

On Thu, Apr 28, 2016 at 9:50 PM, Caroline Trippel <carolin...@gmail.com> wrote:

I'm attaching 3 files

log.0 is missing

I don's even see this address in the log files: 0x2a21118e

That is the address that you reported from gdb with the fault. Repeating what Qin asked: what are the instructions at that address? What is the machine context? The precise address will change from run to run of course. That is likely a code cache address, so the log file will be more useful than gdb.

Where do I look to see the code cache?

As mentioned earlier, the log file at level 4 disassembles each fragment of code exactly as it appears in the cache. Find where the SIGSEGV is shown in the log file and look at the fragment of code it's in.

Derek Bruening

unread,

Apr 29, 2016, 10:45:02 PM4/29/16

to dynamor...@googlegroups.com

https://github.com/DynamoRIO/dynamorio/wiki/Logfiles contains some basic info on what is in the logfiles.

Caroline Trippel

unread,

Apr 30, 2016, 12:25:00 PM4/30/16

to DynamoRIO Users

NOTE: attaching more files in a subsequent message as all do no not fit here when compressed

I'm going to look deeper into the log files now. I've put together a set of files containing the binaries I am currently working with as well as the log files produced from a series of runs. I'm detailing these runs below, and I reference files from the attached directory. Additionally, I am wondering if there any details pertaining to the 6.1-3 release that would be specific to a version of Android other than API 21. gdb doesn't seem to recognize the addresses of my segmentation faults has having meaningful instructions at them. I'm wondering if something is causing a branch to a random part of memory.

—————————————————————————————————————————————————————————————————————————

How I built NDK:

root@ctrippel-VirtualBox:/usr/local/tools# ./android-ndk-r11c/build/tools/make-standalone-toolchain.sh --arch=arm --platform=android-21 --install-dir=./android-ndk-21

HOST_OS=linux

HOST_EXE=

HOST_ARCH=x86_64

HOST_TAG=linux-x86_64

HOST_NUM_CPUS=1

BUILD_NUM_CPUS=2

Auto-config: --toolchain=arm-linux-androideabi-4.9

Copying prebuilt binaries...

Copying sysroot headers and libraries...

Copying c++ runtime headers and libraries...

Copying files to: ./android-ndk-21

Cleaning up...

Done.

—————————————————————————————————————————————————————————————————————————

****************************************See simple.cpp file****************************************

—————————————————————————————————————————————————————————————————————————

How I compiled simple.cpp to cppthread and pushed to phone

root@ctrippel-VirtualBox:/usr/local/tools# ./android-ndk-21/bin/arm-linux-androideabi-g++ -std=c++11 -g -fpie -pie -o cppthread simple.cpp

root@ctrippel-VirtualBox:/usr/local/tools# adb push cppthread /data/local/tmp/apps

344 KB/s (4207476 bytes in 11.921s)

Phone specs: Nexus 6, Quad-core, running Android 5.0

—————————————————————————————————————————————————————————————————————————

Running uninstrumented app on phone

root@shamu:/data/local/tmp # ./apps/cppthread

Hello from main

Greetings from thread2

Hi from thread1

root@shamu:/data/local/tmp #

root@shamu:/data/local/tmp # ./apps/cppthread

Hello from main

Hi from thread1

Greetings from thread2

—————————————————————————————————————————————————————————————————————————

Phone file hierarchy:

/data/local/tmp/

——> build_android/

————> lib32 // from DynamoRIO-ARM-Android-EABI-6.1.1-3/lib32

————> bin32 // from DynamoRIO-ARM-Android-EABI-6.1.1-3/bin32

————> ext // from DynamoRIO-ARM-Android-EABI-6.1.1-3/ext

——> apps/

————> cppthread

—————————————————————————————————————————————————————————————————————————

Running drrun with no client on cppthread

root@shamu:/data/local/tmp # ./build_android/bin32/drrun -- ./apps/cppthread

Hello from main

Segmentation fault

—————————————————————————————————————————————————————————————————————————

Now running with -debug -loglevel 4

Runs fine the first time…

root@shamu:/data/local/tmp # ./build_android/bin32/drrun -debug -loglevel 4 ->

<Paste into GDB to debug DynamoRIO clients:

set confirm off

add-symbol-file '/data/local/tmp/build_android/lib32/debug/libdynamorio.so' 0x00021508

>

<get_memory_info mismatch! (can happen if os combines entries in /proc/pid/maps)

os says: 0xb6802000-0xb6900000 prot=0x00000003

cache says: 0xb6802000-0xb6b00000 prot=0x00000003

>

HiGreetingsHello from thread2

from main

from thread1

****************************************See cppthread.9278.00000000/ directory***************************************

Segmentation fault the second time…

/bin32/drrun -debug -loglevel 4 -- ./apps/cppthread <

<Paste into GDB to debug DynamoRIO clients:

set confirm off

add-symbol-file '/data/local/tmp/build_android/lib32/debug/libdynamorio.so' 0x00021508

>

<get_memory_info mismatch! (can happen if os combines entries in /proc/pid/maps)

os says: 0xb6802000-0xb6900000 prot=0x00000003

cache says: 0xb6802000-0xb6b00000 prot=0x00000003

>

Segmentation fault

****************************************See cppthread.9286.00000000/ directory***************************************

—————————————————————————————————————————————————————————————————————————

Now running with -debug -loglevel 4 and gdb/gdbserver (This time does not create logs)

root@ctrippel-VirtualBox:/usr/local/tools# ./android-ndk-r11c/prebuilt/linux-x86_64/bin/gdb

GNU gdb (GDB) 7.10

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law. Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at:

<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".

Type "apropos word" to search for commands related to "word".

(gdb) set solib-search-path ~/android/:~/android/system_lib:~/android/vendor_lib:~/android/vendor_lib/egl/:/usr/local/tools/DynamoRIO-ARM-Android-EABI-6.1.1-3/lib32/debug/

(gdb) add-symbol-file '/data/local/tmp/build_android/lib32/debug/libdynamorio.so' 0x00021508

add symbol table from file "/data/local/tmp/build_android/lib32/debug/libdynamorio.so" at

.text_addr = 0x21508

(y or n) y

Reading symbols from /data/local/tmp/build_android/lib32/debug/libdynamorio.so...Reading symbols from /data/local/tmp/build_android/lib32/debug/libdynamorio.so.debug...done.

done.

(gdb) target remote :1234

Remote debugging using :1234

warning: Architecture rejected target-supplied description

Reading /data/local/tmp/build_android/bin32/drrun from remote target...

warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.

Reading /data/local/tmp/build_android/bin32/drrun from remote target...

Load new symbol table from "target:/data/local/tmp/build_android/bin32/drrun"? (y or n) y

Reading symbols from target:/data/local/tmp/build_android/bin32/drrun...Reading /data/local/tmp/build_android/bin32/drrun.debug from remote target...

Reading /data/local/tmp/build_android/bin32/drrun.debug from remote target...

Load new symbol table from "target:/data/local/tmp/build_android/bin32/drrun.debug"? (y or n) y

Reading symbols from target:/data/local/tmp/build_android/bin32/drrun.debug...done.

done.

Reading /system/bin/linker from remote target...

Reading symbols from target:/system/bin/linker...Reading /system/bin/.debug/linker from remote target...

(no debugging symbols found)...done.

0xb6ff0a18 in __dl__start () from target:/system/bin/linker

(gdb) break main

Breakpoint 1 at 0x2a01aa60: file /dynamorio_package/tools/drdeploy.c, line 976.

(gdb) cont

Continuing.

Breakpoint 1, main (argc=6, argv=0xbefff9e4) at /dynamorio_package/tools/drdeploy.c:976

976 /dynamorio_package/tools/drdeploy.c: No such file or directory.

(gdb) x/100i 0x2a01aa60

=> 0x2a01aa60 <main>: stmdb sp!, {r4, r5, r6, r7, r8, r9, r10, r11, lr}

…

0x2a01ab72 <main+274>: str r1, [sp, #16]

(gdb)

(gdb) break *0x2a01ab72

Breakpoint 2 at 0x2a01ab72: file /dynamorio_package/tools/drdeploy.c, line 1088.

(gdb) cont

Continuing.

Breakpoint 2, 0x2a01ab72 in main (argc=6, argv=0xbefff9e4) at /dynamorio_package/tools/drdeploy.c:1088

1088 in /dynamorio_package/tools/drdeploy.c

(gdb) x/100i 0x2a01aa60

0x2a01aa60 <main>: stmdb sp!, {r4, r5, r6, r7, r8, r9, r10, r11, lr}

…

=> 0x2a01ab72 <main+274>: str r1, [sp, #16]

(gdb)

(gdb) break *0x2a01ab70

Breakpoint 3 at 0x2a01ab70: file /dynamorio_package/tools/drdeploy.c, line 1088.

(gdb) cont

Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap.

0x2a20f5f8 in ?? ()

(gdb) ni

0x2a20f5fc in ?? ()

(gdb) x/100i 0x2a01ab70

0x2a01ab70 <main+272>: add r1, pc

…

0x2a01ac36 <main+470>: movs r0, r0

(gdb) break *0x2a01ac36

Breakpoint 4 at 0x2a01ac36: file /dynamorio_package/tools/drdeploy.c, line 344.

(gdb) cont

Continuing.

Program received signal SIGSEGV, Segmentation fault.

0x2a232d06 in ?? ()

(gdb) Quit

—————————————————————————————————————————————————————————————————————————

Running again with -debug -loglevel 4 and gdb… (This time does not create logs and fails at the same place as the previous)