Building a custom ARM client from x86

[David]

Hey there,

I am trying to build a custom ARM client (e.g. a copy of bbsize.c).

So far I've followed the Cross-Compiling for ARM on Linux guide and it resulted in a api/bin/ directory with the compiled samples.

Which steps do I need to follow in order to compile my own client for ARM?

When I cd into api/samples/ and try to execute cmake (without modifying anything prior) I'm getting the following error:

$ cmake -DDynamoRIO_DIR=../../cmake .
CMake Error at /home/david/dbi/dynamorio/build_arm/cmake/DynamoRIOConfig.cmake:662 (include):
  include could not find load file:

    /home/david/dbi/dynamorio/build_arm/cmake/DynamoRIOTarget64.cmake
Call Stack (most recent call first):
  CMakeLists.txt:131 (find_package)


CMake Error at /home/david/dbi/dynamorio/build_arm/cmake/DynamoRIOConfig.cmake:1352 (message):
  DynamoRIO Extension "drmgr" not found
Call Stack (most recent call first):
  CMakeLists.txt:156 (use_DynamoRIO_extension)
  CMakeLists.txt:199 (add_sample_client)


-- Configuring incomplete, errors occurred!
See also "/home/david/dbi/dynamorio/build_arm/api/samples/CMakeFiles/CMakeOutput.log".

Searching for a file named "DynamoRIOTarget64.cmake" inside my DynamoRIO directory does not find any file with this name.

How should I proceed?

Thank you very much!

- David

[David]

Additional info (isolated client, x86_64 works, ARM does not):

I've followed the "Write Your DynamoRIO Client" part of the tutorial.

I've written a small testclient.c:

#include "dr_api.h"
#include "drmgr.h"
#include "syscall.h"

static void event_exit() {
    drmgr_exit();
}

static void event_post_syscall(void *drcontext, int sysnum) {
    reg_t result = dr_syscall_get_result(drcontext);
    dr_printf("syscall %d: %d\n", sysnum, result);
}

DR_EXPORT void dr_client_main(client_id_t id, int argc, const char *argv[]) {
    drmgr_init();
    dr_register_exit_event(event_exit);
    drmgr_register_post_syscall_event(event_post_syscall);
}

CMakeLists.txt:

add_library(testclient SHARED testclient.c)
find_package(DynamoRIO)
if (NOT DynamoRIO_FOUND)
  message(FATAL_ERROR "DynamoRIO package required to build")
endif(NOT DynamoRIO_FOUND)
configure_DynamoRIO_client(testclient)
use_DynamoRIO_extension(testclient drmgr)

Build and run via:

$ cmake -DDynamoRIO_DIR=../cmake .
$ make
$ ../bin64/drrun -c ./libtestclient.so -- /bin/ls

This works when built for x86_64.

However, when I replicate the exact same steps for ARM, cmake produces this error:

$ cmake -DDynamoRIO_DIR=../cmake .

CMake Error at /home/david/dbi/dynamorio/build_arm/cmake/DynamoRIOConfig.cmake:662 (include):
  include could not find load file:

    /home/david/dbi/dynamorio/build_arm/cmake/DynamoRIOTarget64.cmake
Call Stack (most recent call first):

  CMakeLists.txt:2 (find_package)

CMake Error at /home/david/dbi/dynamorio/build_arm/cmake/DynamoRIOConfig.cmake:1352 (message):
  DynamoRIO Extension "drmgr" not found
Call Stack (most recent call first):

  CMakeLists.txt:7 (use_DynamoRIO_extension)


CMake Warning (dev) in CMakeLists.txt:
  No cmake_minimum_required command is present.  A line of code such as

    cmake_minimum_required(VERSION 3.5)

  should be added at the top of the file.  The version specified may be lower
  if you wish to support older CMake versions for this project.  For more
  information run "cmake --help-policy CMP0000".
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring incomplete, errors occurred!

So apparently the DynamoRIOConfig.cmake inside the build_arm directory tries to include a DynamoRIOTarget64.cmake which is not there.

I've tried passing either

-DCMAKE_TOOLCHAIN_FILE=../../make/toolchain-arm32.cmake
-DCMAKE_FIND_ROOT_PATH=/usr/arm-linux-gnueabihf
-DTARGET_ABI=linux-gnueabihf

but none of these had an effect on the output.

I feel a little stuck here, what is the correct way to build an ARM client from a x86_64 host?





Thank you!

- David


[hashmi...@googlemail.com]

Hello David,

We've been focussing on 64 bit Arm support and so 32 bit support is not as consistent as it should be.

I'm not sure why your client build doesn't work. Here is my workaround which you can try.

I cross built from a clean dynamorio clone as you did following https://github.com/DynamoRIO/dynamorio/wiki/How-To-Build#cross-compiling-for-arm-on-linux

I built on a Ubuntu 16.04.3 LTS box.

Add your testclient.c to dynamorio/api/samples dir:

$ cat dynamorio/api/samples/testclient.c

#include "dr_api.h"
#include "drmgr.h"
#include "syscall.h"


static void event_exit() {
    drmgr_exit();
}


static void event_post_syscall(void *drcontext, int sysnum) {
    reg_t result = dr_syscall_get_result(drcontext);
    dr_printf("syscall %d: %d\n", sysnum, result);
}


DR_EXPORT void dr_client_main(client_id_t id, int argc, const char *argv[]) {
    drmgr_init();
    dr_register_exit_event(event_exit);
    drmgr_register_post_syscall_event(event_post_syscall);
}

$

Add the following line to dynamorio/api/samples/CMakeLists.txt:

add_sample_client(testclient "testclient.c" "drmgr")

e.g.

$ git diff api/samples/CMakeLists.txt
diff --git a/api/samples/CMakeLists.txt b/api/samples/CMakeLists.txt
index 5c26f37..97e49fc 100644
--- a/api/samples/CMakeLists.txt
+++ b/api/samples/CMakeLists.txt
@@ -213,6 +213,7 @@ add_sample_client(empty       "empty.c"         "")
 add_sample_client(memtrace_simple "memtrace_simple.c;utils.c" "drmgr;drreg;drutil;drx")
 add_sample_client(memval_simple   "memval_simple.c;utils.c"   "drmgr;drreg;drutil;drx")
 add_sample_client(instrace_simple "instrace_simple.c;utils.c" "drmgr;drreg;drx")
+add_sample_client(testclient "testclient.c" "drmgr")                                         <---
 if (X86) # FIXME i#1551, i#1569: port to ARM and AArch64
   add_sample_client(cbr         "cbr.c"           "drmgr")
   add_sample_client(countcalls  "countcalls.c"    "drmgr;drreg")
$

Rebuild. This will create build_arm/api/bin/libtestclient.so:

rm -fr build_arm/
mkdir build_arm
cd build_arm/
cmake -DCMAKE_TOOLCHAIN_FILE=../dynamorio/make/toolchain-arm32.cmake ../dynamorio
make -j


$ file api/bin/libtestclient.so
api/bin/libtestclient.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=e7fea38c58990bc886bcebee9a6ecdf690806b19, not stripped
$

Run a simple binary on an AArch32 box using the client:

$ cat hw.c
#include <stdio.h>
#include <stdlib.h>


int main() {
  printf("Hello DynamoRIO!\n");
  exit(0);
}
$
$ ./build_arm/bin32/drrun -c build_arm/api/bin/libtestclient.so -- ./hw
syscall 45: 143360
syscall 192: -1226899456
syscall 192: -1226915840
syscall 6: 0
syscall 192: -1228206080
syscall 125: 0
syscall 192: -1226940416
syscall 192: -1226928128
syscall 6: 0
syscall 192: -1228210176
syscall 983045: 0
syscall 125: 0
syscall 125: 0
syscall 125: 0
syscall 91: 0
syscall 192: -1226903552
Hello DynamoRIO!
$

Hope that's useful.

Assad

[David]

Hello Assad,

thank your for your reply and for your effort in trying to reproduce the problem.

Two shots in the dark as I am not (yet) too familiar with cmake and DynamoRIO:

- May it be the case that I need to pass a variable to cmake when building my client, hinting that I'd like to build for ARM, similar to CMAKE_TOOLCHAIN_FILE=[...] when running cmake for the entire DynamoRIO project?

- Or maybe there I need to edit the CMakeLists.txt in order to tell it to use ARM for this client instead of my host's architecture (and thus not trying to include DynamoRIOTarget64.cmake)?

Your workaround does work, due to the rather huge overhead compared to only building the client it takes 2-3 minutes to build on my system, which I could imagine to be too tedious to work with in the long term.

Is it possible build the clients? I've tried using ccmake, setting BUILD_CORE from ON to OFF, but when BUILD_CORE is set to OFF, ccmake produces this error when pressing c (at ~90%):

Make Warning (dev) at build_arm/cmake/DynamoRIOConfig.cmake:313 (get_target_property):
   Policy CMP0045 is not set: Error on non-existent target in
   get_target_property.  Run "cmake --help-policy CMP0045" for policy details.
   Use the cmake_policy command to set the policy and suppress this warning.

   get_target_property() called with non-existent target "dynamorio".

 Call Stack (most recent call first):

   build_arm/cmake/DynamoRIOConfig.cmake:1065 (DynamoRIO_add_rel_rpaths)
   build_arm/cmake/DynamoRIOConfig.cmake:1244 (_DR_set_compile_flags)
   clients/drcachesim/CMakeLists.txt:87 (configure_DynamoRIO_standalone)

 This warning is for project developers.  Use -Wno-dev to suppress it.

 CMake Error at build_arm/cmake/DynamoRIOConfig.cmake:314 (file):
   file RELATIVE_PATH must be passed a full path to the file: lib_loc-NOTFOUND
 Call Stack (most recent call first):
   build_arm/cmake/DynamoRIOConfig.cmake:1065 (DynamoRIO_add_rel_rpaths)
   build_arm/cmake/DynamoRIOConfig.cmake:1244 (_DR_set_compile_flags)
   clients/drcachesim/CMakeLists.txt:87 (configure_DynamoRIO_standalone)

I suppose another option could be to use a full-fledged Debian ARM VM and use that to develop ARM clients for DynamoRIO, I think I'll look into that as an alternative as long as this issue is still unresolved.

Another question regarding your note that 32 bit support for ARM is not as consistent as it should be:

- Are there any known limitations as in things that *should* work for 32 bit ARM, but currently don't?

Thank you very much!

- David

[Derek Bruening]

Back to your original attempt, I have built an ARM client on an x86 host, by specifying 

-DCMAKE_TOOLCHAIN_FILE to point at toolchain-arm32.cmake.  Without doing so the code will look for a 64-bit configuration and fail to find 

DynamoRIOTarget64.cmake, which is what you hit.  You mentioned that you tried the toolchain file: what was the result?  I would expect it to work.  The toolchain file is needed for any new cross-compile configuration.

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To post to this group, send email to dynamor...@googlegroups.com.
Visit this group at https://groups.google.com/group/dynamorio-users.
For more options, visit https://groups.google.com/d/optout.

[David]

Thank you for your effort, I've tried again and I am getting the following output now (I may have messed something up last time which led me to believe the flags did not change the output):

$ pwd
/home/david/dbi/dynamorio/build_arm

$ file /home/david/dbi/dynamorio/make/toolchain-arm32.cmake
/home/david/dbi/dynamorio/make/toolchain-arm32.cmake: ASCII text
$ file /home/david/dbi/dynamorio/build_arm/cmake
/home/david/dbi/dynamorio/build_arm/cmake: directory

$ rm CMakeCache.txt
$ rm -rf CMakeFiles/

$ ls -l testclient/
total 24
-rw-rw-r-- 1 david david 262 Feb 20 15:52 CMakeLists.txt
-rw-rw-r-- 1 david david 475 Feb 20 15:52 testclient.c

$ cat testclient/testclient.c

#include "dr_api.h"
#include "drmgr.h"
#include "syscall.h"

static void event_exit() {
    drmgr_exit();
}

static void event_post_syscall(void *drcontext, int sysnum) {
    reg_t result = dr_syscall_get_result(drcontext);
    dr_printf("syscall %d: %d\n", sysnum, result);
}

DR_EXPORT void dr_client_main(client_id_t id, int argc, const char *argv[]) {
    drmgr_init();
    dr_register_exit_event(event_exit);
    drmgr_register_post_syscall_event(event_post_syscall);
}

$ cat testclient/CMakeLists.txt
add_library(testclient SHARED testclient.c)

find_package(DynamoRIO)
if (NOT DynamoRIO_FOUND)
  message(FATAL_ERROR "DynamoRIO package required to build")
endif(NOT DynamoRIO_FOUND)
configure_DynamoRIO_client(testclient)
use_DynamoRIO_extension(testclient drmgr)

$ cmake -DCMAKE_TOOLCHAIN_FILE=/home/david/dbi/dynamorio/make/toolchain-arm32.cmake -DDynamoRIO_DIR=/home/david/dbi/dynamorio/build_arm/cmake ./testclient/
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/arm-linux-gnueabihf-gcc
-- Check for working C compiler: /usr/bin/arm-linux-gnueabihf-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/arm-linux-gnueabihf-g++
-- Check for working CXX compiler: /usr/bin/arm-linux-gnueabihf-g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done

CMake Warning (dev) in CMakeLists.txt:
  No cmake_minimum_required command is present.  A line of code such as

    cmake_minimum_required(VERSION 3.5)

  should be added at the top of the file.  The version specified may be lower
  if you wish to support older CMake versions for this project.  For more
  information run "cmake --help-policy CMP0000".
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring done
-- Generating done
-- Build files have been written to: /home/david/dbi/dynamorio/build_arm

$ make testclient
Scanning dependencies of target testclient
[ 50%] Building C object CMakeFiles/testclient.dir/testclient.o
In file included from /home/david/dbi/dynamorio/build_arm/testclient/testclient.c:1:0:
/home/david/dbi/dynamorio/build_arm/cmake/../include/dr_api.h:53:20: fatal error: dr_app.h: No such file or directory
compilation terminated.
CMakeFiles/testclient.dir/build.make:62: recipe for target 'CMakeFiles/testclient.dir/testclient.o' failed
make[3]: *** [CMakeFiles/testclient.dir/testclient.o] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/testclient.dir/all' failed
make[2]: *** [CMakeFiles/testclient.dir/all] Error 2
CMakeFiles/Makefile2:79: recipe for target 'CMakeFiles/testclient.dir/rule' failed
make[1]: *** [CMakeFiles/testclient.dir/rule] Error 2
Makefile:140: recipe for target 'testclient' failed
make: *** [testclient] Error 2

$ find /home/david/dbi/ -name 'dr_app.h'
/home/david/dbi/dynamorio/core/lib/dr_app.h
/home/david/dbi/dynamorio/build_x86_64/include/dr_app.h

$ find /home/david/dbi/ -name 'dr_api.h'
/home/david/dbi/dynamorio/core/lib/dr_api.h
/home/david/dbi/dynamorio/build_x86_64/include/dr_api.h
/home/david/dbi/dynamorio/build_arm/include/dr_api.h

(notice dr_app.h is missing in the build_arm directory)

Same procedure in build_x86_64:

$ cmake -DDynamoRIO_DIR=/home/david/dbi/dynamorio/build_x86_64/cmake ./testclient/

CMake Warning (dev) in CMakeLists.txt:
  No cmake_minimum_required command is present.  A line of code such as

    cmake_minimum_required(VERSION 3.5)

  should be added at the top of the file.  The version specified may be lower
  if you wish to support older CMake versions for this project.  For more
  information run "cmake --help-policy CMP0000".
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring done
-- Generating done
-- Build files have been written to: /home/david/dbi/dynamorio/build_x86_64

$ make testclient
Scanning dependencies of target testclient
[ 50%] Building C object CMakeFiles/testclient.dir/testclient.o
[100%] Linking C shared library libtestclient.so
[100%] Built target testclient

Here the documentation says to define LINUX and ARM_32 when using dr_api.h.

Is this possibly the cause, and if so, do I need to explicitly define them myself?

So far I've taken some other shots in the dark and tried to:

- Add #define LINUX 1 and #define ARM_32 1 as first lines in testclient.c

- Run cmake with -DLINUX=1 -DARM_32=1

- Add add_compile_options(-D LINUX -D ARM_32) or add_compile_options(-DLINUX=1 -DARM_32=1) as the first line in testclient/CMakeLists.txt

- Run make with CFLAGS="-DLINUX=1 -DARM_32=1"

But I assume it rather has something to do with the missing dr_app.h in the build_arm/ directory. Is there an explanation or a way to find out why it is missing?

Thank you!

 -David

[Derek Bruening]

Generally, you want to point your client at a DR *installation*, not an internal build directory.  Run "make install" in your DR build directory and point at the resulting exported tree.  (-DCMAKE_INSTALL_PREFIX controls where that goes.)

--

[David]

Oh, thank you so much!

Running "make install" first and pointing to the resulting exported tree solved the issue and I could now successfully cross-compile an ARM client on my machine.

Thanks again!