Tips for debugging statically linked DynamoRIO & drmemtrace

[Suyash Mahar]

Hi folks,

I am running into an internal error while attaching to an internal workload with statically linked DynamoRIO & drmemtrace:

<Application /data/users/appname (3186634).  Internal Error: DynamoRIO debug check failure: /mnt/srcs/dynamorio/core/unix/os.c:10307 iter->vm_start == executable_start
(Error occurred @0 frags in tid 3200078)

Unfortunately, debugging is a little bit challenges as DYNAMORIO_OPTIONS does not seem support -debug:

Option parsing error : unknown option -debug. Terminating

I already have these options enabled and am linking statically with dynamorio_static and drmemtrace_static:

    int set_env_result = setenv(
        "DYNAMORIO_OPTIONS",
        "-debug -loglevel 4 -rstats_to_stderr -client_lib ';;-offline -verbose 3'",
        1);

Are there any other recommendations on how to debug this further?

-Suyash

[Derek Bruening]

-debug is a parameter to the launcher drrun and does not mean anything to a build of the DR library: that DR library build is inherently and statically either a debug or a release build; -debug tells drrun for dynamic launches which library to pick. It looks like your statically linked DR library is a debug build as you're seeing an assert fire which only happens in debug. Does this fail in release? In some cases an assert can be more of a curiosity and is not fatal; here it couldn't find the executable, but I'm not sure it really needs it other than for some client APIs to tell a client, and maybe drmemtrace doesn't invoke those. For debugging, check how the code identifies the executable: it might be looking at maps file paths or sthg, which may not be 100% reliable with remappings: so probably your environment has remappings. If regular logging is too verbose, custom logging there may help.

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dynamorio-users/b5a1105a-5029-4565-89af-be2c502dd989n%40googlegroups.com.

[Suyash Mahar]

Yes, it is a debug build that is statically linked.

Attach fails to take over threads on a release build:

<Starting application /path/to/app (1956125)>
<Attached to 1/2015 threads in application /path/to/app (1956125)>
<Application /path/to/app (1956125). Timed out attempting to take over one or more threads. Aborting. Use -unsafe_ignore_takeover_timeout to ignore.>
<Application /path/to/app (1956125). Failed to take over all threads after multiple attempts.>

I think you are right, remapped and duplicate binaries in the address space seem to break DynamoRIO's logic. 

Also, after posting the original message, I realized that my app did not have the dynamorio_so_start and dynamorio_so_end symbols defined. But even after defining the symbols at linking and ignoring the iter->vm_start check with -ignore_assert_list '/mnt/srcs/dynamorio/core/unix/os.c:10307', I'm still running into issues, e.g.:

<(1+x) Handling our fault in a TRY at 0x0000000014ed2388>
<Application /path/to/app (2871742).  Internal Error: DynamoRIO debug check failure: /mnt/srcs/dynamorio/core/unix/os.c:557 libc_errno_loc != NULL
(Error occurred @0 frags in tid 2887858)
version 11.90.0, custom build
-rstats_to_stderr -client_lib ';;-offline -verbose 3' -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -no_inline_ignored_syscalls -no_hook_vsyscall -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct -ignore_assert_list '/mnt/srcs/dynamorio/core/unix/os.c:10307'
...

My current plan is to try this with a simpler workload to see if this a problem with our build environment. Although, a simple benchmark links and traces just fine.

-Suyash