drwrap_skip_call breaks application

Mohammad Ewais

unread,

Jul 5, 2021, 5:07:47 AM7/5/21

to DynamoRIO Users

I am trying to use drwrap_skip_call to skip calls to pthread_create. It, unfortunately, breaks the application unexpectedly after the 2nd skip (not really sure why 2nd, but consistently so).

This is the simple test app:

#include <iostream>

#include <pthread.h>

#include <string>

pthread_t all_threads[8];

uint32_t all_data[7];

void* foo(void* data)

{

std::cout << "FOO CALLED\n";

return nullptr;

}

void* bar(void* x)

{

int32_t val = *(int32_t*)x;

std::cout << "BAR CALLED WITH " + std::to_string(val) + "\n";

return nullptr;

}

int main(int argc, char** argv)

{

std::cout << "Starting thread\n";

pthread_create(&all_threads[0], nullptr, &foo, nullptr);

for (uint32_t i = 1; i < 8; i++)

{

std::cout << "Starting thread\n";

all_data[i-1] = i;

pthread_create(&all_threads[i], nullptr, &bar, &all_data[i-1]);

}

for (uint32_t i = 0; i < 8; i++)

{

std::cout << "Ending thread\n";

pthread_join(all_threads[i], nullptr);

}

return 0;

}

And this is the simple client causing the issue, I track all module loads, and when the pthread module gets loaded I wrap the pthread_create function with only a pre-wrapper:

#include <dr_api.h>

#include <drwrap.h>

#include <drmgr.h>

#include <string>

void PrePThreadCreateAction(void* wrapctx, void** user_data)

{

dr_printf("CLIENT: Before skipping\n");

drwrap_skip_call(wrapctx, (void*)0, sizeof(void*)*4);

}

void ModuleLoadAction(void* drcontext, const module_data_t* info, bool loaded)

{

std::string module_name = dr_module_preferred_name(info);

if (module_name.rfind("libpthread.", 0) == 0)

{

uint64_t pthread_create_address = (uint64_t)dr_get_proc_address(info->handle, "pthread_create");

drwrap_wrap((app_pc)pthread_create_address, PrePThreadCreateAction, nullptr);

}

void ExitAction()

{

drwrap_exit();

drmgr_exit();

}

DR_EXPORT void dr_client_main(client_id_t id, int argc, const char** argv)

{

drmgr_init();

drwrap_init();

drmgr_register_module_load_event(ModuleLoadAction);

dr_register_exit_event(ExitAction);

}

If I comment out the drwrap_skip_call the application executes successfully:

<Starting application /home/mewais/PlayGround/Build/Test (36018)>

<Paste into GDB to debug DynamoRIO clients:

set confirm off

add-symbol-file '/home/mewais/PlayGround/Build/libPlayClient.so' 0x00007fb9b0072580

add-symbol-file '/work/mewais/projects/DCArch/DynamoRIO/lib64/debug/libdynamorio.so' 0x0000000071040fe0

add-symbol-file '/work/mewais/projects/DCArch/DynamoRIO/ext/lib64/debug/libdrwrap.so' 0x00007fb9b00c1150

add-symbol-file '/work/mewais/projects/DCArch/DynamoRIO/ext/lib64/debug/libdrmgr.so' 0x00007fb9b02d05f0

add-symbol-file '/lib/x86_64-linux-gnu/libstdc++.so.6' 0x00007fb9f3cb12e0

add-symbol-file '/lib/x86_64-linux-gnu/libm.so.6' 0x00007fb9f3eff3c0

add-symbol-file '/lib/x86_64-linux-gnu/libc.so.6' 0x00007fb9f3864650

add-symbol-file '/usr/lib64/ld-linux-x86-64.so.2' 0x00007fb9f3eba090

add-symbol-file '/lib/x86_64-linux-gnu/libgcc_s.so.1' 0x00007fb9f3ea05e0

>

<(1+x) Handling our fault in a TRY at 0x000000007129f6b2>

<spurious rep/repne prefix @0x00007fb9f4040e80 (f3 0f 1e fa): >

<curiosity: rex.w on OPSZ_6_irex10_short4!>

<get_memory_info mismatch! (can happen if os combines entries in /proc/pid/maps)

os says: 0x00007fb9f3e86000-0x00007fb9f3e9d000 prot=0x00000001

cache says: 0x00007fb9f3e86000-0x00007fb9f3ea0000 prot=0x00000001

>

Starting thread

CLIENT: Before skipping

FOO CALLED

Starting thread

CLIENT: Before skipping

Starting thread

CLIENT: Before skipping

Starting thread

CLIENT: Before skipping

Starting thread

CLIENT: Before skipping

Starting thread

CLIENT: Before skipping

Starting thread

CLIENT: Before skipping

Starting thread

CLIENT: Before skipping

Ending thread

BAR CALLED WITH 3

BAR CALLED WITH 1

BAR CALLED WITH 7

BAR CALLED WITH 2

BAR CALLED WITH 4

BAR CALLED WITH 5

BAR CALLED WITH 6

Ending thread

<Stopping application /home/mewais/PlayGround/Build/Test (36018)>

If I leave it on, it breaks. Here is the output from a run:

<Starting application /home/mewais/PlayGround/Build/Test (35243)>

<Paste into GDB to debug DynamoRIO clients:

set confirm off

add-symbol-file '/home/mewais/PlayGround/Build/libPlayClient.so' 0x00007fb7403415a0

add-symbol-file '/work/mewais/projects/DCArch/DynamoRIO/lib64/debug/libdynamorio.so' 0x0000000071040fe0

add-symbol-file '/work/mewais/projects/DCArch/DynamoRIO/ext/lib64/debug/libdrwrap.so' 0x00007fb740390150

add-symbol-file '/work/mewais/projects/DCArch/DynamoRIO/ext/lib64/debug/libdrmgr.so' 0x00007fb74059f5f0

add-symbol-file '/lib/x86_64-linux-gnu/libstdc++.so.6' 0x00007fb783f802e0

add-symbol-file '/lib/x86_64-linux-gnu/libm.so.6' 0x00007fb7841ce3c0

add-symbol-file '/lib/x86_64-linux-gnu/libc.so.6' 0x00007fb783b33650

add-symbol-file '/usr/lib64/ld-linux-x86-64.so.2' 0x00007fb784189090

add-symbol-file '/lib/x86_64-linux-gnu/libgcc_s.so.1' 0x00007fb78416f5e0

>

<(1+x) Handling our fault in a TRY at 0x000000007129f6b2>

<spurious rep/repne prefix @0x00007fb78430fe80 (f3 0f 1e fa): >

<curiosity: rex.w on OPSZ_6_irex10_short4!>

<get_memory_info mismatch! (can happen if os combines entries in /proc/pid/maps)

os says: 0x00007fb784155000-0x00007fb78416c000 prot=0x00000001

cache says: 0x00007fb784155000-0x00007fb78416f000 prot=0x00000001

>

Starting thread

CLIENT: Before skipping

Starting thread

<Application /home/mewais/PlayGround/Build/Test (35243). Application exception at PC 0x00007fb78034054a.

Signal 11 delivered to application as default action.

Callstack:

0x00007fb78034054a </home/mewais/PlayGround/Build/Test+0x154a>

0x00007fb783730565 </usr/lib/x86_64-linux-gnu/libc-2.33.so+0x28565>

>

<Stopping application /home/mewais/PlayGround/Build/Test (35243)>

<Application /home/mewais/PlayGround/Build/Test (35243). Internal Error: DynamoRIO debug check failure: /home/travis/build/DynamoRIO/dynamorio/core/heap.c:1913 IF_WINDOWS(doing_detach ||) vmh->num_free_blocks == vmh->num_blocks - unfreed_blocks || ((ever_beyond_vmm IF_WINDOWS(|| get_os_version() >= WINDOWS_VERSION_8_1)) && vmh->num_free_blocks >= vmh->num_blocks - unfreed_blocks)

(Error occurred @3820 frags)

version 8.0.0, build 1

-no_dynamic_options -client_lib '/home/mewais/PlayGround/Build/libPlayClient.so;0;' -code_api -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct

0x00007fb6c03e7e10 0x00000000710dc095

0x00007fb6c03e8060 0x000000007116f1dd

0x00007fb6c03e80a0 0x000000007116f268

0x00007fb6c03e80b0 0x00000000710535d2

0x00007fb6c03e80f0 0x000000007105390e

0x00007fb6c03e8120 0x0000000071053a94

0x00007fb6c03e8140 0x00000000712e2f29

0x00007fb6c03e82a0 0x00000000712e32fe

0x00007fb6c03e82d0 0x00000000712d9b98

0x00007fb6c03e8340 0x00000000712dff40

0x00007fb6c03e84d0 0x00000000712dcc09

0x00007fb6c03e87e0 0x00000000712dfac2

0x00007fb6c03e89b0 0x000000007129f1d3

0x00007ffe6a4f8c50 0x00007fb783730565>

Unfortunately, I could not gdb that as gdb breaks right away on the first SIGILL (looks irrelevant though). But the DR printed backtrace shows it happens in the app not the client (even though the second dr_printf isn't shown, but I am assuming that's just buffering!).

I have also seen this issue https://github.com/DynamoRIO/dynamorio/issues/2298 but my logs have no mention of max wrapped nesting reached or MAX_WRAP_NESTING or anything similar. So this is probably different, what could be the cause?

For the sake of error reproduction, if needed, here's a quick repo: https://github.com/mewais/DRPlayGround

Derek Bruening

unread,

Jul 5, 2021, 4:32:05 PM7/5/21

to dynamor...@googlegroups.com

If the call is skipped without any other remediation, won't you expect the app to crash? A random app might have uninitialized memory for the thread handle; this one would crash when it passes 0 for the handle to pthread_join?

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/3d168b56-9101-4a04-abcc-d3e2dc9a8e79n%40googlegroups.com.

Mohammad Ewais

unread,

Jul 5, 2021, 4:42:16 PM7/5/21

to DynamoRIO Users

Hi Derek,

Yes, it would definitely fail at the at the pthread_join calls (which I am also wrapping in my real application). Regardless, Because of the for loops in the app, I would expect a total of 8 (skipped) calls to pthread_create before the app actually tries to join threads and crashes, which is obviously not happening.

Mohammad Ewais

unread,

Jul 8, 2021, 9:24:38 AM7/8/21

to DynamoRIO Users

Any ideas on what may be causing this? Is it a bug in drwrap_skip_call or something missing on my end?

Derek Bruening

unread,

Jul 8, 2021, 10:38:55 AM7/8/21

to dynamor...@googlegroups.com

If you search the tracker, https://github.com/DynamoRIO/dynamorio/issues/2298 comes up: not sure if it is related; it has a proposed fix you could try.

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/8461c853-f232-47e2-99b2-fa7a307fc479n%40googlegroups.com.

Reply all

Reply to author

Forward