DR internal crash when changing memtrace_simple.c

[tengda w]

Hi,

I'm modifying memtrace_simple.c to add memory access timestamps on aarch64 architecture, but internal crash occurred. 

To figure out where the problem occured, I keep the other parts of memtrace_simple.c unchanged and only modify the following fprintf statement in "memtrace" function:

Original code:

fprintf(allfile,

    "" PIFX ": %2d, %s\n",

    (ptr_uint_t)mem_ref->addr,

    mem_ref->size,

    (mem_ref->type > REF_TYPE_WRITE)

        ? decode_opcode_name(mem_ref->type) /* opcode for instr */

        : (mem_ref->type == REF_TYPE_WRITE ? "w" : "r")

    );

Modified code:

fprintf(allfile,

    "" PIFX ": %2d, %s, %lf\n",

    (ptr_uint_t)mem_ref->addr,

    mem_ref->size,

    (mem_ref->type > REF_TYPE_WRITE)

        ? decode_opcode_name(mem_ref->type) /* opcode for instr */

        : (mem_ref->type == REF_TYPE_WRITE ? "w" : "r")

    ,1.1);

I only add a “%lf” at the end of the formatted output section and print the constant 1.1. However, after the modification, memtrace_simple.c works fine for pinning a single-threaded program, but when pinning a multi-threaded program, the following error occurs:  

Data file /home/wangtd/DynamoRIO-AArch64-Linux-11.2.0/samples/build/bin/memtrace.mthread.out.197423.0000.log created

Data file /home/wangtd/DynamoRIO-AArch64-Linux-11.2.0/samples/build/bin/memtrace.mthread.out.197423.0001.log created

<Application /home/wangtd/demo/mthread.out (197423).  DynamoRIO Sample Client 'memtrace' internal crash at PC 0x0000fffbde02d8ec.  Please report this at http://dynamorio.org/issues.  Program aborted.

Received SIGSEGV at pc 0x0000fffbde02d8ec in thread 197430

Base: 0x0000fffdde1f0000

Registers:      eflags=0x0000000080000000

version 11.2.0, custom build

-no_dynamic_options -client_lib '/home/wangtd/DynamoRIO-AArch64-Linux-11.2.0/samples/build/bin/libmemtrace_simple.so;0;' -client_lib64 '/home/wangtd/DynamoRIO-AArch64-Linux-11.2.0/samples/build/bin/libmemtrace_simple.so;0;' -code_api -stack_size 64K -signal_stack_size 64K -max_elide_jmp 0 -max_elide_call 0 -vmm_block_siz

0x0000fffbde48ed80 0x0000fffbde02ab40

0x0000fffbde48eef0 0x0000fffbde03169c

0x0000fffbde48f4c0 0x0000000072002de4

0x0000fffbde48f5e0 0x00000000403bd014>

What's strange is that the original version of memtrace_simple.c, before modifying the fprintf, could correctly pin a multi-threaded program. How should I go about trying to resolve this error? 

Thanks for your help!

[Doug Evans]

[Replying-all this time]

Nit: The 'l' in "%lf" is ignored.

I wonder if the presence of the additional arg is just a coincidence and the failure happens without it.

Could you maybe try to repro the failure in an unmodified tree, say 20-50 times?
Is the failure consistent with the change and consistently successful without it?

Including a session log of what you did would also help.

What version of linux is this?

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dynamorio-users/a5346edb-407f-4561-9160-fb4b9564a291n%40googlegroups.com.

[tengda w]

Hello!

The linux version is 4.19.90-2012.4.0.0053.oe1.aarch64.

I re-downloaded DynamoRIO-AArch64-Linux-11.2.0 and tested the original libmemtrace_simple.so with multi-thread program multiple times without encountering any issues. 

However, once I modified lines 141-145 in memtrace_simple.c and added the additional %f parameter, the problem occurred. 

The modified libmemtrace_simple.so can still successfully instrument single-threaded programs, but it has never worked for multi-threaded programs in many attempts.

What’s strange is that adding other parameters, such as "%d" or "%s", works fine.

In addition, when adding the %f parameter, unlike single-threaded programs, even setting the number of threads in the multi-threaded program to 1 still doesn’t allow it to run properly. I’m not sure if this is related to instrumentation with the <thread> library. 

lines 141-145:

// fprintf(data->logf, "" PIFX ": %2d, %s\n", (ptr_uint_t)mem_ref->addr,

fprintf(data->logf, "" PIFX ": %2d, %s, %f\n", (ptr_uint_t)mem_ref->addr,

    mem_ref->size,

    (mem_ref->type > REF_TYPE_WRITE)

        ? decode_opcode_name(mem_ref->type) /* opcode for instr */

        : (mem_ref->type == REF_TYPE_WRITE ? "w" : "r"), 0.324);

//      : (mem_ref->type == REF_TYPE_WRITE ? "w" : "r"));

The simple multi-threaded program used for instrumentation :

#include <iostream>

#include <thread>

#include <vector>

int thread_num = 2;

void worker(int id) {

    int tmp = 0;

    for(int i = 1;i <= id + 5;i++)

        tmp += i;

    std::cout << "Thread " << id << "res " << tmp << std::endl;

}

int main() {

    std::vector<std::thread> threads;

    for (int i = 0; i < thread_num; ++i) {

        threads.push_back(std::thread(worker, i));

    }

    for (auto& t : threads) {

        t.join();

    }

    return 0;

}

I hope this is helpful to you.

[Derek Bruening]

Supporting private loading (i.e., for client use) of glibc and of threading libraries like pthread (presumably used by thread) is not easy, especially on recent versions of glibc which have undocumented interactions with pthread and ld.so: https://github.com/DynamoRIO/dynamorio/issues/5437.  Our documentation (https://dynamorio.org/using.html#sec_extlibs) says that problems can occur when clients use the pthread library.  This may be a private library issue.

I would try using dr_fprintf instead of fprintf: probably the problem goes away then.

As with any SIGSEGV, get a callstack with symbols to actually see where the crash is so we don't have to speculate: https://dynamorio.org/page_debugging.html#autotoc_md145

To view this discussion visit https://groups.google.com/d/msgid/dynamorio-users/097d57dd-2953-4d9b-ab27-2bd7df84daf4n%40googlegroups.com.

[Derek Bruening]

The fprintf crash may be this one: https://github.com/DynamoRIO/dynamorio/issues/7293.  As noted, get a callstack to see.

[tengda w]

Hi, Derek!

Sorry for the late reply.

Thank you very much for your response. Due to a job transfer, I am no longer responsible for DR-related projects and no longer have access to the original device, so I am unable to provide further feedback about this crash.

I sincerely apologize.