DR and MPI

[Mohammad Ewais]

Hi,

There have been previous questions about using DR with MPI, mostly focusing on instrumenting an MPI application. I have a different question.

rather than the target app, I would like my DR client itself to be MPI capable! I want to be able to send and receive MPI messages inside the client. I tried the Naive approach of simply throwing MPI_Init inside my dr_client_main, obviously it did not work or I would not be asking here :)

Here's how I run:

mpirun -n 1 --host host1 /home/mewais/DynamoRIO/bin64/drrun -debug -c /home/mewais/SNE/Build/libSNEClient.so --so-many-args -- ./Test1 : -n 1 --host host2 /home/mewais/DynamoRIO/bin64/drrun -debug -c /home/mewais/SNE/Build/libSNEClient.so --so-many-args -- ./Test2

and the output I get (one of the two):

<Initial options = -no_dynamic_options -client_lib '/home/mewais/SNE/Build/libSNEClient.so;0;"--so-many-args"' -code_api -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >

<CURIOSITY : privload_recurse_cnt < 20 in file /home/travis/build/DynamoRIO/dynamorio/core/loader_shared.c line 652

version 8.0.0, build 1

0x00007ffee900fd90 0x00000000711d861a

0x00007ffee900fe00 0x00000000712ecbc9

0x00007ffee9010030 0x00000000712ec5b3

0x00007ffee9010090 0x00000000711d8fe8

0x00007ffee90100b0 0x00000000711d889f

0x00007ffee9010110 0x00000000712ecbc9

0x00007ffee9010340 0x00000000712ec5b3

0x00007ffee90103a0 0x00000000711d8fe8

0x00007ffee90103c0 0x00000000711d7169

0x00007ffee90105f0 0x00000000711d7346

0x00007ffee9010830 0x0000000071052844

0x00007ffee9011060 0x00000000712efc4d

0x00007ffee9012110 0x000000007129f1bd

/home/mewais/SNE/Build/libSNEClient.so=0x00007f5d4ee2b000

/home/mewais/DynamoRIO/ext/lib64/debug/libdrwrap.so=0x00007f5d4f273000

/lib/x86_64-linux-gnu/libmpi_cxx.so.40=0x00007f5d93087000

/lib/x86_64-linux-gnu/libstdc++.so.6=0x00007f5d924dd000

/lib/x86_64-linux-gnu/libgcc_s.so.1=0x00007f5d928f0000

/opt/lib/libmpi.so.40=0x00007f5d92f5d000

/opt/lib/libopen-rte.so.40=0x00007f5d92df2000

/lib/x86_64-linux-gnu/libz.so.1=0x00007f5d92ef50>

<Paste into GDB to debug DynamoRIO clients:

set confirm off

add-symbol-file '/home/mewais/SNE/Build/libSNEClient.so' 0x00007f5d4f0b7b50

add-symbol-file '/home/mewais/DynamoRIO/lib64/debug/libdynamorio.so' 0x0000000071040fe0

add-symbol-file '/lib/x86_64-linux-gnu/libmpi_cxx.so.40' 0x00007f5d93097660

add-symbol-file '/opt/lib/libmpi.so.40' 0x00007f5d92f893f0

add-symbol-file '/opt/lib/libopen-rte.so.40' 0x00007f5d92e0d6e0

add-symbol-file '/opt/lib/libopen-pal.so.40' 0x00007f5d92d5db50

add-symbol-file '/lib/x86_64-linux-gnu/libdl.so.2' 0x00007f5d92f57220

add-symbol-file '/lib/x86_64-linux-gnu/libc.so.6' 0x00007f5d9298e650

add-symbol-file '/usr/lib64/ld-linux-x86-64.so.2' 0x00007f5d92f1f090

add-symbol-file '/lib/x86_64-linux-gnu/libutil.so.1' 0x00007f5d92f193e0

add-symbol-file '/lib/x86_64-linux-gnu/libhwloc.so.15' 0x00007f5d92ce6d10

add-symbol-file '/lib/x86_64-linux-gnu/libm.so.6' 0x00007f5d92ba03c0

add-symbol-file '/lib/x86_64-linux-gnu/libudev.so.1' 0x00007f5d92b6be50

add-symbol-file '/lib/x86_64-linux-gnu/libpthread.so.0' 0x00007f5d9294ca60

add-symbol-file '/lib/x86_64-linux-gnu/libevent_core-2.1.so.7' 0x00007f5d92914b00

add-symbol-file '/lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7' 0x00007f5d92f132e0

add-symbol-file '/lib/x86_64-linux-gnu/libz.so.1' 0x00007f5d92ef7280

add-symbol-file '/lib/x86_64-linux-gnu/libstdc++.so.6' 0x00007f5d9257e2e0

add-symbol-file '/lib/x86_64-linux-gnu/libgcc_s.so.1' 0x00007f5d928f35e0

add-symbol-file '/home/mewais/DynamoRIO/ext/lib64/debug/libdrwrap.so' 0x00007f5d4f276150

add-symbol-file '/home/mewais/DynamoRIO/ext/lib64/debug/libdrmgr.so' 0x00007f5d4f4855f0

>

<(1+x) Handling our fault in a TRY at 0x000000007129f6b2>

<Application ./Test1 (29572).  Tool internal crash at PC 0x00007f5d92daeda0.  Please report this at your tool's issue tracker.  Program aborted.

Received SIGSEGV at pc 0x00007f5d92daeda0 in thread 29572

Base: 0x0000000071000000

Registers:eax=0x00007f5c00000000 ebx=0x00007f5d92dcac20 ecx=0x0000000000002000 edx=0x0000000000000000

esi=0x0000000000000001 edi=0x00007f5d92decde0 esp=0x00007ffee9010020 ebp=0x00007f5d92dcaf00

r8 =0x00007f5d92decdf8 r9 =0x00007f5ccef31ef0 r10=0x0000000000000000 r11=0x00007f5ccef2dee0

r12=0x0000000000000000 r13=0x00007f5d92dcafc0 r14=0x00007f5d92dcadd0 r15=0x00007f5d92dcad00

eflags=0x0000000000010206

version 8.0.0, build 1

-no_dynamic_options -client_lib '/home/mewais/SNE/Build/libSNEClient.so;0;"--so-many-args"' -code_api -stack_size 56K -signal_stack_size 32K -

0x00007f5d92dcaf00 0x0003000200010001>

--------------------------------------------------------------------------

Primary job  terminated normally, but 1 process returned

a non-zero exit code. Per user-direction, the job has been aborted.

--------------------------------------------------------------------------

I am still new to DR, so I could not really gdb into this yet, but the whole thing just works if I comment out MPI_Init so I am pretty sure it is the cause (I also have NOT yet added any other MPI calls)

Is there a solution to this? I can think of some workarounds, like maybe having a separate process be solely responsible for MPI, and make the client talk to it via sockets or pipes or so, would that be possible? Is there a way to get MPI and DR to work nicely together without having to go through complex and potentially slow workarounds?

[Derek Bruening]

DR tries to load private copies of libraries the client wants to use and to isolate those libraries from the application, but there can be complications.  You would have to get into the details to see exactly what the failure is and whether it is easy to solve: sometimes an extra import redirect can solve it; sometimes it is not simple, especially on Windows with various layers of system services getting involved.

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/b20ff0e5-33c3-44ed-a7d7-81451d08476bn%40googlegroups.com.

[Mohammad Ewais]

Thanks a lot Derek for your help, appreciate it.

I have tried diving into this and debugging it, the stacktrace from GDB is as follows:

#0  0x00007ffff7d16ee0 in yy_get_next_buffer () at keyval_lex.c:1284

#1  opal_util_keyval_yylex () at keyval_lex.c:1155

#2  0x00007ffff7d0b80a in parse_line () at ../../../../opal/util/keyval_parse.c:162

#3  opal_util_keyval_parse (filename=filename@entry=0x7fff34038930 "/etc/openmpi/openmpi-mca-params.conf", callback=callback@entry=0x7ffff7cff340 <save_value>) at ../../../../opal/util/keyval_parse.c:105

#4  0x00007ffff7cff4be in mca_base_parse_paramfile (paramfile=paramfile@entry=0x7fff34038930 "/etc/openmpi/openmpi-mca-params.conf", list=list@entry=0x7ffff7d52280) at ../../../../../opal/mca/base/mca_base_parse_paramfile.c:43

#5  0x00007ffff7cf6dea in read_files (file_list=<optimized out>, file_values=file_values@entry=0x7ffff7d52280, sep=sep@entry=44 ',') at ../../../../../opal/mca/base/mca_base_var.c:1269

#6  0x00007ffff7cf9b7e in mca_base_var_cache_files (rel_path_search=rel_path_search@entry=false) at ../../../../../opal/mca/base/mca_base_var.c:538

#7  0x00007ffff7cd4267 in opal_init_util (pargc=pargc@entry=0x7fffffffbb0c, pargv=pargv@entry=0x7fffffffbb00) at ../../../opal/runtime/opal_init.c:425

#8  0x00007ffff7eef531 in ompi_mpi_init (argc=<optimized out>, argc@entry=0, argv=<optimized out>, argv@entry=0x0, requested=0, provided=0x7fffffffbbc4, reinit_ok=<optimized out>) at ../../../ompi/runtime/ompi_mpi_init.c:428

#9  0x00007ffff7e92551 in PMPI_Init (argc=0x0, argv=0x0) at pinit.c:69

#10 0x00007fffb42232d3 in SNE::UTILS::MPIInit (argc=0x0, argv=0x0) at /work/mewais/projects/DCArch/SingleNodeEmulator/Sources/Utils/MPI.cpp:21

#11 0x00007fffb423fb5a in dr_client_main (id=0, argc=13, argv=0x7fff33f81aa0) at /work/mewais/projects/DCArch/SingleNodeEmulator/Sources/SNEClient.cpp:53

#12 0x00000000711e20d7 in instrument_init () at /home/travis/build/DynamoRIO/dynamorio/core/lib/instrument.c:770

#13 0x00000000710528a0 in dynamorio_app_init () at /home/travis/build/DynamoRIO/dynamorio/core/dynamo.c:680

#14 0x00000000712efc4d in privload_early_inject (sp=0x7fffffffda60, old_libdr_base=0x7ffff797f000 "fD\017o\322fD\017o\336fD\017d\305fD\017d\311fD\017d\325fD\017d\332fE\017\333\301fE\017\333\323fD\017\333\307fD\017\333\327fA\017\353\310fA\017\353\322f\017t\301f\017t\312f\017\370\310f\017\327с\352\377\377", old_libdr_size=6815744) at /home/travis/build/DynamoRIO/dynamorio/core/unix/loader.c:2012

#15 0x000000007129f1bd in reloaded_xfer () at /home/travis/build/DynamoRIO/dynamorio/core/arch/x86/x86.asm:1179

#16 0x0000000000000001 in ?? ()

#17 0x00007fffffffddb3 in ?? ()

#18 0x0000000000000000 in ?? ()

The error is internal to OpenMPI, although obviously caused by DR as it never shows up on its own. 

The problem, as you can see in the stack trace, is that it happens at flex/bison (or lex/yacc) generated files, so it is very hard to follow. But, I could see that they have a lot of mallocs, reallocs, file reads, etc. I have seen DR alternatives for these functions, so not sure if they are safe for use by clients? Can this be causing the issues?

One more thing: I also just noticed this line in the DR output: <(1+x) Handling our fault in a TRY at 0x000000007129f6b2>

Any significance to this? Does it give any clues about what could the cause be?

PS: There is also a lot of calls to "fork" in the OpenMPI files, although none of which have been hit so far. Would these cause problems in the future too?

[Derek Bruening]

On Tue, Jun 22, 2021 at 5:20 AM Mohammad Ewais <mohammad...@gmail.com> wrote:

Thanks a lot Derek for your help, appreciate it.

I have tried diving into this and debugging it, the stacktrace from GDB is as follows:

#0  0x00007ffff7d16ee0 in yy_get_next_buffer () at keyval_lex.c:1284

#1  opal_util_keyval_yylex () at keyval_lex.c:1155

#2  0x00007ffff7d0b80a in parse_line () at ../../../../opal/util/keyval_parse.c:162

#3  opal_util_keyval_parse (filename=filename@entry=0x7fff34038930 "/etc/openmpi/openmpi-mca-params.conf", callback=callback@entry=0x7ffff7cff340 <save_value>) at ../../../../opal/util/keyval_parse.c:105

#4  0x00007ffff7cff4be in mca_base_parse_paramfile (paramfile=paramfile@entry=0x7fff34038930 "/etc/openmpi/openmpi-mca-params.conf", list=list@entry=0x7ffff7d52280) at ../../../../../opal/mca/base/mca_base_parse_paramfile.c:43

#5  0x00007ffff7cf6dea in read_files (file_list=<optimized out>, file_values=file_values@entry=0x7ffff7d52280, sep=sep@entry=44 ',') at ../../../../../opal/mca/base/mca_base_var.c:1269

#6  0x00007ffff7cf9b7e in mca_base_var_cache_files (rel_path_search=rel_path_search@entry=false) at ../../../../../opal/mca/base/mca_base_var.c:538

#7  0x00007ffff7cd4267 in opal_init_util (pargc=pargc@entry=0x7fffffffbb0c, pargv=pargv@entry=0x7fffffffbb00) at ../../../opal/runtime/opal_init.c:425

#8  0x00007ffff7eef531 in ompi_mpi_init (argc=<optimized out>, argc@entry=0, argv=<optimized out>, argv@entry=0x0, requested=0, provided=0x7fffffffbbc4, reinit_ok=<optimized out>) at ../../../ompi/runtime/ompi_mpi_init.c:428

#9  0x00007ffff7e92551 in PMPI_Init (argc=0x0, argv=0x0) at pinit.c:69

#10 0x00007fffb42232d3 in SNE::UTILS::MPIInit (argc=0x0, argv=0x0) at /work/mewais/projects/DCArch/SingleNodeEmulator/Sources/Utils/MPI.cpp:21

#11 0x00007fffb423fb5a in dr_client_main (id=0, argc=13, argv=0x7fff33f81aa0) at /work/mewais/projects/DCArch/SingleNodeEmulator/Sources/SNEClient.cpp:53

#12 0x00000000711e20d7 in instrument_init () at /home/travis/build/DynamoRIO/dynamorio/core/lib/instrument.c:770

#13 0x00000000710528a0 in dynamorio_app_init () at /home/travis/build/DynamoRIO/dynamorio/core/dynamo.c:680

#14 0x00000000712efc4d in privload_early_inject (sp=0x7fffffffda60, old_libdr_base=0x7ffff797f000 "fD\017o\322fD\017o\336fD\017d\305fD\017d\311fD\017d\325fD\017d\332fE\017\333\301fE\017\333\323fD\017\333\307fD\017\333\327fA\017\353\310fA\017\353\322f\017t\301f\017t\312f\017\370\310f\017\327с\352\377\377", old_libdr_size=6815744) at /home/travis/build/DynamoRIO/dynamorio/core/unix/loader.c:2012

#15 0x000000007129f1bd in reloaded_xfer () at /home/travis/build/DynamoRIO/dynamorio/core/arch/x86/x86.asm:1179

#16 0x0000000000000001 in ?? ()

#17 0x00007fffffffddb3 in ?? ()

#18 0x0000000000000000 in ?? ()

The error is internal to OpenMPI, although obviously caused by DR as it never shows up on its own. 

The problem, as you can see in the stack trace, is that it happens at flex/bison (or lex/yacc) generated files, so it is very hard to follow. But, I could see that they have a lot of mallocs, reallocs, file reads, etc. I have seen DR alternatives for these functions, so not sure if they are safe for use by clients? Can this be causing the issues?

You can see the current list of redirects here: https://github.com/DynamoRIO/dynamorio/blob/master/core/unix/loader.c#L1338

 

One more thing: I also just noticed this line in the DR output: <(1+x) Handling our fault in a TRY at 0x000000007129f6b2>

Any significance to this? Does it give any clues about what could the cause be?

This happens in some cases on every run on some is-this-an-ELF queries and is unlikely to be significant.

 

PS: There is also a lot of calls to "fork" in the OpenMPI files, although none of which have been hit so far. Would these cause problems in the future too?

DR won't know to follow a native (non-managed) fork.  The two potential problems with any non-managed, non-redirected action are A) if they use non-reentrant app libraries or tweak app resources which can break the app or deadlock or whatnot; B) if they modify the app's state that DR cares about: creating threads; changing memory protections; allocating memory; that sort of thing done by a client without going through DR will confuse DR.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/3bd2af05-1a78-4698-88bc-16eddf976f71n%40googlegroups.com.

[Mohammad Ewais]

I've been at this for quite a while. At this point, I do not think it will work, at least not with my level of knowledge. I also gave a quick try with MPICH instead of OpenMPI and this time DR was the one that failed.

Anyway, thanks a lot for the help and guidance, Derek.

I think it makes more since now to have a sister process that's not DR managed handle MPI communications. That said, I want to know if there's any (safe) way for the DR client to communicate with this process?

[Derek Bruening]

I think any type of IPC would work: pipes and shared memory have been used in the past.  Anything with threads or signals or memory allocation in managed processes is best done through DR is all.

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/4719042e-fc52-4aac-bce9-183ce8beef6fn%40googlegroups.com.

[Mohammad Ewais]

Thanks for the help Derek. 

FYI: I have successfully moved to using shared memory IPC and it is going well.