Running DR on a system with multiple libc
51 views
Skip to first unread message
Mohammad Ewais
Feb 4, 2025, 12:27:03 PMFeb 4
to DynamoRIO Users
Hi,
I have successfully built and ran my client on a local machine with no issues. I am now trying to run my client on a SLURM cluster. Naturally, they provide all sorts of softwares through a module system, which includes multiple versions of libc. I load the module for gcc 11.3.0 which also loads libc 2.30 (actively avoiding any libc after 2.34).
I have successfully built and ran my client on a local machine with no issues. I am now trying to run my client on a SLURM cluster. Naturally, they provide all sorts of softwares through a module system, which includes multiple versions of libc. I load the module for gcc 11.3.0 which also loads libc 2.30 (actively avoiding any libc after 2.34).
The system has a /lib64/libc.so.6 version 2.28. When we load modules, the modules exist under other paths like /cvmfs/soft.computecanada.ca/gentoo/2020/lib64/. The compilers are configured to use the loaded modules instead of the /lib64 path. For example, if I do gcc -v this is the output:
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/11.3.0/libexec/gcc/x86_64-pc-linux-gnu/11.3.0/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
Target: x86_64-pc-linux-gnu
Configured with: ../configure --with-sysroot=/cvmfs/soft.computecanada.ca/gentoo/2020 --enable-languages=c,c++,fortran --without-cuda-driver --enable-offload-targets=nvptx-none --enable-lto --enable-checking=release --disable-multilib --enable-shared=yes --enable-static=yes --enable-threads=posix --enable-plugins --enable-gold --enable-ld=default --prefix=/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/11.3.0 --with-local-prefix=/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/11.3.0 --enable-bootstrap --with-isl=/tmp/ebuser/avx2/GCCcore/11.3.0/system-system/gcc-11.3.0/stage2_stuff --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.3.0 (GCC)
And all the binaries included under their modules are compiled with RPATH to use the correct libraries (at least according to their documentation).
Using this, my CMake build runs fine. The client links with a couple of static libraries that are also built under the same configuration, plus of course a dynamic libc. After the build, I run this ldd Release/libDCSClient.so and I get the following output, which seems to be fine:
Trial 1
The problem I have is that when I run, DR seems to load the incorrect libc. I ran with the -debug flag and got this as part of my output. You can see the multiple lookup errors, the path to the wrong libc, and the segmentation fault at the first use of libc in my client:
Trial 2
I also exec my client through another process, so I tried adding this LD_LIBRARY_PATH to the exec: LD_LIBRARY_PATH=/cvmfs/soft.computecanada.ca/gentoo/2020/lib64/:/cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0/. This has the exact same paths reported by ldd above. This time, DR doesn't even have the time to print any debugging output, instead it dies with just this:
Inconsistency detected by ld.so: dl-call-libc-early-init.c: 37: _dl_call_libc_early_init: Assertion `sym != NULL' failed!
Trial 3
Since LD_LIBRARY_PATH didn't work, I tried setting -Wl,-rpath directly during build. I add this to my build target_link_options(DCSClient PUBLIC -Wl,-rpath /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0/ -Wl,-rpath=/cvmfs/soft.computecanada.ca/gentoo/2020/lib64/) and again these are the same two paths reported by ldd above. Now, when I run DR with -debug flag I get the following:
You can see there are no more symbol lookup errors. You can also see that libgcc, libstdc++ and libm get loaded from the correct path now. But libc does not (even though it exists at the same path as libm!), same for ld-linux-x86-64 though I don't know if that carries any significance as DR uses its private loader?!
Trial 4
Because using -rpath didn't work as well, I tried explicitly linking to all libc libraries with their full path by doing the following in CMake:
but it seems that the DR linker does not respect these as the output looks exactly like that of Trial 1.
So, it seems Trial 3 is the closest to solving the issue, but is still incomplete. How do I get DR to load the correct libc?
Using this, my CMake build runs fine. The client links with a couple of static libraries that are also built under the same configuration, plus of course a dynamic libc. After the build, I run this ldd Release/libDCSClient.so and I get the following output, which seems to be fine:
linux-vdso.so.1 (0x00007f14aa40e000)
libdrwrap.so => not found
libdrutil.so => not found
libdrx.so => not found
libdrreg.so => not found
libdrsyms.so => not found
libdrmgr.so => not found
libdynamorio.so => not found
libstdc++.so.6 => /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0/libstdc++.so.6 (0x00007f14aa12d000)
libgcc_s.so.1 => /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0/libgcc_s.so.1 (0x00007f14a9fcf000)
/cvmfs/soft.computecanada.ca/gentoo/2020/lib64/ld-linux-x86-64.so.2 (0x00007f14aa410000)
Trial 1
The problem I have is that when I run, DR seems to load the incorrect libc. I ran with the -debug flag and got this as part of my output. You can see the multiple lookup errors, the path to the wrong libc, and the segmentation fault at the first use of libc in my client:
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNKSt10filesystem7__cxx114path5_List13_Impl_deleterclEPNS2_5_ImplE>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNSt10filesystem7__cxx114path5_ListC1Ev>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNSt10filesystem6statusERKNS_7__cxx114pathE>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEC1Ev>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZSt28__throw_bad_array_new_lengthv>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNSt10filesystem18create_directoriesERKNS_7__cxx114pathE>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNSt10filesystem12current_pathERKNS_7__cxx114pathE>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNSt10filesystem7__cxx114path14_M_split_cmptsEv>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNKSt10filesystem7__cxx114path11parent_pathEv>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNSt10filesystem12current_pathB5cxx11Ev>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNKRSt7__cxx1115basic_stringbufIcSt11char_traitsIcESaIcEE3strEv>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNKSt10filesystem7__cxx114path7compareERKS1_>
<Paste into GDB to debug DynamoRIO clients:
set confirm off
add-symbol-file '/lustre06/project/6005345/mewais/DCSim/Workspace/SingleNode_Local128GB/GBBS/../../../Release/libDCSClient.so' 0x000000007257cb50
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/lib64/debug/libdynamorio.so' 0x00007fc1b2558fe0
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/ext/lib64/debug/libdrwrap.so' 0x0000000074003150
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/ext/lib64/debug/libdrmgr.so' 0x000000003f4a85f0
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/ext/lib64/debug/libdrutil.so' 0x00000000750015c0
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/ext/lib64/debug/libdrx.so' 0x0000000077003d20
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/ext/lib64/debug/libdrreg.so' 0x00000000780028d0
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/ext/lib64/debug/libdrsyms.so' 0x0000000076009b10
add-symbol-file '/usr/lib64/libc.so.6' 0x00007fc131f24c40
add-symbol-file '/usr/lib64/ld-linux-x86-64.so.2' 0x00007fc131cd1080
add-symbol-file '/usr/lib64/libstdc++.so.6' 0x00007fc1319c9b90
add-symbol-file '/usr/lib64/libm.so.6' 0x00007fc1315c3520
add-symbol-file '/usr/lib64/libgcc_s.so.1' 0x00007fc1313a0e00
>
<Application /lustre06/project/6005345/mewais/DCSim/Bench/gbbs/build/execroot/__main__/bazel-out/k8-opt/bin/benchmarks/BFS/NonDeterministicBFS/BFS_main (2547091). Tool internal crash at PC 0x0000000072895ae7. Please report this at your tool's issue tracker. Program aborted.
Received SIGSEGV at client library pc 0x0000000072895ae7 in thread 2547091
Base: 0x00007fc1b2518000
Registers:eax=0x0000000000000000 ebx=0x000000003f49fb28 ecx=0x00007fc1325ce8e0 edx=0x0000000000000000
esi=0x0000000000000001 edi=0x000000007305e9f0 esp=0x00007ffd6ab30090 ebp=0x00007ffd6ab30130
r8 =0x00007fc1325ce940 r9 =0x00007ffd6ab2fe60 r10=0x00007ffd6ab2fe91 r11=0x0000000000000246
r12=0x00007ffd6ab30110 r13=0x00007fc132520540 r14=0x00007ffd6ab30120 r15=0x000000007305ea00
eflags=0x0000000000010246
version 8.0.0, build 1
-no_dynamic_options -disasm_mask 1 -loglevel 4 -client_lib '/lustre06/project/6005345/mewais/DCSim/Workspace/SingleNode_Local128GB/GBBS/../../../Release/libDCSClient.so;0;"--config" "/lustre06/project/6005345/mewais/DCSim/Workspace/SingleNode_Local128GB/GBBS/BFS.cfg" "--node-name" "Node/NV_1C_64C_S_128GB_Node-0" "--proce
0x00007ffd6ab30130 0x0000000000000012
0x000000003f49fb58 0x0000000000000000
/lustre06/project/6005345/mewais/DCSim/Workspace/SingleNode_Local128GB/GBBS/../../../Release/libDCSClient.so=0x0000000072000000
/usr/lib64/libstdc++.so.6=0x00007fc13193a000
/usr/lib64/libgcc_s.so.1=0x00007fc13139e000
/usr/lib64/libm.so.6=0x00007fc1315b7000
/lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/ext/lib64/debug/libdrsyms.so=0x0000000076000000
/usr/lib64/libc.so.6=0x00007fc131f03000
/usr/lib64/ld-linux-x86-64.so.2=0x00007fc131cd0>
I also exec my client through another process, so I tried adding this LD_LIBRARY_PATH to the exec: LD_LIBRARY_PATH=/cvmfs/soft.computecanada.ca/gentoo/2020/lib64/:/cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0/. This has the exact same paths reported by ldd above. This time, DR doesn't even have the time to print any debugging output, instead it dies with just this:
Inconsistency detected by ld.so: dl-call-libc-early-init.c: 37: _dl_call_libc_early_init: Assertion `sym != NULL' failed!
Trial 3
Since LD_LIBRARY_PATH didn't work, I tried setting -Wl,-rpath directly during build. I add this to my build target_link_options(DCSClient PUBLIC -Wl,-rpath /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0/ -Wl,-rpath=/cvmfs/soft.computecanada.ca/gentoo/2020/lib64/) and again these are the same two paths reported by ldd above. Now, when I run DR with -debug flag I get the following:
<Paste into GDB to debug DynamoRIO clients:
set confirm off
add-symbol-file '/lustre06/project/6005345/mewais/DCSim/Workspace/SingleNode_Local128GB/GBBS/../../../Release/libDCSClient.so' 0x000000007257cb50
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/lib64/debug/libdynamorio.so' 0x00007fae95f4afe0
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/ext/lib64/debug/libdrwrap.so' 0x0000000074003150
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/ext/lib64/debug/libdrmgr.so' 0x00000000486015f0
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/ext/lib64/debug/libdrutil.so' 0x00000000750015c0
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/ext/lib64/debug/libdrx.so' 0x0000000077003d20
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/ext/lib64/debug/libdrreg.so' 0x00000000780028d0
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/ext/lib64/debug/libdrsyms.so' 0x0000000076009b10
add-symbol-file '/usr/lib64/libc.so.6' 0x00007fae15916c40
add-symbol-file '/usr/lib64/ld-linux-x86-64.so.2' 0x00007fae156c3080
add-symbol-file '/cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0//libstdc++.so.6' 0x00007fae154c11b0
add-symbol-file '/cvmfs/soft.computecanada.ca/gentoo/2020/lib64//libm.so.6' 0x00000000488181e0
add-symbol-file '/cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0//libgcc_s.so.1' 0x0000000048952310
>
Received SIGSEGV at client library pc 0x0000000072895ae7 in thread 3288107
Base: 0x00007fae95f0a000
Registers:eax=0x0000000000000000 ebx=0x00000000485f8b28 ecx=0x00007fae15fc1020 edx=0x0000000000000000
esi=0x0000000000000001 edi=0x000000007305e9f0 esp=0x00007ffe888495d0 ebp=0x00007ffe88849670
r8 =0x000000007305ea48 r9 =0x00007ffe888493a0 r10=0x00007ffe888493d1 r11=0x0000000000000246
r12=0x00007ffe88849650 r13=0x00007fae15f12540 r14=0x00007ffe88849660 r15=0x000000007305ea00
eflags=0x0000000000010246
version 8.0.0, build 1
-no_dynamic_options -disasm_mask 1 -loglevel 4 -client_lib '/lustre06/project/6005345/mewais/DCSim/Workspace/SingleNode_Local128GB/GBBS/../../../Release/libDCSClient.so;0;"--config" "/lustre06/project/6005345/mewais/DCSim/Workspace/SingleNode_Local128GB/GBBS/BFS.cfg" "--node-name" "Node/NV_1C_64C_S_128GB_Node-0" "--proce
0x00007ffe88849670 0x0000000000000012
0x00000000485f8b58 0x0000000000000000
/lustre06/project/6005345/mewais/DCSim/Workspace/SingleNode_Local128GB/GBBS/../../../Release/libDCSClient.so=0x0000000072000000
/lustre06/proje>
You can see there are no more symbol lookup errors. You can also see that libgcc, libstdc++ and libm get loaded from the correct path now. But libc does not (even though it exists at the same path as libm!), same for ld-linux-x86-64 though I don't know if that carries any significance as DR uses its private loader?!
Trial 4
Because using -rpath didn't work as well, I tried explicitly linking to all libc libraries with their full path by doing the following in CMake:
target_link_libraries(DCSClient /cvmfs/soft.computecanada.ca/gentoo/2020/lib64/libc.so.6)
target_link_libraries(DCSClient /cvmfs/soft.computecanada.ca/gentoo/2020/lib64/libm.so.6)
target_link_libraries(DCSClient /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0/libstdc++.so.6)
target_link_libraries(DCSClient /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0/libgcc_s.so.1)
target_link_libraries(DCSClient /cvmfs/soft.computecanada.ca/gentoo/2020/lib64/ld-linux-x86-64.so.2)
but it seems that the DR linker does not respect these as the output looks exactly like that of Trial 1.
So, it seems Trial 3 is the closest to solving the issue, but is still incomplete. How do I get DR to load the correct libc?
Derek Bruening
Feb 4, 2025, 4:29:07 PMFeb 4
to Mohammad Ewais, DynamoRIO Users
Please see https://dynamorio.org/using.html#sec_rpath which documents how to set a custom search path via DynamoRIO_RPATH. Your custom link flag may collide with the cmake code setting the client link flags: best to use the supported route. Did you confirm DT_RPATH was set?
Use debug logging and the open source code to see exactly what is happening: the search sequences is at https://github.com/DynamoRIO/dynamorio/blob/master/core/unix/loader.c#L1034. Logs will print out exactly where it's searching for each library.
--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dynamorio-users/0a71229d-7d01-4747-9b72-046e8daed46cn%40googlegroups.com.
Mohammad Ewais
Feb 4, 2025, 4:45:31 PMFeb 4
to DynamoRIO Users
Hey Derek,
First of all, thanks a lot for your help.
Apologies for not mentioning this in my original question, but I had actually tried using the RPATH solution you pointed to, so I did this in my CMakeLists.txt:
I also exported DT_RPATH in my environment as follows:
export DT_RPATH=/cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0/:/cvmfs/soft.computecanada.ca/gentoo/2020/lib64/
Not that it matters because CMake fails with the following error repeated for the configure_DynamoRIO_client and use_DynamoRIO_extension lines:
First of all, thanks a lot for your help.
Apologies for not mentioning this in my original question, but I had actually tried using the RPATH solution you pointed to, so I did this in my CMakeLists.txt:
set(DynamoRIO_RPATH ON)
configure_DynamoRIO_client(DCSClient)
I also exported DT_RPATH in my environment as follows:
export DT_RPATH=/cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0/:/cvmfs/soft.computecanada.ca/gentoo/2020/lib64/
Not that it matters because CMake fails with the following error repeated for the configure_DynamoRIO_client and use_DynamoRIO_extension lines:
CMake Error at /home/mewais/projects/def-pc/mewais/DynamoRIO-Linux-8.0.0-1/cmake/DynamoRIOConfig.cmake:321 (file):
file RELATIVE_PATH must be passed a full path to the directory:
target_loc-NOTFOUND
Call Stack (most recent call first):
/home/mewais/projects/def-pc/mewais/DynamoRIO-Linux-8.0.0-1/cmake/DynamoRIOConfig.cmake:1078 (DynamoRIO_add_rel_rpaths)
/home/mewais/projects/def-pc/mewais/DynamoRIO-Linux-8.0.0-1/cmake/DynamoRIOConfig.cmake:1159 (_DR_set_compile_flags)
CMakeLists.txt:240 (configure_DynamoRIO_client)
CMake Error at /home/mewais/projects/def-pc/mewais/DynamoRIO-Linux-8.0.0-1/cmake/DynamoRIOConfig.cmake:321 (file):
file RELATIVE_PATH must be passed a full path to the directory:
target_loc-NOTFOUND
Call Stack (most recent call first):
/home/mewais/projects/def-pc/mewais/DynamoRIO-Linux-8.0.0-1/cmake/DynamoRIOConfig.cmake:1410 (DynamoRIO_add_rel_rpaths)
CMakeLists.txt:241 (use_DynamoRIO_extension)
Mohammad Ewais
Feb 4, 2025, 11:25:50 PMFeb 4
to DynamoRIO Users
I don't fully understand what's going on in the debug outputs!
Here's the relevant part of my CMakeLists.txt
Which causes libm and libstdc++ to be loaded correctly. But not libc (Similar to Trial 3 above). The relevant parts of the output show the following (same order):
- libc completely ignores the rpath supplied by CMake, and starts looking into the hardcoded paths in https://github.com/DynamoRIO/dynamorio/blob/release_8.0.0-1/core/unix/loader.c#L75
- libstdc++ is correctly using rpath, also adds the paths to the search list and finds the library in the correct place
Here's the relevant part of my CMakeLists.txt
target_compile_definitions(DCSClient PUBLIC DCSCLIENT)
target_include_directories(DCSClient PUBLIC $ENV{EBROOTGENTOO}/lib/gcc/x86_64-pc-linux-gnu/11.3.0/include/g++-v11)
target_include_directories(DCSClient PUBLIC $ENV{EBROOTGENTOO}/lib/gcc/x86_64-pc-linux-gnu/11.3.0/include/g++-v11/x86_64-pc-linux-gnu)
target_include_directories(DCSClient PUBLIC $ENV{EBROOTGENTOO}/lib/gcc/x86_64-pc-linux-gnu/11.3.0/include/g++-v11/backward)
target_include_directories(DCSClient PUBLIC $ENV{EBROOTGENTOO}/lib/gcc/x86_64-pc-linux-gnu/11.3.0/include)
target_include_directories(DCSClient PUBLIC $ENV{EBROOTGENTOO}/lib/gcc/x86_64-pc-linux-gnu/11.3.0/include-fixed)
target_include_directories(DCSClient PUBLIC $ENV{EBROOTGENTOO}/include)
target_link_directories(DCSClient PUBLIC $ENV{EBROOTGENTOO}/lib64)
target_link_directories(DCSClient PUBLIC $ENV{EBROOTGENTOO}/lib)
target_link_directories(DCSClient PUBLIC $ENV{EBROOTGENTOO}/../lib64)
target_link_directories(DCSClient PUBLIC $ENV{EBROOTGENTOO}/../lib)
target_link_directories(DCSClient PUBLIC $ENV{EBROOTGENTOO}/lib/gcc/x86_64-pc-linux-gnu/11.3.0/)
target_link_directories(DCSClient PUBLIC $ENV{EBROOTGENTOO}/x86_64-pc-linux-gnu/lib/)
target_link_directories(DCSClient PUBLIC ${EXTERNAL_INSTALL_LOCATION}/lib)
target_link_directories(DCSClient PUBLIC ${EXTERNAL_INSTALL_LOCATION}/lib64)
target_link_options(DCSClient PUBLIC -Wl,-rpath=$ENV{EBROOTGENTOO}/lib64
-Wl,-rpath=$ENV{EBROOTGENTOO}/lib
-Wl,-rpath=$ENV{EBROOTGENTOO}/../lib64
-Wl,-rpath=$ENV{EBROOTGENTOO}/../lib
-Wl,-rpath=$ENV{EBROOTGENTOO}/lib/gcc/x86_64-pc-linux-gnu/11.3.0/
-Wl,-rpath=$ENV{EBROOTGENTOO}/x86_64-pc-linux-gnu/lib/)
configure_DynamoRIO_client(DCSClient)
use_DynamoRIO_extension(DCSClient drwrap)
use_DynamoRIO_extension(DCSClient drmgr)
use_DynamoRIO_extension(DCSClient drutil)
use_DynamoRIO_extension(DCSClient drx)
use_DynamoRIO_extension(DCSClient drreg)
use_DynamoRIO_extension(DCSClient drsyms)
privload_locate: looking for /lustre06/project/6005345/mewais/DCSim/Workspace/SingleNode_Local128GB/GBBS/../../../Release/libc.so.6
privload_locate: looking for /lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/ext/lib64/debug/libc.so.6
privload_locate: looking for /lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/drmemory/drmf/lib64/debug/libc.so.6
privload_locate: looking for ./libc.so.6
privload_locate: looking for /lib/tls/i686/cmov/libc.so.6
privload_locate: looking for /usr/lib/libc.so.6
privload_locate: looking for /lib/libc.so.6
privload_locate: looking for /usr/local/lib/libc.so.6
privload_locate: looking for /lib64/tls/i686/cmov/libc.so.6
privload_locate: looking for /usr/lib64/libc.so.6
privload_load: loading /usr/lib64/libc.so.6
privload_locate: looking for /lustre06/project/6005345/mewais/DCSim/Workspace/SingleNode_Local128GB/GBBS/../../../Release/libstdc++.so.6
privload_locate: looking for /lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/ext/lib64/debug/libstdc++.so.6
privload_locate: looking for /lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/drmemory/drmf/lib64/debug/libstdc++.so.6
privload_locate: looking for ./libstdc++.so.6
privload_search_rpath: added search dir "/cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib64"
privload_search_rpath: looking for /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib64/libstdc++.so.6
privload_search_rpath: looking for /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib/libstdc++.so.6
privload_search_rpath: added search dir "/cvmfs/soft.computecanada.ca/gentoo/2020/usr/../lib64"
privload_search_rpath: looking for /cvmfs/soft.computecanada.ca/gentoo/2020/usr/../lib64/libstdc++.so.6
privload_search_rpath: looking for /cvmfs/soft.computecanada.ca/gentoo/2020/usr/../lib/libstdc++.so.6
privload_search_rpath: added search dir "/cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0/"
privload_search_rpath: looking for /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0//libstdc++.so.6
privload_search_rpath: added search dir "/cvmfs/soft.computecanada.ca/gentoo/2020/usr/x86_64-pc-linux-gnu/lib/"
privload_load: loading /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0//libstdc++.so.6
privload_locate: looking for /lustre06/project/6005345/mewais/DCSim/Workspace/SingleNode_Local128GB/GBBS/../../../Release/libm.so.6
privload_locate: looking for /lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/ext/lib64/debug/libm.so.6
privload_locate: looking for /lustre06/project/6005345/mewais/DynamoRIO-Linux-8.0.0-1/drmemory/drmf/lib64/debug/libm.so.6
privload_locate: looking for /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib64/libm.so.6
privload_locate: looking for /cvmfs/soft.computecanada.ca/gentoo/2020/usr/../lib64/libm.so.6
privload_load: loading /cvmfs/soft.computecanada.ca/gentoo/2020/usr/../lib64/libm.so.6
- libstdc++ is correctly using rpath, also adds the paths to the search list and finds the library in the correct place
- libm seems to find the library in the correct place based on the paths added by libstdc++
I still don't understand why libc doesn't use my rpath? And I am still unable to follow this https://dynamorio.org/using.html#sec_rpath (either I misunderstand what the steps are, or something else is wrong, error shown in my previous message) and I can't find any examples of it online.
I still don't understand why libc doesn't use my rpath? And I am still unable to follow this https://dynamorio.org/using.html#sec_rpath (either I misunderstand what the steps are, or something else is wrong, error shown in my previous message) and I can't find any examples of it online.
Mohammad Ewais
Feb 6, 2025, 10:44:06 AMFeb 6
to DynamoRIO Users
I finally managed to get it working. I think the issue was that modern gcc uses RUNPATH instead of RPATH. That shouldn't really be a problem, but for some reason that I still don't get libc didn't respect it. I tried forcing gcc to use RPATH instead, by adding -Wl,--disable-new-dtags to my link options. Now all libraries are loaded correctly by the DR loader.
Derek Bruening
Feb 6, 2025, 12:50:54 PMFeb 6
to Mohammad Ewais, DynamoRIO Users
It would be good to help future users if you could file an issue on the "file RELATIVE_PATH must be passed a full path to the directory" error you hit, and ideally look into and fix that. Not really sure we want to recommend any of these custom link flags you are passing: better to get the cmake route ironed out and avoid toolchain-specific custom flags on the user side.
To view this discussion visit https://groups.google.com/d/msgid/dynamorio-users/2184b8ef-ffa7-441b-b2cf-49d7840f8e5bn%40googlegroups.com.
Mohammad Ewais
Feb 7, 2025, 11:08:38 AMFeb 7
to DynamoRIO Users
I am not that proficient with CMake and pretty much have to google my way around things. But I will give it a try and see if I can fix it.
Reply all
Reply to author
Forward
0 new messages