<ERROR: using undefined symbol!>

[Mohammad Ewais]

Hi, I was using DynamoRIO 8.0.1 for a couple of years with no issues. On a somewhat old system where libc version was 2.33.

I am now forced to move to a new system, libc version is 2.37, and I am trying to work with the latest DynamoRIO 11.3.0. I don't know if the libc is causing my issues, but I am mentioning it just in case. My understanding after looking at this is that the issues are (at least for now) fixed, and even used with drmemory.

I am getting an obscure `<ERROR: using undefined symbol!>` message at the initialization of my client, before anything runs. Using gdb doesn't help, as it doesn't pause at the error, just exits right away.

I did get the following by using `-debug -loglevel 4`, It seems fine thought the addresses for all DR extensions look different from what I am used to:

<Paste into GDB to debug DynamoRIO clients:
set confirm off
add-symbol-file '/lustre06/project/6005345/mewais/DCSim/Debug/libDCSClient.so' 0x000000007269ec80
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-11.3.0-1/lib64/debug/libdynamorio.so' 0x00001555550f1000
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-11.3.0-1/ext/lib64/debug/libdrwrap.so' 0x00000000740037d0
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-11.3.0-1/ext/lib64/debug/libdrmgr.so' 0x0000000073004cb0
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-11.3.0-1/ext/lib64/debug/libdrutil.so' 0x00000000750026b0
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-11.3.0-1/ext/lib64/debug/libdrx.so' 0x00000000770054d0
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-11.3.0-1/ext/lib64/debug/libdrreg.so' 0x0000000078003df0
add-symbol-file '/lustre06/project/6005345/mewais/DynamoRIO-Linux-11.3.0-1/ext/lib64/debug/libdrsyms.so' 0x0000000076007d60
add-symbol-file '/usr/lib64/libz.so.1' 0x0000155354d4c740
add-symbol-file '/usr/lib64/libc.so.6' 0x0000155354781c40
add-symbol-file '/usr/lib64/ld-linux-x86-64.so.2' 0x000015535452e080
add-symbol-file '/usr/lib64/libpthread.so.0' 0x0000155354312840
add-symbol-file '/usr/lib64/libstdc++.so.6' 0x0000155354005b90
add-symbol-file '/usr/lib64/libm.so.6' 0x0000155353bff520
add-symbol-file '/usr/lib64/libgcc_s.so.1' 0x00001553539dce00
>
<ERROR: using undefined symbol!>
[Inferior 1 (process 400016) exited with code 0377]

I am very sure that I have the correct flags in my CMake (after all, it was working fine before, and I haven't changed it):

target_compile_definitions(DCSClient PUBLIC DCSCLIENT)
target_include_directories(DCSClient PUBLIC Headers)
target_include_directories(DCSClient PUBLIC Third)
target_include_directories(DCSClient PUBLIC .)
target_include_directories(DCSClient PUBLIC ${EXTERNAL_INSTALL_LOCATION}/include)
target_include_directories(DCSClient PUBLIC SYSTEM ${MPI_INCLUDE_PATH})
target_link_directories(DCSClient PUBLIC ${EXTERNAL_INSTALL_LOCATION}/lib)
target_link_directories(DCSClient PUBLIC ${EXTERNAL_INSTALL_LOCATION}/lib64)
target_link_libraries(DCSClient libdocopt.a)
target_link_libraries(DCSClient libconfig++.a)
target_link_libraries(DCSClient libyaml-cpp.a)
target_link_libraries(DCSClient libxed.a)
configure_DynamoRIO_client(DCSClient)
use_DynamoRIO_extension(DCSClient drwrap)
use_DynamoRIO_extension(DCSClient drmgr)
use_DynamoRIO_extension(DCSClient drutil)
use_DynamoRIO_extension(DCSClient drx)
use_DynamoRIO_extension(DCSClient drreg)
use_DynamoRIO_extension(DCSClient drsyms)

The log file isn't much help either, the file ends with a bunch of allocations, followed by the sudden error, no extra info:

Variable-size block: allocating 0x000015535506dd70 (736 bytes [736 aligned] in 784 block)
Variable-size block: allocating 0x000015535506e0b8 (690 bytes [696 aligned] in 1552 block)
Variable-size block: allocating 0x000015535506dd70 (736 bytes [736 aligned] in 784 block)
Variable-size block: allocating 0x000015535506e700 (520 bytes [520 aligned] in 3088 block)
Variable-size block: allocating 0x000015535506dd70 (784 bytes [784 aligned] in 784 block)
SYSLOG_CRITICAL: ERROR: using undefined symbol!

Any guidance would be appreciated.

[Kai Luk]

> # DynamoRIO Extensions
> # Failure to call this in your client CMakeLists.txt when using an extension like
> # drmgr or drx will result in an obscure "using undefined symbol!" runtime
> # error on Linux.

Can you check if other extensions are being used besides what you listed below?

> use_DynamoRIO_extension(DCSClient drwrap)
> use_DynamoRIO_extension(DCSClient drmgr)
> use_DynamoRIO_extension(DCSClient drutil)
> use_DynamoRIO_extension(DCSClient drx)
> use_DynamoRIO_extension(DCSClient drreg)
> use_DynamoRIO_extension(DCSClient drsyms)

Kai

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dynamorio-users/829c1eed-b193-4071-a0a3-c514578a55c0n%40googlegroups.com.

[Mohammad Ewais]

Yes, double checked again now, 100% no more extensions.

This was working (as is, no changes) on the old libc and old DR, if I forgot to add any extensions in the CMake I think it would have broken before too.

[Kai Luk]

Can you check if you can find the undefined symbols based on the following discussions ? They

were posted by Abhinav before:

https://github.com/DynamoRIO/dynamorio/issues/1712

https://groups.google.com/g/DynamoRIO-Users/c/2scmr8ky1dI

[Mohammad Ewais]

I tried `-stderr_mask 15` but it doesn't seem to be showing anything more than what `-debug -loglevel 4` was doing. I did notice something though, I used to get some `symbol lookup errors` before on the old DR, they were for things my client truly didn't use, so they never were an issue. I still get some of them now, but I just realized that they seem to be different, some of them are from libc now:

<WARNING! symbol lookup error: libDCSClient.so undefined symbol __libc_single_threaded>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol stat>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNKSt10filesystem7__cxx114path5_List13_Impl_deleterclEPNS2_5_ImplE>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNSt10filesystem7__cxx114path5_ListC1Ev>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNSt10filesystem6statusERKNS_7__cxx114pathE>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZN10ramulator212DeviceConfig10set_deviceEPNS_15IDRAMControllerE>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZN10ramulator24BHO313get_num_coresEv>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEC1Ev>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZN10ramulator212DeviceConfig16get_flat_bank_idERKNS_7RequestE>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZSt28__throw_bad_array_new_lengthv>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZN10ramulator212DeviceConfigD1Ev>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNSt10filesystem18create_directoriesERKNS_7__cxx114pathE>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol __cxa_call_terminate>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZN10ramulator212DeviceConfigC1Ev>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNSt10filesystem12current_pathERKNS_7__cxx114pathE>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol fstat64>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNSt7__cxx1119basic_istringstreamIcSt11char_traitsIcESaIcEEC1EONS_12basic_stringIcS2_S3_EESt13_Ios_Openmode>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol fstat>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE15_M_replace_coldEPcmPKcmm>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZN10ramulator27BHO3LLC18get_mshrs_per_coreEv>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNSt10filesystem7__cxx114path14_M_split_cmptsEv>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNSt7__cxx1115basic_stringbufIcSt11char_traitsIcESaIcEE3strEONS_12basic_stringIcS2_S3_EE>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZN10ramulator24BHO37get_llcEv>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZN10ramulator27BHO3LLC23set_blacklist_max_mshrsEii>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNKSt10filesystem7__cxx114path11parent_pathEv>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNSt10filesystem12current_pathB5cxx11Ev>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZNKRSt7__cxx1115basic_stringbufIcSt11char_traitsIcESaIcEE3strEv>
<WARNING! symbol lookup error: libDCSClient.so undefined symbol _ZN10ramulator27BHO3LLC13add_blacklistEi>

Using `nm -D` and grepping for all undefined, I see A LOT OF libc AND DR undefined things! A portion of it just as an example:
                 U abort@GLIBC_2.2.5
                 U __assert_fail@GLIBC_2.2.5
                 U atof@GLIBC_2.2.5
                 U atoll@GLIBC_2.2.5
                 U calloc@GLIBC_2.2.5
                 U clearerr@GLIBC_2.2.5
                 U __ctype_b_loc@GLIBC_2.3
                 U __cxa_allocate_exception@CXXABI_1.3
                 U __cxa_atexit@GLIBC_2.2.5
                 U __cxa_begin_catch@CXXABI_1.3
                 U __cxa_call_terminate@CXXABI_1.3.15
                 U __cxa_end_catch@CXXABI_1.3
                 U __cxa_free_exception@CXXABI_1.3
                 U __cxa_guard_abort@CXXABI_1.3
                 U __cxa_guard_acquire@CXXABI_1.3
                 U __cxa_guard_release@CXXABI_1.3
                 U __cxa_rethrow@CXXABI_1.3
                 U __cxa_thread_atexit@CXXABI_1.3.7
                 U __cxa_throw@CXXABI_1.3
                 U disassemble_set_syntax
                 U dr_call_on_clean_stack
                 U dr_client_thread_set_suspendable
                 U dr_close_file
                 U dr_create_client_thread
                 U dr_delay_flush_region
                 U dr_delete_fragment
                 U dr_exit_process
                 U dr_file_seek
                 U dr_file_size
                 U dr_flush_region
                 U dr_fragment_app_pc
                 U dr_free_module_data
                 U dr_get_current_drcontext
                 U dr_get_main_module
                 U dr_get_mcontext
                 U dr_get_proc_address
                 U dr_get_process_id
                 U dr_get_thread_id
                 U dr_global_alloc
                 U dr_global_free
                 U dr_insert_cbr_instrumentation_ex
                 U dr_insert_clean_call
                 U dr_map_file
                 U dr_memory_protect
                 U drmgr_exit
                 U drmgr_get_tls_field
                 U drmgr_init

On the current system there are multiple gccs and libcs, and a module system to pick between them. Might be a similar issue to this one:  Running DR on a system with multiple libc to see if it goes away. I will report back if it doesn't.

Thanks a lot.