Having problems debugging drmemory

[Pastori]

I was trying to work on a specific issue in drmemory and wanted to make sure all the tests pass and the application doesn't crash, then when i tested the drmemory debug build on a regular hello world program, i was surprised that i encounter several SIGSEGV, SIGBUS and SIGILL faults, and although you can step over these, the breakpoints i set specifically in the report_summary (which always seems to run regardless), don't actually hit although the logging is clearly there, i have tried several different ways to mitigate this problem tried different gcc/g++ versions, different glibc versions, different system libraries and images (in docker), and tried to install system libraries with debug symbols and still encounter the same problem, i hope someone can tell me what am i doing wrong. This might have to do with vma coalescing but for now i can only guess. Now i really wanna know is this actually expected or not because i think dr might be handling these signals internally.
details:

os: arch-linux
kernel: 6.12.44-1-lts

cpu architecture: x86_64
docker images: ubuntu (16.04, 20.04, 22.04)

[Derek Bruening]

Dr. Memory is built on DynamoRIO which uses signals for various purposes: https://dynamorio.org/page_debugging.html#autotoc_md142

Application code is not executed from its original location.  https://dynamorio.org/page_debugging.html#autotoc_md139 suggests "Use read watchpoints instead of breakpoints in application code".

(Ideally there would be debugger augmentation to make things easier but that does not exist at this time.)

--

---
You received this message because you are subscribed to the Google Groups "Dr. Memory Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to drmemory-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/drmemory-users/8398dacd-ac2d-4f04-b3b0-e4536fc30476n%40googlegroups.com.

[Pastori]

Thank you for the quick reply and clarification with the links, i guess i can safely ignore the signals, however i am trying to debug the drmemory build in itself, I actually set the breakpoints in the drmemory internal functions not in the application code which in this case is a hello world program (it's just a placeholder), do you mean that I should use watchpoints while debugging drmemory as well ?

To view this discussion visit https://groups.google.com/d/msgid/drmemory-users/CAKFbokehTLwf05%2BVsEwrF4Cc6K%3D7cRLMuH3pmxW-7MFn%3D8hAMg%40mail.gmail.com.

[Derek Bruening]

Breakpoints in the drmemory code itself should fire, if in the drmemory code that runs unmanaged (i.e., not in something like the string replacement routines which run managed in the code cache).

report_summary is hit when I try it.

Did the debugger successfully locate the symbols?

DR prints them out in debug build; there is some python gdb scripting automating their loading but it is bit-rotted.

You can do it manually; something like:

$ gdb --args bin64/drmemory -dr_ops "-msgbox_mask 15" -- tests/hello

GNU gdb (Debian 16.3-1) 16.3

...

(gdb) r

Starting program /yxz/drmemory/git/build_x64_dbg/bin64/drmemory -dr_ops -msgbox_mask\ 15 -- tests/hello

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

WARNING: using debug DynamoRIO since release not found

process 1970162 is executing new program /yxz/drmemory/git/build_x64_dbg/dynamorio/lib64/debug/libdynamorio.so

<Starting application /yxz/drmemory/git/build_x64_dbg/tests/hello (1970162)>

<press enter to continue>

Program received signal SIGILL, Illegal instruction.

syscall_ready () at /yxz/drmemory/git/src/dynamorio/core/drlibc/drlibc_x86.asm:187

187             pop      REG_XBX

(gdb) c

Continuing.

<Initial options = -no_dynamic_options -logdir /yxz/drmemory/git/build_x64_dbg/logs/dynamorio' -client_lib /yxz/drmemory/git/build_x64_dbg/bin64/debug/libdrmemorylib.so;0;-logdir /yxz/drmemory/git/build_x64_dbg/logs` -symcache_dir /yxz/drmemory/git/build_x64_dbg/logs/symcache` -resfile 1970162 ' -client_lib64 /yxz/drmemory/git/build_x64_dbg/bin64/debug/libdrmemorylib.so;0;-logdir /yxz/drmemory/git/build_x64_dbg/logs` -symcache_dir /yxz/drmemory/git/build_x64_dbg/logs/symcache` -resfile 1970162 ' -code_api -msgbox_mask 15 -stack_size 56K -signal_stack_size 32K -disable_traces -no_enable_traces -max_elide_jmp 0 -max_elide_call 0 -no_shared_traces -bb_ibl_targets -bb_single_restore_prefix -no_shared_trace_ibl_routine -no_enable_reset -no_reset_at_switch_to_os_at_vmm_limit -reset_at_vmm_percent_free_limit 0 -no_reset_at_vmm_full -reset_at_commit_free_limit 0B -reset_every_nth_pending 0 -vm_size 256M -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >

<press enter to continue>

<Paste into GDB to debug DynamoRIO clients:

set confirm off

add-symbol-file /yxz/drmemory/git/build_x64_dbg/bin64/debug/libdrmemorylib.so' 0x00007fffe3a18000

add-symbol-file /yxz/drmemory/git/build_x64_dbg/dynamorio/lib64/debug/libdynamorio.so' 0x00007ffff7a4f000

add-symbol-file '/lib/x86_64-linux-gnu/libunwind.so.8' 0x00007ffff7fa0400

add-symbol-file '/lib/x86_64-linux-gnu/libc.so.6' 0x00007ffff7647400

add-symbol-file '/usr/lib64/ld-linux-x86-64.so.2' 0x00007ffff7f65000

add-symbol-file '/lib/x86_64-linux-gnu/liblzma.so.5' 0x00007ffff7f365c0

add-symbol-file '/lib/x86_64-linux-gnu/libz.so.1' 0x00007ffff7f14340

>

<press enter to continue>

^C

Program received signal SIGINT, Interrupt.

syscall_ready () at /yxz/drmemory/git/src/dynamorio/core/drlibc/drlibc_x86.asm:187

187             pop      REG_XBX

(gdb) set confirm off

(gdb) add-symbol-file /yxz/drmemory/git/build_x64_dbg/bin64/debug/libdrmemorylib.so' 0x00007fffe3a18000

add symbol table from file /yxz/drmemory/git/build_x64_dbg/bin64/debug/libdrmemorylib.so" at

        .text_addr = 0x7fffe3a18000

Reading symbols from /yxz/drmemory/git/build_x64_dbg/bin64/debug/libdrmemorylib.so...

(gdb) add-symbol-file /yxz/drmemory/git/build_x64_dbg/dynamorio/lib64/debug/libdynamorio.so' 0x00007ffff7a4f000

add symbol table from file /yxz/drmemory/git/build_x64_dbg/dynamorio/lib64/debug/libdynamorio.so" at

        .text_addr = 0x7ffff7a4f000

Reading symbols from /yxz/drmemory/git/build_x64_dbg/dynamorio/lib64/debug/libdynamorio.so...

Reading symbols from /yxz/drmemory/git/build_x64_dbg/dynamorio/lib64/debug/libdynamorio.so.debug...

(gdb) add-symbol-file '/lib/x86_64-linux-gnu/libunwind.so.8' 0x00007ffff7fa0400

add symbol table from file "/lib/x86_64-linux-gnu/libunwind.so.8" at

        .text_addr = 0x7ffff7fa0400

Reading symbols from /lib/x86_64-linux-gnu/libunwind.so.8...

(No debugging symbols found in /lib/x86_64-linux-gnu/libunwind.so.8)

(gdb) add-symbol-file '/lib/x86_64-linux-gnu/libc.so.6' 0x00007ffff7647400

add symbol table from file "/lib/x86_64-linux-gnu/libc.so.6" at

        .text_addr = 0x7ffff7647400

Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...

Reading symbols from /usr/lib/debug/.build-id/61/e1dea1f540b3b4b4d8ec76716e409cec096ece.debug...

warning: td_ta_new failed: generic error

warning: File "/usr/lib/x86_64-linux-gnu/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".

To enable execution of this file add

        add-auto-load-safe-path /usr/lib/x86_64-linux-gnu/libthread_db.so.1

line to your configuration file /yxz/gdbinit".

To completely disable this security protection add

        set auto-load safe-path /

line to your configuration file /yxz/gdbinit".

For more information about this security protection see the

"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:

        info "(gdb)Auto-loading safe path"

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

(gdb) add-symbol-file '/usr/lib64/ld-linux-x86-64.so.2' 0x00007ffff7f65000

add symbol table from file "/usr/lib64/ld-linux-x86-64.so.2" at

        .text_addr = 0x7ffff7f65000

Reading symbols from /usr/lib64/ld-linux-x86-64.so.2...

Reading symbols from /usr/lib/debug/.build-id/08/cccc2a9388d5247ccb3e864f3063b975b0a15d.debug...

(gdb) add-symbol-file '/lib/x86_64-linux-gnu/liblzma.so.5' 0x00007ffff7f365c0

add symbol table from file "/lib/x86_64-linux-gnu/liblzma.so.5" at

        .text_addr = 0x7ffff7f365c0

Reading symbols from /lib/x86_64-linux-gnu/liblzma.so.5...

(No debugging symbols found in /lib/x86_64-linux-gnu/liblzma.so.5)

(gdb) add-symbol-file '/lib/x86_64-linux-gnu/libz.so.1' 0x00007ffff7f14340

add symbol table from file "/lib/x86_64-linux-gnu/libz.so.1" at

        .text_addr = 0x7ffff7f14340

Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...

(No debugging symbols found in /lib/x86_64-linux-gnu/libz.so.1)

(gdb) b report_summary

Breakpoint 1 at 0x7fffe3b25c16: file /yxz/drmemory/git/src/drmemory/report.c, line 1912.

(gdb) c

Continuing.

~~Dr.M~~ Dr. Memory version 2.6.20133

<curiosity: rex.w on OPSZ_6_irex10_short4!>

<press enter to continue>

<get_memory_info mismatch! (can happen if os combines entries in /proc/pid/maps)

        os says: 0x00007ffff7ef9000-0x00007ffff7f11000 prot=0x00000001

        cache says: 0x00007ffff7ef9000-0x00007ffff7f14000 prot=0x00000001

>

<press enter to continue>

<spurious rep/repne prefix @0x00007ffff7ffe020 (f3 0f 1e fa): >

<press enter to continue>

Hello world!

<Stopping application /yxz/drmemory/git/build_x64_dbg/tests/hello (1970162)>

<press enter to continue>

Breakpoint 1, report_summary () at /yxz/drmemory/git/src/drmemory/report.c:1912

1912        report_summary_to_file(f_global, true, true, false);

(gdb) bt

#0  report_summary () at /yxz/drmemory/git/src/drmemory/report.c:1912

#1  0x00007fffe3b25d4a in report_exit () at /yxz/drmemory/git/src/drmemory/report.c:1926

#2  0x00007fffe3a1aad5 in event_exit () at /yxz/drmemory/git/src/drmemory/drmemory.c:455

#3  0x00007ffff7bfe1ce in instrument_exit_event () at /yxz/drmemory/git/src/dynamorio/core/lib/instrument.c:906

#4  0x00007ffff7a61db1 in dynamo_shared_exit (toexit=0x0) at /yxz/drmemory/git/src/dynamorio/core/dynamo.c:1078

#5  0x00007ffff7a622c7 in dynamo_process_exit_cleanup () at /yxz/drmemory/git/src/dynamorio/core/dynamo.c:1394

#6  0x00007ffff7a62455 in dynamo_process_exit () at /yxz/drmemory/git/src/dynamorio/core/dynamo.c:1449

#7  0x00007ffff7cc42c6 in cat_done_saving_dstack () at /yxz/drmemory/git/src/dynamorio/core/arch/x86/x86.asm:614

#8  0x00007ffff7cc4325 in global_do_syscall_sysenter () at /yxz/drmemory/git/src/dynamorio/core/arch/x86/x86.asm:805

#9  0x00007ffde3a9f000 in ?? ()

#10 0x00007ffde3a84680 in ?? ()

#11 0x00000000000000e7 in ?? ()

#12 0x0000000000000000 in ?? ()

To view this discussion visit https://groups.google.com/d/msgid/drmemory-users/CACi1_SPz3sm8OTv_fb%2BUwBJbSGN9XyvSMT_S%3DvEPDq783iEOPQ%40mail.gmail.com.