Debugging of application running under DynamoRIO

[Дмитрий Жерегеля]

Hi!

I'm interested in debugging an application running under DynamoRIO.

I've searched across the web and in https://dynamorio.org/page_debugging.html, and looks like the most manuals are focused on debugging the clients, rather than target application itself.

Is there any convenient way to debug (especially in GDB) such application? Can I add target application debug symbols into the debugging session somehow? Is it possible to add such symbols with correct address mapping?

Thanks for your help! 

[sharma...@google.com]

Hi,

When an application runs under DynamoRIO, all execution happens from DR's code cache instead of the app executable. As far as I know, there's no way to set breakpoints in the application code when it is run under DR.

Can you give more details about your use case? I understand that you are trying to debug the application itself. And for some reason you need to run it under DR at the same time? Is it because the app bug is not reproducible without running under DR?

Abhinav

[Дмитрий Жерегеля]

Thanks for your answer!

My setup is quite tricky: I'm using DynamoRIO with custom client in order to reproduce application bug. And the bug is also reproducible without DynamoRIO too, but it is harder to do so.

And I'm recording such execution under DynamoRIO with RR - https://github.com/rr-debugger/rr - in order to debug it. I want to understand, is it possible to debug such RR trace conveniently or not.

вторник, 23 ноября 2021 г. в 19:45:41 UTC+3, sharma...@google.com:

[Abhinav Sharma]

Hi,

I see from rr-project.org that it enhances gdb by allowing reversing execution from a breakpoint/watchpoint. I'm not familiar with rr-debugger, or what debugging interface it provides, but I suppose you're interested in setting breakpoint/watchpoint for using that feature on a program run under DR.

As I mentioned, application symbols are not really usable with gdb while running under DR. There are some ideas that come to mind that you can try. They are not very convenient though.

- For a function that you want to set a breakpoint on, use drwrap_wrap_ex (https://dynamorio.org/page_drwrap.html) in your client to wrap it and set pre- and post-function callbacks. You can then set gdb breakpoints on these callbacks.

- You can try setting breakpoints/watchpoints on the code cache PCs.

If you run DR with the following options, it'll print out all fragments and their code cache PCs as they are created:

$ gdb --args ./bin64/drrun -loglevel 4 -logmask 0x60  -log_to_stderr -- <app>

(gdb) break exit_interp_build_bb

(gdb) run

Look for something like the following printed before each exit_interp_build_bb:

```

Fragment 1, tag 0x0000000000401000, flags 0x1030, private, size 88:

  0x0000000042735008  48 c7 c0 01 00 00 00 mov    $0x0000000000000001 -> %rax

  0x000000004273500f  48 c7 c7 01 00 00 00 mov    $0x0000000000000001 -> %rdi

  0x0000000042735016  48 c7 c6 00 20 40 00 mov    $0x0000000000402000 -> %rsi

  0x000000004273501d  48 c7 c2 06 00 00 00 mov    $0x0000000000000006 -> %rdx

  0x0000000042735024  eb 05                jmp    $0x000000004273502b

  0x0000000042735026  e9 07 00 00 00       jmp    $0x0000000042735032 <exit stub 0> 

  0x000000004273502b  0f 05                syscall  -> %rcx

  0x000000004273502d  e9 17 00 00 00       jmp    $0x0000000042735049 <exit stub 1> 

  -------- exit stub 0: -------- <target: 0x000000000040101c> type: fall-through/speculated/IAT

  0x0000000042735032  67 65 48 a3 00 00 00 addr32 mov    %rax -> %gs:0x00[8byte]

                      00

  0x000000004273503a  48 b8 a0 fd c2 f7 fd mov    $0x00007ffdf7c2fda0 -> %rax

                      7f 00 00

  0x0000000042735044  e9 f7 ce fc ff       jmp    $0x0000000042701f40 <fcache_return> 

  -------- exit stub 1: -------- <target: 0x000000000040101e> type: fall-through/speculated/IAT

  0x0000000042735049  67 65 48 a3 00 00 00 addr32 mov    %rax -> %gs:0x00[8byte]

                      00

  0x0000000042735051  48 b8 c0 fd c2 f7 fd mov    $0x00007ffdf7c2fdc0 -> %rax

                      7f 00 00

  0x000000004273505b  e9 e0 ce fc ff       jmp    $0x0000000042701f40 <fcache_return> 

```

Then you can add a breakpoint on one of these PCs:

(gdb) break *0x4a81b008

This may not be very helpful on large programs though.

- You may be able to get the address of some symbols using drsym_lookup_symbol in drsysms (https://dynamorio.org/page_drsyms.html), and look for those addresses in your client.

+Derek in case you have more helpful ideas.

Abhinav

--
You received this message because you are subscribed to a topic in the Google Groups "DynamoRIO Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dynamorio-users/NsL6AazyaZs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dynamorio-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/f71fdfd7-a4d9-45c5-9085-bbe35db84db5n%40googlegroups.com.

[Derek Bruening]

Our docs https://dynamorio.org/page_debugging.html#autotoc_md137 recommend using read watchpoints for the very first execution of an application address:

"Use read watchpoints instead of breakpoints in application code, as the trap instruction inserted by the debugger into the application code can end up copied into DynamoRIO's code cache, resulting in an unhandled trap."

The watchpoint will trigger on DR decoding the app code.  From there you can break on enter_fcache and disassemble the code cache (or look at the logs as Abhinav suggested).

We would love to have automated debugging support where a layer would translate an application address into a code cache address, but no one has had resources to spend enough time on that.

Xref https://github.com/DynamoRIO/dynamorio/issues/532, https://drmemory.org/page_projects.html#autotoc_md35, and the "project-drdbg" branch: https://github.com/DynamoRIO/dynamorio/tree/project-drdbg

[Дмитрий Жерегеля]

Abhinav,  Derek, thanks for the great explanation and provided ideas!

I have one more question to ask: how can I modify DynamoRIO code in a such a way that the message like

"Paste into GDB to debug DynamoRIO clients:

add-symbol-file '/home/user/dr/lib64/debug/libdynamorio.so' 0x00007f152865c000"

will be printed at every execution, even without using debug build? For my use case I can't access gdb_priv_cmds and want to have such message constantly. I've tried to modify privload_add_gdb_cmd() but haven't succeed.

суббота, 27 ноября 2021 г. в 00:00:16 UTC+3, Derek Bruening:

[Derek Bruening]

SYSLOG_INTERNAL_INFO is debug-only.  A local tweak would be print_file(STDERR.

The best solution would be revive the tools/libdynamorio.so-gdb.py script and have everything automatically loaded for you: https://github.com/DynamoRIO/dynamorio/issues/2100

You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/1fdf09f5-6ff0-4cc5-ac8a-bfdcb7786ae5n%40googlegroups.com.

[Дмитрий Жерегеля]

Thanks a lot! You've been very helpful:)

понедельник, 6 декабря 2021 г. в 18:43:08 UTC+3, Derek Bruening: