Issues with drrun running simple programs on windows

[Alex Brown]

Greetings,

I am having an issue with DynamoRIO on windows, any help will be much appreciated.

Originally, I was trying to get WinAFL working on windows 10. Down the rabbit hole I ended up here, as a dry run without any client cannot be competed without an error.

I tried the the latest version, stable version and previous stable version with no luck.
I compiled a debug build from source, with the same results.

I tried the following test cases:

drrun.exe -debug -loglevel 4 -- ipconfig
drrun.exe -debug -loglevel 4 -- hello.exe
drrun.exe -debug -loglevel 4 -- hello_c.exe

- The ipconfig results in the same error I always get with almost any program.
- The hello.exe is a simple hello world program written in asm. - this one gets some weird OOM error
- The hello_c.exe is a simple hello world program written in C - tihs one actually "works" with some "curiosities"

I am attaching all the logs and sources for hello and hello_c.
I also opened an issue on github here: https://github.com/DynamoRIO/dynamorio/issues/6763 , but was directed here instead.

My windows is actually a fully-accelerated VM, nevertheless, please do not let the fact that the environment is virtualized discourage you.

- Host OS: mac os sonoma 14.4.1
- Guest OS: Windows 10.0.19045 Build 19045
- Hypervisor: QEMU 8.2.1

Now, I am aware this may be not easily reproducible, still I would love to get to the bottom of this. Therefore, I will you provide you with any more information you will need, we can even schedule an online debugging session.

---

I could not attach the logs directly here, as one test case was too big. You can find all the outputs and logs here: https://drive.google.com/file/d/16l9sXrYUADJyimfLw88lbRi4oa7CMVJI/view?usp=sharing (password  is "dynamorio", sorry for the inconvenience! this was the only way to share files via google drive)

[Derek Bruening]

A non-virtualized 22H2 build 19045.4046 works fine so I can't reproduce it on the closest machine I have available.  Mine has the same 2 CURIOSITY and Cleaning messages in debug build.
The first step would be to attach or run under windbg and get a callstack of the dispatch assert as it is not clear how that is firing.

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/f1aeaef9-a9e8-4c89-b00a-16c6d968dbacn%40googlegroups.com.

[Alex Brown]

Here is the stacktrace I got:

 # Child-SP          RetAddr               Call Site
00 00000015`e0b1eec0 00007ff8`9ba63bba     ntdll!LdrpDoDebuggerBreak+0x30
01 00000015`e0b1ef00 00007ff8`9ba04dbb     ntdll!LdrpInitializeProcess+0x1eda
02 00000015`e0b1f320 00007ff8`9ba04c43     ntdll!LdrpInitialize+0x15f
03 00000015`e0b1f3c0 00007ff8`9ba04bee     ntdll!LdrpInitialize+0x3b
04 00000015`e0b1f3f0 00000000`00000000     ntdll!LdrInitializeThunk+0xe

Is that correct? I expected more "traditional" stacktrace, but I guess there's some kind of runtime patching.

[Derek Bruening]

No, that is probably not the right thread: is that the windbg-inserted thread?  We want the stacktrace of the assert: with dynamorio.dll frames.  See also the debugging docs such as for loading the dynamorio.dll symbols: https://dynamorio.org/page_debugging.html#autotoc_md156

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/c7ecdd0d-5b79-400b-bdc3-abdb484eedc1n%40googlegroups.com.

[Alex Brown]

Well, hopefully this one is it:

# Child-SP RetAddr Call Site 00 0000016a`fc813e98 00000000`153c799f ntdll!NtRaiseHardError+0x14 01 0000016a`fc813ea0 00000000`1537bceb dynamorio!nt_messagebox(wchar_t * msg = 0x00000000`155c4ff0 "Application C:\Windows\system32\ipconfig.exe (3916). Internal Error: DynamoRIO debug check failure: C:\tools\src\dynamorio\core\dispatch.c:793 dc == NULL || OWN_NO_LOCKS(dc).(Error occurred @0 frags in tid 8).version 10.0.19818, custom build.-no_dynamic_options -code_api -probe_api -stack_size 56K -max_elide_jmp 0 -max_elide_call 0 -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct .c:\tools\src\dynamorio\build/lib64\debug\dynamorio.dll=0x0000000015000000", wchar_t * title = 0x00000000`1558a100 "DynamoRIO Notice: C:\Windows\system32\ipconfig.exe(3916)")+0x17f [C:\tools\src\dynamorio\core\win32\ntdll.c @ 3844] 02 0000016a`fc813f40 00000000`151056af dynamorio!debugbox(char * msg = 0x0000016a`fc813fc0 "Application C:\Windows\system32\ipconfig.exe (3916). Internal Error: DynamoRIO debug check failure: C:\tools\src\dynamorio\core\dispatch.c:793 dc == NULL || OWN_NO_LOCKS(dc).(Error occurred @0 frags in tid 8).version 10.0.19818, custom build.-no_dynamic_options -code_api -probe_api -stack_size 56K -max_elide_jmp 0 -max_elide_call 0 -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct .c:\tools\src\dynamorio\build/lib64\debug\dynamorio.dll=0x0000000015000000")+0x5b [C:\tools\src\dynamorio\core\win32\os.c @ 5399] 03 0000016a`fc813f70 00000000`15104eb4 dynamorio!d_r_notify(syslog_event_type_t priority = SYSLOG_ERROR (0n4), char internal = 0n0 '', char synch = 0n0 '', unsigned int message_id = 0xc0ff03f0, unsigned int substitution_num = 3, char * prefix = 0x00000000`15439178 "SYSLOG_ERROR", char * fmt = 0x00000000`1546e2c8 "Application %s (%s). Internal Error: %s")+0x27f [C:\tools\src\dynamorio\core\utils.c @ 1942] 04 0000016a`fc8149d0 00000000`150ffab7 dynamorio!report_dynamorio_problem(struct _dcontext_t * dcontext = 0x0000016a`fc7f5200, unsigned int dumpcore_flag = 8, unsigned char * exception_addr = 0x00000000`00000000 "", unsigned char * report_ebp = 0x00000000`00000104 "--- memory read error at address 0x00000000`00000104 ---", char * fmt = 0x00000000`1546c9a0 "DynamoRIO debug check failure: %s:%d %s.(Error occurred @%d frags in tid %I64d)")+0x904 [C:\tools\src\dynamorio\core\utils.c @ 2234] 05 0000016a`fc814b40 00000000`150dcaaf dynamorio!d_r_internal_error(char * file = 0x00000000`15467d98 "C:\tools\src\dynamorio\core\dispatch.c", int line = 0n793, char * expr = 0x00000000`15453260 "dc == NULL || OWN_NO_LOCKS(dc)")+0x1a7 [C:\tools\src\dynamorio\core\utils.c @ 193] 06 0000016a`fc814cc0 00000000`150da669 dynamorio!dispatch_enter_dynamorio(struct _dcontext_t * dcontext = 0x0000016a`fc7f5200)+0x19f [C:\tools\src\dynamorio\core\dispatch.c @ 793] 07 0000016a`fc814e80 00000000`154264b2 dynamorio!d_r_dispatch(struct _dcontext_t * dcontext = 0x0000016a`fc7f5200)+0x19 [C:\tools\src\dynamorio\core\dispatch.c @ 161] 08 0000016a`fc814fe0 0000016a`fc7f5200 dynamorio!call_switch_stack(void)+0x48 [C:\tools\src\dynamorio\build\core\x86.asm_core.s @ 1938] 09 0000016a`fc814fe8 abababab`abababab 0x0000016a`fc7f5200 0a 0000016a`fc814ff0 00000000`00000001 0xabababab`abababab 0b 0000016a`fc814ff8 00000024`af8bf2b0 0x1 0c 0000016a`fc815000 00000000`00000000 0x00000024`af8bf2b0

[Derek Bruening]

Unfortunately there are multiple paths through call_switch_stack.  The logs may help but it looks like there is a password on the logs you shared in your first email.

Other questions that will help:

• What lock is held here (triggering the assert)?
• Can you walk back through the call_switch_stack and get the callstack prior to that: might not be simple; have to find the prior stack pointer and pass it to windbg for a custom callstack.
• Where is this during setup: if can't get pre-switch callstack, what is going on in at the end of the logs.

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/ee885ad4-c0b9-4bc6-8e90-f6162da97868n%40googlegroups.com.

[Alex Brown]

- How can I find the lock? Should traversing the dcontext help, or do I need to trace the lock call itself?

- The password is "dynamorio", without that password, google disallowed me to share it with world.

- I am going to try to get the previous stacktrace, then.

[Derek Bruening]

The logs tell us to look at an earlier point:

logs/test_case_ipconfig/ipconfig.exe.6712.00000000/ipconfig.exe.0.836.html

---------------------------------------------------------------------------

os_list_threads: thread 0 handle=0x0000000000000104

os_list_threads: thread 1 handle=0x0000000000000108

os_list_threads: thread 2 handle=0x000000000000010c

os_list_threads: thread 3 handle=0x0000000000000110

TAKEOVER: iteration 0

TAKEOVER: taking over thread 5572

SYSLOG_ERROR: Application C:\Windows\system32\ipconfig.exe (6712).  Internal Error: DynamoRIO debug check failure: C:\tools\src\dynamorio\core\dispatch.c:793 dc == NULL || OWN_NO_LOCKS(dc)

(Error occurred @0 frags in tid 836)

---------------------------------------------------------------------------

3 extra threads: not ideal but that seems to happen on recent Windows.

logs/test_case_ipconfig/ipconfig.exe.6712.00000000/log.0.836.html

---------------------------------------------------------------------------

ASYNCH intercepted exception in thread 836 at pc 0x0000000000000000

        exception code = 0x00000218c0000005, ExceptionFlags=0x0000000000000000

        record=0x0000000000000000, params=2

        PC 0x0000000000000000 tried to execute address 0x0000000000000000

<...>

Exception was generated by call to RaiseException

asynch_take_over 0x1558d1de

<...>

transfer_to_dispatch: pc=0x1558d1de, xsp=0x0000002a4395eff0, on-initstack=0

SYSLOG_ERROR: Application C:\Windows\system32\ipconfig.exe (6712).  Internal Error: DynamoRIO debug check failure: C:\tools\src\dynamorio\core\dispatch.c:793 dc == NULL || OWN_NO_LOCKS(dc)

(Error occurred @0 frags in tid 836)

---------------------------------------------------------------------------

Here we go: the lock assert is on a path trying to deal with an earlier exception.  I would forget about the lock assert and get this exception in the debugger -- well, we want to know what the prior branch was, may have to step from last known good point (should be a mode to turn on LBR...)

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/df2e5633-3c01-4b37-b293-d689a1689e57n%40googlegroups.com.

[Alex Brown]

Ok, hope I got the it right here:

WARNING: Stack pointer is outside the normal stack bounds. Stack unwinding can be inaccurate. # Child-SP RetAddr Call Site 00 00000213`e6e63f78 00000000`153d2897 0x0 01 00000213`e6e63f80 00000000`15382d5c dynamorio!nt_get_context_size(unsigned long flags = 0x10000b)+0x17 [C:\tools\src\dynamorio\core\win32\ntdll.c @ 5405] 02 00000213`e6e63fc0 00000000`153755b6 dynamorio!os_take_over_exit(void)+0x4c [C:\tools\src\dynamorio\core\win32\os.c @ 2097] 03 00000213`e6e64040 00000000`150166e6 dynamorio!os_fast_exit(void)+0xd6 [C:\tools\src\dynamorio\core\win32\os.c @ 1240] 04 00000213`e6e64090 00000000`15025c31 dynamorio!dynamo_shared_exit(struct _thread_record_t * toexit = 0x00000000`00000000, char detach_stacked_callbacks = 0n0 '')+0xb36 [C:\tools\src\dynamorio\core\dynamo.c @ 1169] 05 00000213`e6e64190 00000000`150157f4 dynamorio!dynamo_process_exit_cleanup(void)+0x1b1 [C:\tools\src\dynamorio\core\dynamo.c @ 1394] 06 00000213`e6e641e0 00000000`154266f0 dynamorio!dynamo_process_exit(void)+0x184 [C:\tools\src\dynamorio\core\dynamo.c @ 1453] 07 00000213`e6e64250 00000213`e6e45200 dynamorio!cleanup_and_terminate(void)+0x78 [C:\tools\src\dynamorio\build\core\x86.asm_core.s @ 2321] 08 00000213`e6e64258 00000000`154266d6 0x00000213`e6e45200 09 00000213`e6e64260 00000213`e6e45200 dynamorio!cleanup_and_terminate(void)+0x5e [C:\tools\src\dynamorio\build\core\x86.asm_core.s @ 2309] 0a 00000213`e6e64268 00000000`00000000 0x00000213`e6e45200

[Derek Bruening]

The process exit doesn't look right (isn't this all during init?) but ntdll_RtlGetExtendedContextLength does look like a problem, initialized under YMM_ENABLED but used outside.  Probably your VM does not have it enabled.  Looks like a real bug.  You've already filed one so that can track it.  If you'd like to submit a PR as you are able to test it locally that would be great.  Presumably those Rtl routines are still there and still work: is that YMM_ENABLED conditional needed?

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/d0bb1a79-eb88-4709-9937-acaf7db2fa5fn%40googlegroups.com.

[Alex Brown]

I am basically struggling with loading symbols on-the-fly, so I used the -msgbox_type 15 switch and created a coredump, when opened, symbols were there.

What exactly should I do with YMM_ENABLED? - I have no idea what that is, I found some #define the codebase.

Also, could we also test the hello in assembler i bundled with the logs? You don't have to use the bundled exe, I made acomment in the header of hello.asm of how to build it yourself. Without drrun it runs fine, with drrun it throws some weird OOM error.

[Alex Brown]

Well,

I figured that YMM are registers associated with some later generation of SSE instruction set.

I checked the VM settings, adn the cpu was not set, i.e. was default. I set it to qemu64, and made sure all sse features are enabled.
I also created another build, just in case you use any compile-time check for cpu features.

Now it seems like process crashed, with a stacktrace similar to the previous:

# Child-SP RetAddr Call Site 00 0000006a`57f0f718 00000000`153d2897 0x0 01 0000006a`57f0f720 00000000`15384b62 dynamorio!nt_get_context_size(unsigned long flags = 0x10000b)+0x17 [C:\tools\src\dynamorio\core\win32\ntdll.c @ 5405] 02 0000006a`57f0f760 00000000`15375b8b dynamorio!os_take_over_thread(struct _dcontext_t * dcontext = 0x0000022f`15b95200, void * hthread = 0x00000000`00000100, unsigned int64 tid = 0x1ab0, char suspended = 0n0 '')+0x72 [C:\tools\src\dynamorio\core\win32\os.c @ 2512] 03 0000006a`57f0f7e0 00000000`15015980 dynamorio!os_take_over_all_unknown_threads(struct _dcontext_t * dcontext = 0x0000022f`15b95200)+0x26b [C:\tools\src\dynamorio\core\win32\os.c @ 2728] 04 0000006a`57f0f880 00000000`1534e7a0 dynamorio!dynamorio_take_over_threads(struct _dcontext_t * dcontext = 0x0000022f`15b95200)+0x170 [C:\tools\src\dynamorio\core\dynamo.c @ 2925] 05 0000006a`57f0f930 00000000`15026d10 dynamorio!dynamo_start(struct _priv_mcontext_t * mc = 0x0000006a`57f0faa0)+0xd0 [C:\tools\src\dynamorio\core\arch\x86_code.c @ 112] 06 0000006a`57f0f9d0 00000000`15027115 dynamorio!dynamorio_app_take_over_helper(struct _priv_mcontext_t * mc = 0x0000006a`57f0faa0)+0x300 [C:\tools\src\dynamorio\core\dynamo.c @ 2999] 07 0000006a`57f0fa30 00000000`15426e49 dynamorio!dynamorio_earliest_init_takeover_C(unsigned char * arg_ptr = 0x0000022f`15811000 "", struct _priv_mcontext_t * mc = 0x0000006a`57f0faa0)+0x135 [C:\tools\src\dynamorio\core\dynamo.c @ 3068] 08 0000006a`57f0fa80 0000022f`15811000 dynamorio!dynamorio_earliest_init_takeover(void)+0x83 [C:\tools\src\dynamorio\build\core\x86.asm_core.s @ 4877] 09 0000006a`57f0fa88 0000006a`57f0faa0 0x0000022f`15811000 0a 0000006a`57f0fa90 00000000`00000000 0x0000006a`57f0faa0

[Derek Bruening]

That callstack looks right.  I would paste that into the bug and update the title.  As for your updating the hardware and still seeing this: the OS has to enable the support too and might need a reinstall; certainly DR thinks the OS doesn't support YMM still.

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/f9b66206-2e3f-48ca-bb21-c81ab2ee2d7dn%40googlegroups.com.

[Alex Brown]

I downloaded some windows update and it updates the cpu features and seems to work fine now.

Thank You!

Also,  how about the assembler hello world program? It still ends in the OOM error. Did you try running it? I know it's probably a separate issue but it would be worth at least reporting it.

[Prasun]

I'm also unable to run any programs on some Windows machines. There is an immediate failure with the following message (also discussed here https://github.com/DynamoRIO/drmemory/issues/2447)

Out of memory.  Program aborted.  Source I, type 0x0000000000000001, code 0x00000000c000001c

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/CAO1ikSa-kAJr%3DNk8GLOhMAKnC6v%2Be8Hjgtct_xM%3D3NiAaAPL7A%40mail.gmail.com.