drmemory: The bcrypt.dll desaster with Windows 10/11 Workstation, Pro, Enterprise

[Cedric Blancher]

Good afternoon!

Derek&co, could you please look at the bug tracker, and review all issues with bcrypt.dll

So far NONE of our Windows 10 Pro, Windows 10 Workstation, Windows 11 Pro and Windows 11 Workstation machines, VMware and raw metal based, can run drmemory. Tested with drmemory 2.6.0 and cronbuild from march.

They ALL fail with this message:

Dr. Memory failed to start the target application, perhaps due to interference from invasive security software. Try disabling other software or running in a virtual machine. WARNING: Examine the following unusual libraries in this process to help identify invasive software that may have affected the target application: C:\Windows\System32\bcrypt.dll

For example, one system version is "Microsoft Windows [Version 10.0.19045.4894]" per "ver" cmd output on VMware 17.5.2 build-23775571/Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz.

The only common denominator is that they are not "normal" consumer Windows installations, these are Workstation and Enterprise installations. Turning Windows Defender and Firewall OFF has no effect.

Installing the same hardware with consumer Windows versions seems to fix the problems, but IT dep. very clear they are not going to tolerate that.

Ced

[Derek]

Reproducing is difficult for me if it does not occur on consumer Windows.  The bcrypt message may be a red herring but it is likely the private loader that would cause a startup problem.
Does plain DynamoRIO run: dynamorio/bin64/drrun -- <application and args>?  If so that is more evidence it is the private loader.
If you start up under a debugger likely there is a crash: what is the callstack?

[Derek]

Does cronbuild-2.6.19989 fare better?

[Roland Mainz]

On Tuesday, September 24, 2024 at 5:28:49 AM UTC+2 Derek wrote:

Does cronbuild-2.6.19989 fare better?

I tried this, but build 19989 still fails for "calc.exe" on Windows 10/Workstation AMD64 on VMware 17.5.2 build-23775571:

---- snip ----

$ drmemory -version
Dr. Memory version 2.6.19989 -- build 0

$ cmd.exe /C 'ver'
Microsoft Windows [Version 10.0.19045.4957]

$ drmemory -debug -- "$(cygpath -w /cygdrive/c/Windows/system32/calc)"
~~Dr.M~~ WARNING: unable to locate results file: can't open C:\Users\roland_mainz\AppData\Roaming\Dr. Memory/resfile.3116 (code=2).

Dr. Memory failed to start the target application, perhaps due to
interference from invasive security software.
Try disabling other software or running in a virtual machine.
WARNING: Examine the following unusual libraries in this process to help identify
invasive software that may have affected the target application:

        C:\Windows\System32\bcrypt.dll

Please file a bug about this at http://drmemory.org/issues
~~Dr.M~~ WARNING: application exited with abnormal code 0xffffffff

---- snip ----

Also the resfile warning has the last slash wrong, on Windows it should be a backslash ("Dr. Memory/resfile.3116" vs "Dr. Memory\resfile.3116")

Interestingly drmemory works if I have a binary like nfs_mount.exe (e.g. $ drmemory -debug -ignore_asserts -ignore_kernel -- "$(cygpath -w /home/roland_mainz/work/msnfs41_uidmapping/ms-nfs41-client/destdir/cygdrive/c/cygwin64/sbin/nfs_mount)" #) from https://github.com/kofemann/ms-nfs41-client ... but that binary does not use bcrypt.dll (or maybe bcrypt.dll is not the problem ?!) ...

----

Bye,

Roland

[Roberto Villalobos]

Hi, dealing with the same issue on Windows 11 Enterprise inside VMWare Fusion. Curiously enough, if I run something like "drmemory -- notepad" I get not only the bcrypt issue but several others

~~Dr.M~~ WARNING: unable to locate results file: can't open C:\Users\user\AppData\Roaming\Dr. Memory/resfile.13952 (code=2). Dr. Memory failed to start the target application, perhaps due to interference from invasive security software. Try disabling other software or running in a virtual machine. WARNING: Examine the following unusual libraries in this process to help identify invasive software that may have affected the target application: C:\Windows\SYSTEM32\daxexec.dll C:\Windows\SYSTEM32\capauthz.dll C:\Windows\SYSTEM32\AppXAllUserStore.dll C:\Windows\SYSTEM32\container.dll C:\Windows\SYSTEM32\windows.staterepositorycore.dll C:\Windows\SYSTEM32\apisethost.appexecutionalias.dll C:\Windows\SYSTEM32\USERENV.dll C:\Windows\SYSTEM32\windows.staterepositoryclient.dll C:\Windows\System32\Windows.StateRepositoryPS.dll C:\Windows\SYSTEM32\kernel.appcore.dll C:\Windows\SYSTEM32\wintypes.dll C:\Windows\SYSTEM32\windows.storage.dll C:\Windows\SYSTEM32\profapi.dll C:\Windows\SYSTEM32\ntmarta.dll C:\Windows\System32\bcrypt.dll C:\Windows\System32\OLEAUT32.dll C:\Windows\System32\SHCORE.dll C:\Windows\System32\clbcatq.dll C:\Windows\System32\msvcp_win.dll C:\Windows\System32\shlwapi.dll C:\Windows\System32\combase.dll Please file a bug about this at http://drmemory.org/issues ~~Dr.M~~ WARNING: application exited with abnormal code 0xc0000005

[Derek]

I think I have reproduced the problem, or at least a problem, in a Windows Enterprise environment.  Plain DynamoRIO fails, not just Dr. Memory: DynamoRIO fails very early trying to allocate its code cache memory.  It looks exactly like https://github.com/DynamoRIO/drmemory/issues/2447 with STATUS_INVALID_SYSTEM_SERVICE (0xc000001c) returned by VirtualAlloc.  Have not yet installed enough to experiment further but this is requesting PAGE_EXECUTE_READWRITE so one theory is some kind of "DEP" where +x heap is disallowed in general??

[Cedric Blancher]

On Thu, 3 Oct 2024 at 03:04, Derek <derek.b...@gmail.com> wrote:
>
> I think I have reproduced the problem, or at least a problem, in a Windows Enterprise environment. Plain DynamoRIO fails, not just Dr. Memory: DynamoRIO fails very early trying to allocate its code cache memory. It looks exactly like https://github.com/DynamoRIO/drmemory/issues/2447 with STATUS_INVALID_SYSTEM_SERVICE (0xc000001c) returned by VirtualAlloc. Have not yet installed enough to experiment further but this is requesting PAGE_EXECUTE_READWRITE so one theory is some kind of "DEP" where +x heap is disallowed in general??

Try this as system Admin, REBOOT and then test again
bcdedit.exe /set nx AlwaysOff

Ced

[Cedric Blancher]

Useful for bug reporting:
https://www.shellspells.net/windows/system-ops/system-info/data-execution-prevention

Ced
--
Cedric Blancher <cedric....@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

[Derek]

The DEP was not the culprit.  The issue is that the CrowdStrike Falcon Sensor Module has hooks on key system call wrappers which prevent DynamoRIO from obtaining the numbers.  I filed this as https://github.com/DynamoRIO/dynamorio/issues/7024.  There are additional issues making it hard to get diagnostic info which made this a confusing silent failure: https://github.com/DynamoRIO/dynamorio/issues/7025.

I don't know that CrowdStrike is the culprit for everyone else's failures but I would suggest running a console application like "msg" to get information.  For me it looks like this:

C:\Users\bruening\DrMemory-Windows-2.6.19984\DrMemory-Windows-2.6.19984>bin64\drmemory -- msg
<Application C:\Windows\system32\msg.exe (1692).  Out of memory.  Program aborted.  Source I, type 0x0000000000000001, code 0x00000000c000001c.>
~~Dr.M~~ WARNING: unable to locate results file: can't open C:\Users\bruening\DrMemory-Windows-2.6.19984\DrMemory-Windows-2.6.19984\drmemory\logs/resfile.1692 (code=2).

Dr. Memory failed to start the target application, perhaps due to
interference from invasive security software.
Try disabling other software or running in a virtual machine.
WARNING: Examine the following unusual libraries in this process to help identify
invasive software that may have affected the target application:

        C:\Windows\System32\umppc18613.dll

Please file a bug about this at http://drmemory.org/issues
~~Dr.M~~ WARNING: application exited with abnormal code 0xffffffff

"umppc18613.dll" is CrowdStrike Falcon.

(It used to also print "C:\Windows\System32\bcrypt.dll" but that was added to the known library list in https://github.com/DynamoRIO/drmemory/pull/2516)