Deadlock between COM RPC Threads [LdrpLoaderLock & GlobalMutex]
saravanan.so...@bayer.com
I have a windows COM Server application running on Windows Server
2003. This application has a UI and during startup, before the main
window is created , two RPC threads get into a deadlock.
I created a minidump using windbg and reviewed the call stack of all
the threads in the process.I found that the LoaderLock cs is held by
one RPC Thread, the UI also waits for it. The UI thread is trying to
open an ADO connection to a SQL Server database. I have pasted the call
stack below. The application intializes a STA and then intializes COM
process level Security to nothing by a call to CoInitializeSecurity.
This deadlock occurs only when the windows server is shutdown and
restarted [reboot]. After killing the process and restarting the
application doesn't deadlock. Any suggestions to get to the root cause
of the issue is appreciated.
Thank You
Saravanan
0:000> !locks
CritSec ntdll!LdrpLoaderLock+0 at 77fc2340
LockCount 3
RecursionCount 2
OwningThread 88c
EntryCount 4
ContentionCount 4
*** Locked
CritSec rpcrt4!GlobalMutex+0 at 77ced0d0
LockCount 1
RecursionCount 1
OwningThread 880
EntryCount 1
ContentionCount 1
*** Locked
-----------------------------------------------------------------------------------
t88c acquired the loader lock and waits for the GlobalMutex.
t880 acquired the GlobalMutex and waits for the loader lock.
-----------------------------------------------------------------------------------
-------------------------t88c---------------------------------
77f5d64e 000004cc 00000000 SharedUserData!SystemCallStub+0x4
000004cc 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
000004cc 77c5e329 77ced0d0 ntdll!RtlpWaitForCriticalSection+0x126
77ced0d0 001b74e8 001c7138 ntdll!RtlEnterCriticalSection+0x46
001c71b0 001c70c8 0d53e864 rpcrt4!LoadableTransportInfo+0x19
001c71b0 001c70c8 0d53e864 rpcrt4!OsfMapRpcProtocolSequence+0x25
0d53e8f8 0063006e 00630061 rpcrt4!DCE_BINDING::CreateBindingHandle+0x9b
001c8908 0d53e96c 76620160 rpcrt4!RpcBindingFromStringBindingW+0x99
00000000 765af084 765af098 setupapi!RpcpBindRpc+0x1a4
00000000 00000000 765a0000 setupapi!PNP_HANDLE_bind+0x19
00000000 00000000 0d53e9d8 setupapi!PnPGetGlobalHandles+0xa8
00010156 0d53ec4c 00000000 setupapi!CMP_RegisterNotification+0x97
00010156 0d53ec4c 00000000 user32!RegisterDeviceNotificationWorker+0x3f
WARNING: Stack unwind information not available. Following frames may
be wrong.
77d23e78 73d260a0 73d260ce DLWin32!GetEvent+0xe2cb
73d260a0 00010156 00000110 DLWin32!GetEvent+0xe32e
00000000 73d260a0 00010156 user32!UserCallDlgProcCheckWow+0x147
00000000 00000110 0001015a user32!DefDlgProcWorker+0xa6
00010156 00000110 0001015a user32!DefDlgProcA+0x21
77d0488e 00010156 00000110 user32!InternalCallWinProc+0x1b
00000000 77d0488e 00010156 user32!UserCallWinProcCheckWow+0x151
77d0488e 00010156 00000110 user32!CallWindowProcAorW+0x96
77d0488e 00010156 00000110 user32!CallWindowProcA+0x19
00000110 0001015a 00000000 mfc42!CWnd::DefWindowProcA+0x42
00d84148 00000000 73d21fa4 mfc42!CWnd::Default+0x25
0001015a 00000000 00d84148 mfc42!CDialog::HandleInitDialog+0x54
00000110 0001015a 73dc00f0 mfc42!CWnd::OnWndMsg+0x485
00000110 0001015a 00000000 mfc42!CWnd::WindowProc+0x22
84ca00c4 00010156 00000110 mfc42!AfxCallWndProc+0x91
00010156 00000110 0001015a mfc42!AfxWndProc+0x34
00010156 00000110 0001015a DLWin32!GetEvent+0xfcf2
------------------------------------------------------------------------
-----------------------------t880--------------------------------------
77f5d64e 000000a8 00000000 SharedUserData!SystemCallStub+0x4
000000a8 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
000000a8 77f49beb 77fc2340 ntdll!RtlpWaitForCriticalSection+0x126
77fc2340 00000000 001b8488 ntdll!RtlEnterCriticalSection+0x46
00000001 00000000 0d23fc64 ntdll!LdrLockLoaderLock+0xea
00151390 0d23fcd0 0d23fcb0 ntdll!LdrLoadDll+0x79
77c9c71c 00000000 00000000 kernel32!LoadLibraryExW+0x1af
77c9c71c 001c6408 77c91ca3 kernel32!LoadLibraryW+0xd
77c9c71c 0d23fd20 001c6408 rpcrt4!DLL::DLL+0x25
00000038 21040002 0c01010a
rpcrt4!OSF_SCONNECTION::EnsureSecur32DllLoaded+0x58
001c4db8 0d23fe6c 0d23fe78 rpcrt4!OSF_SCONNECTION::AcceptFirstTime+0x21
0014eb10 0000007d 0014e068
rpcrt4!OSF_SCONNECTION::AssociationRequested+0x385
00000000 0014eb10 0000007d
rpcrt4!OSF_SCONNECTION::ProcessReceiveComplete+0x3bf
0014e068 0000000c 00000000
rpcrt4!ProcessConnectionServerReceivedEvent+0x20
77c650a1 0014e068 00000000
rpcrt4!LOADABLE_TRANSPORT::ProcessIOEvents+0x1b6
0014e068 00000000 00000000 rpcrt4!ProcessIOEventsWrapper+0x9
0014c8b0 77e4a990 001af060 rpcrt4!BaseCachedThreadRoutine+0x9c
001af060 00000000 00000000 rpcrt4!ThreadStartRoutine+0x17
77c65170 001af060 00000000 kernel32!BaseThreadStart+0x34
----------------------------------------------------------------------------------------------------
----------------------------UIThread-------------------------------------------------------
77f5d64e 000000a8 00000000 SharedUserData!SystemCallStub+0x4
000000a8 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
000000a8 77f51c5c 77fc2340 ntdll!RtlpWaitForCriticalSection+0x126
77fc2340 0c279070 0d6192e4 ntdll!RtlEnterCriticalSection+0x46
0c1f0000 0012e2a0 00000000 ntdll!LdrpGetProcedureAddress+0x10c
0c1f0000 0012e2a0 00000000 ntdll!LdrGetProcedureAddress+0x17
0c1f0000 0c279070 00000013 kernel32!GetProcAddress+0x41
0c279070 0d61807a 0c2704d0 sqlsrv32!ODBC___GetSetupProc+0x44
0d61807a 0c2792a8 0c278944 sqlsrv32!SQLGetPrivateProfileStringW+0xd
0d61807a 0c2792a8 001c8fe8 sqlsrv32!ParseAttrStr+0x2f3
001c8fe8 0d617c30 00000000 sqlsrv32!FillAttrStr+0xf0
001c8fe8 00000000 0bf32b70 sqlsrv32!SQLDriverConnectW+0xad
0bf32980 00000000 0bf32b70 odbc32!SQLInternalDriverConnectW+0x142
0bf32980 00000000 00f40d00 odbc32!SQLDriverConnectW+0x6ba
01a408c0 00000000 00f40d00 msdasql!CODBCHandle::OHDriverConnect+0x80
00000000 00f40d00 00000054 msdasql!CHdbcNode::DriverConnect+0x20
00000000 0c93f38c 0c944e8c msdasql!CImpIDBInitialize::Initialize+0x161
0c93b150 0012f160 0c944e8c oledb32!CDBInitialize::DoInitialize+0x32
0c93f38c 001a0c24 001a0c10 oledb32!CDBInitialize::Initialize+0x58
00f4c58c 0012f260 0012f288 oledb32!CDCMPool::CreateResource+0x1fa
001a0c24 0012f260 0012f288
comsvcs!CHolder::SafeDispenserDriver::CreateResource+0x25
00000000 0012f260 0012f288 comsvcs!CHolder::AllocResource+0x325
01a0df80 0c944de8 00000017 oledb32!CDCMPool::DrawResource+0x9d
01a0df80 0c944de8 00000017 oledb32!CDCMPoolManager::DrawResource+0x47
0c944df8 00000000 0c944d58 oledb32!CDPO::Initialize+0x136
0c944d58 00000000 0c944b58 msado15!_ConnectAsync+0x1da
0c944d58 00e93ad8 0c944b58 msado15!ConnectAsync+0x1f
0c944b58 001c480c 001c6edc msado15!CConnection::Open+0xa26
----------------------------------------------------------------------------------------------------------------------------------------------
Ivan Brugiolo [MSFT]
It is not clear from the top-part posted here why it would be holding the
loader lock.
--
--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
<saravanan.so...@bayer.com> wrote in message
news:1163529507....@e3g2000cwe.googlegroups.com...
saravanan.so...@bayer.com
Please find the full stack below.
Thank You
Saravanan
-----t88c------------------------------------
0d571482 00010156 00000110 user32!InternalCallWinProc+0x1b
00000000 0d571482 00010156 user32!UserCallWinProcCheckWow+0x151
00974b78 00970e30 0001015a user32!SendMessageWorker+0x4bb
0d540000 00974b78 00000148 user32!InternalCreateDialog+0x9c9
0d540000 0d58b888 00000000 user32!CreateDialogIndirectParamAorW+0x31
0d540000 0d58b888 00000000 user32!CreateDialogIndirectParamA+0x19
0d58b888 00000000 0d540000 mfc42!CWnd::CreateDlgIndirect+0x175
0d58b888 00000000 00000000 mfc42!CDialog::CreateIndirect+0x27
0d58b888 00000000 0d540000 mfc42!CDialog::CreateIndirect+0x1f
000003e8 00000000 0d583600 mfc42!CDialog::Create+0x4d
00000000 00d84148 0d53f430 DLWin32!GetEvent+0xe17f
0d571519 00000000 0d540000 DLWin32!GetEvent+0xe102
0d540000 00000000 00000000 DLWin32!AboutDriverLINX+0x5f05
0d540000 00000001 00000000 DLWin32!GetEvent+0x100b3
0d571821 0d540000 00000001 ntdll!LdrpCallInitRoutine+0x14
00000000 00000000 0d53fb20 ntdll!LdrpRunInitializeRoutines+0x34e
00000000 00143780 0d53f96c ntdll!LdrpLoadDll+0x2fe
00143780 0d53f96c 0d53f94c ntdll!LdrLoadDll+0x133
7ffa8c00 00000000 00000000 kernel32!LoadLibraryExW+0x1af
0d53f9c0 00000000 00000000 kernel32!LoadLibraryExA+0x1d
0d53f9c0 445c3a43 494c7672 kernel32!LoadLibraryA+0xb1
0d53fb58 00000000 00000001 DrvLNX32!Volts2CalibratedCode+0x1cb
0d53fb58 00000000 00000001 DrvLNX32!Volts2CalibratedCode+0x290
0d53fd00 00d9a9bc 00d9a9bc DrvLNX32!Ordinal52+0x1162
0001014c 0c864b4c 00000001 DrvLNX32!OpenDriverLINX+0x11
0001014c 00000401 00000000
_UtilitiesSurrogate!DllUnregisterServer+0x8c23
00d9a9bc 00000401 00000000
_UtilitiesSurrogate!DllUnregisterServer+0x8f4d
00d9a9c4 0001014c 00000401 user32!InternalCallWinProc+0x1b
00000000 00d9a9c4 0001014c user32!UserCallWinProcCheckWow+0x151
00974658 00000401 00000000 user32!DispatchClientMessage+0xd9
0d53fe30 00000018 00974658 user32!__fnDWORD+0x22
77d0e4be 0001014c 00000401 ntdll!KiUserCallbackDispatcher+0x13
0001014c 00000401 00000000 user32!NtUserMessageCall+0xc
0001014c 00000401 00000000 user32!SendMessageTimeoutWorker+0x76
0001014c 00000401 00000000 user32!SendMessageTimeoutW+0x1f
0c864b4c 00000000 00d99fe0
_UtilitiesSurrogate!DllUnregisterServer+0x8961
00000000 00d99fe0 00d99fe0
_UtilitiesSurrogate!DllUnregisterServer+0xab91
0c85b758 00000001 00000000
_UtilitiesSurrogate!DllUnregisterServer+0xb9c7
00000000 00d99fe0 77d02292
_UtilitiesSurrogate!DllUnregisterServer+0xa7be
00000000 00000000 00d99fe0
_UtilitiesSurrogate!DllUnregisterServer+0xa976
77e4a990 00d99fe0 00000000
_UtilitiesSurrogate!DllUnregisterServer+0xb72b
0c85c82b 00d99fe0 00000000
_UtilitiesSurrogate!DllUnregisterServer+0xb6dc
----------------------------------------------------------------------------------------------------------------------------------
Ivan Brugiolo [MSFT]
(probably a dialog-box) from within DllMain.
This is the root cause of the problem you are seeing.
Making a non trivial call from DllMain ends up
in a RPC call, that will deadlock RPC-Rt4 and the loader-lock.
000003e8 00000000 0d583600 mfc42!CDialog::Create+0x4d
00000000 00d84148 0d53f430 DLWin32!GetEvent+0xe17f
0d571519 00000000 0d540000 DLWin32!GetEvent+0xe102
0d540000 00000000 00000000 DLWin32!AboutDriverLINX+0x5f05
0d540000 00000001 00000000 DLWin32!GetEvent+0x100b3
0d571821 0d540000 00000001 ntdll!LdrpCallInitRoutine+0x14
00000000 00000000 0d53fb20 ntdll!LdrpRunInitializeRoutines+0x34e
00000000 00143780 0d53f96c ntdll!LdrpLoadDll+0x2fe
00143780 0d53f96c 0d53f94c ntdll!LdrLoadDll+0x133
7ffa8c00 00000000 00000000 kernel32!LoadLibraryExW+0x1af
0d53f9c0 00000000 00000000 kernel32!LoadLibraryExA+0x1d
0d53f9c0 445c3a43 494c7672 kernel32!LoadLibraryA+0xb1
--
--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
<saravanan.so...@bayer.com> wrote in message
news:1163536160.2...@h48g2000cwc.googlegroups.com...
Jochen Kalmbach [MVP]
> Making a non trivial call from DllMain ends up
> in a RPC call, that will deadlock RPC-Rt4 and the loader-lock.
Just for reference:
http://msdn.microsoft.com/library/en-us/dllproc/base/dllmain.asp
<quote>
Calling functions that require DLLs other than Kernel32.dll may result
in problems that are difficult to diagnose. For example, calling User,
Shell, and COM functions can cause access violation errors, because some
functions load other system components. Conversely, calling functions
such as these during termination can cause access violation errors
because the corresponding component may already have been unloaded or
uninitialized.
</quote>
<quote>
There are serious limits on what you can do in a DLL entry point. To
provide more complex initialization, create an initialization routine
for the DLL.
</quote>
--
Greetings
Jochen
My blog about Win32 and .NET
http://blog.kalmbachnet.de/
saravanan.so...@bayer.com
loaded by a 3rd party driver for which I don't have the source code. I
did change the configuration and prevented the DLL from being loaded
and tested the same scenario a few times and verified that the
problem is not occuring and put it back to the original configuration
to reproduce the problem.
Why is it that the problem occurs only after a shutdown/reboot of the
server and if I kill the application and restart it, I am not able to
reproduce it?
Could it be because of the other services and applications that run
under the same priorirty during reboot get equal time slice by the OS
cauing the right amout of delay for the threads in this process to get
into a deadlock?
Thank You
Saravanan
Ivan Brugiolo [MSFT]
well-knonw
thread that is NOT holding the loader lock.
Possibly the 3rd party DLL is forcing the RPC layer to create
a binding handle for the PnP-Manager RPC interface from the wrong place.
If you do that first, then, the second time it might work better.
It's hard to suggest a more viable workaround without knowing the full
call-sequence.
--
--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
<saravanan.so...@bayer.com> wrote in message
news:1163549767.7...@h48g2000cwc.googlegroups.com...
saravanan.so...@bayer.com
I found a RegisterDeviceNotification API function. I will try that.
Thanks
Saravanan