Remote desktop deadlock on XP SP2

[Jacques]

I have stumbled across an issue with remote desktop on Windows XP SP2
when used in conjunction with a VPN application. The VPN application
works by forwarding network packets from the kernel to usermode where
the network packets are encapsulated and decapsulated in UDP.

The following steps reproduce the problem:
* Connect with the client to the VPN. This assigns an IP in the
internal office network to the client.
* Attempt to use remote desktop to this client from another PC in the
office network.

The remote desktop session freezes for about 15s after the logon
phase. During this 15s no network traffic can be sent or received by
the client. After doing some debugging in windbg I noticed that the
following thread is executing inside a critical section :

kd> !kdexts.locks -v 0x8629b468

Resource @ 0x8629b468 Exclusively owned
Contention Count = 3084
NumberOfExclusiveWaiters = 6
Threads: 8588d6f0-01<*>

THREAD 8588d6f0 Cid 05a4.05ac Teb: 7ffdd000 Win32Thread:
e2449b38 WAIT: (UserRequest) KernelMode Alertable
85a067b0 NotificationEvent
85a067e0 NotificationEvent
8588d7e0 NotificationTimer
IRP List:
85eb0ea8: (0006,0094) Flags: 00000004 Mdl: 00000000
Not impersonating
DeviceMap e1004460
Owning Process 8588f6e8 Image: csrss.exe
Wait Start TickCount 258112 Ticks: 73
(0:00:00:01.140)
Context Switch Count 44 LargeStack
UserTime 00:00:00.031
KernelTime 00:00:00.171
Start Address 0x75b6b329
Stack Init baa50000 Current baa4f680 Base baa50000 Limit baa4c000
Call 0
Priority 15 BasePriority 15 PriorityDecrement 0 DecrementCount 0
ChildEBP RetAddr
baa4f698 8050017a nt!KiSwapContext+0x2e (FPO: [Uses EBP] [0,0,4])
baa4f6a4 804f973e nt!KiSwapThread+0x46 (FPO: [0,0,0])
baa4f6dc f75faba0 nt!KeWaitForMultipleObjects+0x284 (FPO: [Non-
Fpo])
baa4f718 bac5f771 termdd!IcaWaitForMultipleObjects+0x76 (FPO:
[Non-Fpo])
baa4f73c bac62077 RDPWD!WDW_WaitForConnectionEvent+0x29 (FPO:
[Non-Fpo])
baa4f75c bac5adbe RDPWD!WDWDDConnect+0x21f (FPO: [Non-Fpo])
baa4f790 f75fb2a0 RDPWD!WD_Ioctl+0xf72 (FPO: [Non-Fpo])
baa4f7a8 f75fb612 termdd!_IcaCallSd+0x30 (FPO: [Non-Fpo])
baa4f7c8 f75fc274 termdd!_IcaCallStack+0x42 (FPO: [Non-Fpo])
baa4f7ec f75fcc72 termdd!IcaCallDriver+0x74 (FPO: [Non-Fpo])
baa4f824 f75f931d termdd!IcaDeviceControlVirtual+0x188 (FPO: [Non-
Fpo])
baa4f87c f75f9cfa termdd!IcaDeviceControlChannel+0x2a1 (FPO: [Non-
Fpo])
baa4f890 f75f9f92 termdd!IcaDeviceControl+0x26 (FPO: [Non-Fpo])
baa4f8a8 804eddf9 termdd!IcaDispatch+0x13a (FPO: [Non-Fpo])
baa4f8b8 bf96ac88 nt!IopfCallDriver+0x31 (FPO: [0,0,0])
baa4f8cc bf93ab86 win32k!CtxDeviceIoControlFile+0x7e (FPO: [Non-
Fpo])
baa4f904 bff686bb win32k!EngFileIoControl+0x27 (FPO: [Non-Fpo])
baa4f96c bff68204 RDPDD!DDInit+0xf1 (FPO: [Non-Fpo])
baa4f99c bf8b4beb RDPDD!DrvEnableSurface+0x16a (FPO: [Non-Fpo])
baa4f9d4 bf892607 win32k!WatchdogDrvEnableSurface+0x36 (FPO: [Non-
Fpo])
baa4f9f0 bf8b2873 win32k!PDEVOBJ::bMakeSurface+0x43 (FPO: [Non-
Fpo])
baa4fa1c bf8b3272 win32k!hCreateHDEV+0x3a9 (FPO: [Non-Fpo])
baa4fb94 bf8b9b84 win32k!DrvCreateMDEV+0x4dc (FPO: [Non-Fpo])
baa4fc88 bf8bbe20 win32k!DrvChangeDisplaySettings+0x251 (FPO:
[Non-Fpo])
baa4fccc bf917e76 win32k!xxxUserChangeDisplaySettings+0x141 (FPO:
[Non-Fpo])
baa4fd40 bf800ff4 win32k!xxxRemoteReconnect+0x1f1 (FPO: [Non-
Fpo])
baa4fd54 8053c808 win32k!NtUserCallOneParam+0x23 (FPO: [Non-Fpo])
baa4fd54 7c90eb94 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @
baa4fd64)
WARNING: Frame IP not in any known module. Following frames may be
wrong.
0049fea0 75b6b4f0 0x7c90eb94
0049fff4 00000000 0x75b6b4f0

Threads Waiting On Exclusive Access:
85ac9790 85a97790 85b80790
85a05790
8587c6f0 8637b448

1 total locks, 1 locks currently held

One of the threads waiting on exclusive access is the usermode
application responsible for encapsulating and decapsulating packets in
UDP. This application uses an event loop with
MsgWaitForMultipleObjects() to determine when overlapped IO (that
represents network data coming from / going to the kernel mode
component) is complete. When investigating the stack for this thread
I could see that the call to MsgWaitForMultipleObjects results in a
call to win32k!NtUserCallNoParam which then blocks on trying to enter
the same critical section that the csrss.exe thread shown above has
entered via the call to win32k!NtUserCallOneParam. Examing the above
call stack further, I notice a call to RDPWD!
WDW_WaitForConnectionEvent().

It seems to me as if the csrss.exe (remote desktop) thread is waiting
for network data while the application that is responsible for
providing this data is blocked waiting for the csrss.exe thread to
exit the critical section.

Has anyone on this list seen similar behaviour? I have verified that
the same scenario works on Vista without any problems.

regards,
jacques

[Ivan Brugiolo [MSFT]]

The connection sequence of TS connection is more complicated
than what you have experiences and reported below.
In general what happens is:
The TDI-Client that acts as the TCP/IS server receives enough
connection request bytes to dispatch the request to connect
a session to termsrv.dll
TermSrv.dll creates a new session for the purpose of displaying the logon
prompt. The session creation is done by smss.exe, and, once the new
csrss.exe is up and running, it will be connected by termsrv,
and, instructed to initialize the video-stack in that session.
Once credentials are acruired for the temporary session, winlogon
will instruct termsrv.dll to disconnect the temporary session
and re-connect to old session. The one reported below
is one of the pieces of this sequence.
The session context in kernel-space associates to each session
is managed by a pair of drivers, rdpwd.sys and tdtcp.sys
(plus termdd.sys acting as a broker).
The code path below is the junction point of the output path
driven by the session space code, and the input path, that normally
lives on a DPC routine for the network miniport
(that upcalls to the tcpip.sys / tdi.sys code,
that upcalls to tdtcp.sys, that upcalls to rdpwd.sys,
that upcalls to termdd.sys).
The chain of upcalls to the stack go through a worker thread,
that implements the inverted call paradigm to termsrv.dll,
or to other coponents (suck as the Raw Input Thread in csrss.exe).

That explained, you need to be looking for the input
path of the RDP connection, and see what is blocking that one.

Vista and WinXp have a radically different network stack, and,
I recall having seen issues in WinXp with third party firewall
and other generic interception frameworks for TCP / NDIS
that causes general slowdowns and deadlocks.

--

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

"Jacques" <jacques...@gmail.com> wrote in message
news:e1662a3b-e1f7-420f...@w34g2000hsg.googlegroups.com...

[Jacques]

Thanks for the reply Ivan. I now understand a bit more about the TS
connection process.

That being said, I still don't understand why
the call path shown below blocks every user application that is doing
a system call via NtUserCallXXX. This is the reason why the input
path is blocked because in my case the user mode application that
is responsible for network packet processing is blocked inside a
MsgWaitForMultipleObjects() which blocks for the whole period that
the TS connection is frozen, even if I specify a wait timeout. It
does
not make sense to me that the code path shown below is doing a
KeWaitForMultipleObjects() without exiting the critical section
entered via NtUserCallOneParam().

I have verified that the TS connection works as expected if I don't
forward packets from our NDIS IM to usermode via an inverted call
mechanism but do all packet processing in the NDIS IM instead.

regards,
jacques

> "Jacques" <jacques.fou...@gmail.com> wrote in message

[Ivan Brugiolo [MSFT]]

I was almost sure that you had written (or you had been using)
a firewall that did pop-up a UI while holding the completion routine
for the TDI client, and you had stumbled into this.

Doing all of the processing in TDI/NDIS is the proper desing here.

Almost every api in win32k.sys exposed via NtUserXXX does
take an ERESOURCE called win32k!gpresUser.
That resource protects the window tree (and other things)
This naturally includes xxxRemote[Dis|Re]Connect code-path,
because they might cause a desktop-redraw, that you might want to issue
while you hold a lock while navigating the window tree.

There are in the win32k sub-system quite a few places where
interactions with the hardware (in form of DeviceIoControl)
happens while holding gpresUser.
Immagine calling SystemParametersInfo, that will call
NtUserSystemParametersInfo, that will issue a synchronus
read/write in order to extract a setting from an INI file
(or the registry, for what matters).

While the design is not ideal, that is the way the video-stack
reconnection was designed, and how the termsrv components affects that.
--

--
This posting is provided "AS IS" with no warranties, and confers no rights.

Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

"Jacques" <jacques...@gmail.com> wrote in message

news:29060b9f-10df-4b81...@n20g2000hsh.googlegroups.com...

[Jacques]

Thanks for the clarification - I really appreciate it.
I'm going to move all packet processing out of usermode.

regards,
jacques

> I was almost sure that you had written (or you had been using)
> a firewall that did pop-up a UI while holding the completion routine
> for the TDI client, and you had stumbled into this.
>
> Doing all of the processing in TDI/NDIS is the proper desing here.
>
> Almost every api in win32k.sys exposed via NtUserXXX does
> take an ERESOURCE called win32k!gpresUser.
> That resource protects the window tree (and other things)
> This naturally includes xxxRemote[Dis|Re]Connect code-path,
> because they might cause a desktop-redraw, that you might want to issue
> while you hold a lock while navigating the window tree.
>
> There are in the win32k sub-system quite a few places where
> interactions with the hardware (in form of DeviceIoControl)
> happens while holding gpresUser.
> Immagine calling SystemParametersInfo, that will call
> NtUserSystemParametersInfo, that will issue a synchronus
> read/write in order to extract a setting from an INI file
> (or the registry, for what matters).
>
> While the design is not ideal, that is the way the video-stack
> reconnection was designed, and how the termsrv components affects that.
> --
>
> --
> This posting is provided "AS IS" with no warranties, and confers no rights.

> ...
>
> read more >>