SendMessage lockup within the same thread

[Kirill 'Big K' Katsnelson]

I posted that question a while back into some microsoft.* newsgroups
(as <1f1hcvkn3mrrm9ucn...@4ax.com>) but received no
replies. I would very much appreciate any thoughts on this problem!
I was still unable to do anything about it.

Environment: Windows 2000 SP3 Retail build + some security patches.

Short question: what can cause a call to SendMessage made from the
thread to which the target window belongs to block in
_NtUserMessageCall()?

Long question follows. I am sorry for not being able to squeeze a
shorter quintessence of it. :)

I am experiencing an unexplainable UI lock-up in the property page
of a DirectShow filter that I develop. The filter is a generic adapter
to host some other media technology filters in DirectShow audio
applications, so I control neither my host's nor my tenant's windowing.

I am passed a parent window handle to my implementation of
IPropertyPage::Activate, and create another empty dialog window to pass
to my tenant. The tenant creates its own property page in this empty
dialog window. This works in all cases except when the tenant creates
a button there. Upon clicking the button, the program locks forever.

Here is what happens. In its message loop, the host application
(written in MFC) retrieves the WM_LBUTTONDOWN message for the button
with GetMessage() (in CWinThread::PumpMessage). Then it calls
IsDialogMessage() for the window and all its consecutive parents
(MFC call CWnd::WalkPreTranslateTree()). On some iteration,
IsDialogMessage calls SendMessage (.. WM_GETDLGCODE) to a combo box
placed by the host in its property frame. This call results
ultimately in a call to NtUserMessageCall() that never returns.

All windows belong to one thread. Other threads in the host, according
to Spy++, do not host windows. Here is the stack of the UI thread:

USER32.DLL!_NtUserMessageCall@28() + 0xb
NTDLL.DLL!_KiUserCallbackDispatcher@12() + 0x13
USER32.DLL!_SendMessageWorker@20() + 0x3a5
USER32.DLL!_SendMessageW@16() + 0x44
USER32.DLL!_xxxRemoveDefaultButton@8() + 0x44
USER32.DLL!_xxxCheckDefPushButton@12() + 0xf3
USER32.DLL!_IsDialogMessageW@8() + 0x11b
USER32.DLL!_IsDialogMessageA@8() + 0x131
mfc42.dll!CWnd::IsDialogMessageA(tagMSG * lpMsg=0x0087fdec) Line 181 + 0xd C++
SONAR.EXE!004c0eed()
mfc42.dll!CWnd::WalkPreTranslateTree(HWND__ * hWndStop=0x00160606, tagMSG * pMsg=0x0087fdec) Line 2667 + 0xb C++
mfc42.dll!CWinThread::PreTranslateMessage(tagMSG * pMsg=0x0087fdec) Line 672 + 0xe C++
SONAR.EXE!0042211e()
mfc42.dll!CWinThread::PumpMessage() Line 821 + 0x10 C++
mfc42.dll!CWinThread::Run() Line 487 + 0x7 C++

According to SPY++, all windows belong to one thread in the application, so I am
excluding a cross-thread SendMessage block.

Window structure, abriged, per SPY++ is as follows. Numbers in [] refer to
footnotes below the diagram.

+ WS_POPUP window, class = Afx:....
+ lone WS_CHILD, class = Afx:... [1]
+ WS_CHILD, class = Dialog. [3]
+ (some windows omitted)
+ A combobox HWND=9029E. [2]
+ lone WS_CHILD, class = Dialog. [4]
+ many, many controls!
(somewhere deeper) A button which I click.

[1] This is the argument to IsDialogMessage on the stack above.
[2] This is the target of the SendMessage call that never returns.
[3] This is the window I receive as parent in IPropertyPage::Activate.
[4] This is what I create and pass as parent to the "other" techology.

I found an old thread on dejanews that explains symptoms quite like mine:

http://groups.google.com/groups?hl=en&lr=&threadm=%231M5t4haBHA.380%40tkmsftngp05

But the suggestion given there I cannot implement, since I am not in
control of what windows are created in my property page.

I am really confused as to what may cause a symptom like this, and would
appreciate any clue or suggestion!

-kkm