Problem using DF_ALLOWOTHERACCOUNTHOOK and SetWindowsHookEx

[Nobody]

I have a problem with SetWindowsHookEx() failing with error code
ERROR_ACCESS_DENIED(5) when the target process is running as another user.
In this case, both users are members of the Administrators group. The OS is
XP Pro+SP2, and I am running it on the console on the same desktop and using
right-click "Run as" to run the target process.

I am using thread specific hooks(WH_GETMESSAGE) and the hook procedure
resides in a multithreaded DLL written in VC6. The target process is
enabling DF_ALLOWOTHERACCOUNTHOOK flag on the desktop and all calls succeed
when enabling that flag.

So what could be causing SetWindowsHookEx() to fail?

If I start the target process as the same user, it works fine, and I see
messages. The target process is a simple single threaded application that I
have written for testing. The real application is what I want to automate
for a customer, and since I don't have the source code for the real process,
I would use AppInit_DLLs to enable that flag on the real target process.
Needless to say I have already written that DLL and it loads fine and calls
to set the flag succeed, but SetWindowsHookEx() still fails, so I am using
my own simple target process until I found the real cause. Here is the
output from the code below using DebugView:

[3528] OpenInputDesktop succeeded.
[3528] SetUserObjectInformation succeeded

I already tried GetThreadDesktop, and OpenDesktop("Default") with the same
result. Here is the code in the target process which I run the first thing
when WinMain is called:

// Allow other hooks code
USEROBJECTFLAGS uof;
BOOL bRet;
char szDebug[200];

HDESK hDesktop = OpenInputDesktop(DF_ALLOWOTHERACCOUNTHOOK, FALSE,
DESKTOP_HOOKCONTROL|DESKTOP_READOBJECTS|DESKTOP_WRITEOBJECTS);
// HDESK hDesktop = GetThreadDesktop(GetCurrentThreadId());
if (hDesktop!=0) {
sprintf(szDebug, "OpenInputDesktop succeeded.\n");
OutputDebugString(szDebug);
} else {
sprintf(szDebug, "OpenInputDesktop failed, GetLastError = %u\n",
GetLastError());
OutputDebugString(szDebug);
}
uof.fInherit = TRUE;
uof.fReserved = 0;
uof.dwFlags = DF_ALLOWOTHERACCOUNTHOOK;
bRet = SetUserObjectInformation(hDesktop, UOI_FLAGS, &uof,
sizeof(USEROBJECTFLAGS));
if (bRet!=0) {
sprintf(szDebug, "SetUserObjectInformation succeeded\n");
OutputDebugString(szDebug);
} else {
sprintf(szDebug, "SetUserObjectInformation failed, GetLastError = %u\n",
GetLastError());
OutputDebugString(szDebug);
}

if (hDesktop) {
CloseDesktop(hDesktop);
}

Thanks in advance for any help...

[Nobody]

I have found the cause of the problem, and it turned out that I wasn't
enabling DF_ALLOWOTHERACCOUNTHOOK properly. I needed to call
SetThreadDesktop() for some reason for the changes to take effect. Below is
a function that one could call to allow other processes to hook your
process.

// Returns TRUE when successful, FALSE otherwsie
BOOL AllowOthersToHookMe()
{
USEROBJECTFLAGS uof;
BOOL bRet;

HDESK hDesktop = GetThreadDesktop(GetCurrentThreadId());
if (hDesktop==0) {
// GetThreadDesktop failed
return FALSE;

}
uof.fInherit = TRUE;
uof.fReserved = 0;
uof.dwFlags = DF_ALLOWOTHERACCOUNTHOOK;
bRet = SetUserObjectInformation(hDesktop, UOI_FLAGS, &uof,
sizeof(USEROBJECTFLAGS));

if (bRet==0) {
// SetUserObjectInformation failed
CloseDesktop(hDesktop);
return FALSE;
}

bRet = SetThreadDesktop(hDesktop);
if (bRet==0) {
// SetThreadDesktop failed
CloseDesktop(hDesktop);
return FALSE;
}
CloseDesktop(hDesktop);
return TRUE;
}