ShellExecute not returning a hInstance

WindHawk

unread,

Feb 18, 1998, 3:00:00 AM2/18/98

to

I'm having trouble using ShellExecute. I define a hInstance to
receive the results from ShellExecute, but when my application runs,
it ShellExecute doesn't return the hInstance, instead, it returns 2A
(in hex) which is 42 (dec), higher than the error messages. What could
be wrong? I've tried executing a different program (notepad.exe), but
I still get the same hInstance value. My program and notepad.exe both
execute fine when I call ShellExecute, I just don't get the program's
hInstance.
My final goal is to be able to run EnumWindows, and use the
callback function to perform GetWindowLong(hWnd, GWL_HINSTANCE) and
compare the hInstance returned with the hInstance returned by
ShellExecute... if they match, then I can save the hWnd for that
function and send it messages. (Particularly WM_QUIT) If there is a
simpler way to do either of these, I'd love to know.

Thank you! Please email me:

jb...@cornell.edu

David Crow

unread,

Feb 18, 1998, 3:00:00 AM2/18/98

to

According to ShellExecute's documentation, nothing is wrong. Any return
value larger than HINSTANCE_ERROR is considered a legitimate instance
handle.

WindHawk <jb...@cornell.deletethis.edu> wrote in article
<34ea2dd2...@newsstand.cit.cornell.edu>...

Raymond Chen

unread,

Feb 19, 1998, 3:00:00 AM2/19/98

to

In Win32, HINSTANCEs are relative to your own process, so
ShellExecute can't return you the HINSTANCE for the process it
created since that number means nothing to you.
--
(My return address is intentionally invalid; delete ".---" to get my real address.
My responses are not to be considered official technical support or advice.)

Josh Pollak

unread,

Feb 19, 1998, 3:00:00 AM2/19/98

to

These two posts seem to be both conflicting and logical. Why would ShellExecute return
a hInstance if it doesn't let your program reference the started application? As to
David Crow's post, that what I thought when I read the documentation, but 2A certainly
isn't the hInstance (and I can verify that with Spy++) , its not and error code, so
whats up?

Chris Marriott

unread,

Feb 19, 1998, 3:00:00 AM2/19/98

to

In article <34EBC8EF...@postoffice5.mail.cornell.edu>, Josh Pollak
<jb...@postoffice5.mail.cornell.edu> writes

>Why would ShellExecute return
>a hInstance if it doesn't let your program reference the started application?

Backward compatibility with old 16-bit code. If you want to be able to
"control" the application, use "ShellExecuteEx" instead.

Chris

----------------------------------------------------------------
Chris Marriott, SkyMap Software, UK (ch...@skymap.com).
Visit our web site at: http://www.skymap.com
Astronomy software written by astronomers, for astronomers.

Raymond Chen

unread,

Feb 19, 1998, 3:00:00 AM2/19/98

to

On Thu, 19 Feb 1998 00:53:51 -0500, Josh Pollak
<jb...@postoffice5.mail.cornell.edu> wrote:
>These two posts seem to be both conflicting and logical. Why would ShellExecute return

>a hInstance if it doesn't let your program reference the started application?

It returns an hInstance because that's what the 16-bit
ShellExecute returned, and we had to keep it that way to make it
easier for people to convert their 16-bit applications to 32-bit.

However, the only thing you can do with the hInstance that is
returned by ShellExecute is to cast it to an integer and compare
it against the error codes. It is not really an HINSTANCE; it's
just an integer that has been cast to an HINSTANCE for
compatibility reasons.