Runtime way of catching stack imbalance (ala PInvokeStackImbalance

[Marek]

Hi
Our end users need to be able to define native function calls (dll,
entry-point, calling conventions, parameters etc.) and these functions are
then called at runtime. In debug mode the MDA steps and a displays the
PInvokeStackImbalance dialog, but at runtime the function call just fails -
no exceptions are raised to be caught.

The question is how could I notify the users that their function declaration
is wrong?

[Willy Denoyette [MVP]]

"Marek" <m...@community.nospam> wrote in message
news:AD4D370B-7B79-4348...@microsoft.com...

How to enable MDA's:
http://msdn2.microsoft.com/en-us/library/d21c150d(en-US,VS.80).aspx

Willy.

[Jeffrey Tan[MSFT]]

Hi mav,

Thanks for your post. Yes, I can reproduce out this behavior. I will spend
some more time in this issue. Thanks

Best regards,
Jeffrey Tan
Microsoft Online Partner Support
Get Secure! - www.microsoft.com/security
This posting is provided "as is" with no warranties and confers no rights.

[Jeffrey Tan[MSFT]]

Hi Willy,

Yes, I have tried to use registry and configuration file in this link.
However, I still did not managed to get it work, so I will spend some more
time on it. Thanks

[Willy Denoyette [MVP]]

""Jeffrey Tan[MSFT]"" <v-j...@online.microsoft.com> wrote in message
news:QLzdOQZ...@TK2MSFTNGXA02.phx.gbl...

Try to set the environment variable COMPLUS_MDA to 1, this forces the CLR to
search the mda config file.

set COMPLUS_MDA=1

config file name must look like:
application.exe.mda.config

Willy.

[Marek]

Hi Willy
Thanks (again). The problem is not enabling the MDA - I already have it
enabled and it works fine in a debug build. The problem is in a release
build where the MethodInfo Invoke method does not nothing - no nice exception
to catch. It literally does nothing. I need to be able to do something like
check the stack status before and after and see whether something has become
corrupted or get the Invoke method to raise a meaningful exception.

Thanks for you interest in this.

Marek

[Jeffrey Tan[MSFT]]

Hi Marek,

Thanks for your feedback.

I have tried to a configuration file and environment variable to enable MDA
in CLR for both debug build and release build. When the p/invoke call stack
is not balance, I can get the MDA PInvokeStackImbalance, which causes a JIT
debugger to be selected. Do you get the same behavior as me?

Also, I have got confirmation from our product team that although we can
enable MDA at runtime, it is not shown out as exception, try...catch will
have no effect on this. It is something like Assert, not exception. So if
you just want a JIT to popup, yes, you can enable MDA at runtime. However,
there is no way to catch it as an exception.

Hope this helps

[Marek]

Hi Jeffrey
Thanks very much for looking into this for me. I can't really let my end
users be confronted with with a JIT window. Is there anything that could be
done in code to test the state of the stack before and after the invoke call
which would provide me with any information that I could use to determine
whether a stack imbalance had occurred?

Thanks again,

Marek

[Jeffrey Tan[MSFT]]

Hi Marek,

Thanks for your feedback.

Yes, I have already consulted our product team for this issue. However,
they confirmed that MDA can not help in this scenario. If you are curious,
below is the reply email content:
"The StackImbalance MDA also isn’t a catch-all. If you’ve got an
application model where the user can cause the CLR to invoke arbitrary
native functions calls, then it’s ALWAYS going to be unsafe and have the
potential to cause corruption. Invoking native methods is inherently
unsafe. The application could use various heuristics to try and weed out
some obvious errors, but the CLR doesn’t have any direct support to help
them with that."

Thanks

[Marek]

Hi Jeffrey
Thank you again. Please could you point me in the right direction of these
"various heuristics".

Best regards,

Marek

[Jeffrey Tan[MSFT]]

Hi Marek,

If you are curious, below is the feedback from our product team:

"In theory an app could do something like PInvoke to their own native
wrapper function which would do the work of calling the user-specified
native method and trying to detect some forms of corruption that can result
from an incorrect call (specifically looking for stack imbalance). Doing
this however would be non-trivial and would require a good understanding of
the native calling convention etc. Regardless, there are many other ways
an incorrect native call could corrupt memory which would be difficult or
impossible to detect, and so anything the application were to do here would
only provide partial protection (and perhaps a false sense of security).
For that reason, I don't think we should be encouraging the customer taking
this approach"

Hope this helps