Data Execution Protection and Regsvr32

[Stephen Ing]

I'm trying to install a DirectShow filter that is statically linked to cv.dll
(OpenCV http://sourceforge.net/projects/opencv/). When I run my *.msi
installer, I get the following error message when it get's to registering my
filter.

"Module c:\Program Files\...\mycomponent.ax failed to register. HRESULT
-2147023898. Contact your support personel."

When I try to register the component manually using regsvr32.exe, I get:

"LoadLibrary("...\mycomponent.ax") failed - Invalid access to memory location"

I know this problem is related to DEP and can make it go away by turning off
the DEP but I would like to leave the DEP on for obvious reasons.

So, I started investigating this problem and this is what I found:

1. I tried to debug the cv.dll to see where the problem might be occuring
and I set a breakpoint in the DllMain and launched it in debug mode with
regsvr32 as the debug exe but it never hits the DllMain. It never hits the
DllRegisterServer entry point in my filter either. But it does give me the
error I mentioned above.

2. I wrote my own program which I am pasting at the bottom of my message to
register my own filter and it registers it successfully.

So what is regsvr32 doing that my program isn't? Or is my program simply
not being monitored for DEP. My boot.ini setting is OptIn. If set it to
AlwaysOn, I can't even get to the login screen.

Does anyone have any ideas?

Thanks,
Steve

// RegFilt.cpp : Defines the entry point for the application.
//

#define WIN32_LEAN_AND_MEAN // Exclude rarely-used stuff from Windows headers

#include <windows.h>

void NoDLLSpecified()
{
MessageBox(NULL, "No DLL Name Specified.\n\n"
"Usage: RegFilt [/u] dllname\n"
"/u - Unregister Server\n"
"/s - Silent; display no message boxes",
"RegFilt",
MB_OK|MB_ICONEXCLAMATION);
}

void LoadLibraryFailed(LPCTSTR szName)
{
char szMsg[255];

wsprintf(szMsg, "LoadLibrary(\"%s\") failed - The specified module could
not be found.", szName);

MessageBox(NULL, szMsg,
"RegFilt",
MB_OK|MB_ICONEXCLAMATION);
}

int APIENTRY WinMain(HINSTANCE hInstance,
HINSTANCE hPrevInstance,
LPSTR lpCmdLine,
int nCmdShow)
{
typedef HRESULT (__stdcall * fpEntryPoint)();

fpEntryPoint fp;

char szMsg[255];

int offset=0;

bool bUnreg=false;
bool bSilent=false;

if (lpCmdLine[0] == 0) {
NoDLLSpecified();
return -1;
}

while (lpCmdLine[offset] == '/') {
if (!strnicmp(lpCmdLine+offset, "/u ", 2))
bUnreg = true;

if (!strnicmp(lpCmdLine+offset, "/s ", 2))
bSilent = true;

int len = strlen(lpCmdLine+offset);

if (len >= 2)
offset += 2;
else
offset += len;

while (lpCmdLine[offset] == ' ')
offset++;
}

if (lpCmdLine[offset] == 0) {

if (!bSilent)
NoDLLSpecified();

return -1;

}

if (bUnreg) {

char* token = strtok(lpCmdLine+offset, "\"");

if (token) {

HINSTANCE hInst = LoadLibrary(token);

if (!hInst) {

if (!bSilent)
LoadLibraryFailed(token);

return -1;
}

fp = (fpEntryPoint)GetProcAddress(hInst, "DllUnregisterServer");

if (SUCCEEDED(fp())) {

wsprintf(szMsg, "DllUnregisterServer in %s succeeded.", token);

if (!bSilent)
MessageBox(NULL, szMsg, "RegFilt", MB_OK|MB_ICONINFORMATION);

}

} else
NoDLLSpecified();
} else {
char* token = strtok(lpCmdLine+offset, "\"");

HINSTANCE hInst = LoadLibrary(token);

if (!hInst) {

if (!bSilent)
LoadLibraryFailed(token);

return -1;

}

fp = (fpEntryPoint)GetProcAddress(hInst, "DllRegisterServer");

if (SUCCEEDED(fp())) {

wsprintf(szMsg, "DllRegisterServer in %s succeeded.", token);

if (!bSilent)
MessageBox(NULL, szMsg, "RegFilt", MB_OK|MB_ICONINFORMATION);

}
}

return 0;
}

[Stephen Ing]

So, I disabled some startup code and was able to boot up Windows with the DEP
option set to AlwaysOn and now my program to register the filter fails to
load the component. When I try to load the cv.dll directly by regsvr32, this
is what the output window display in VisualStudio 6. I guess I'm wondering
how to fix the DEP problem in OpenCV when the problem happens even before the
DllMain is executed.

Loaded 'C:\windows\system32\regsvr32.exe', no matching symbolic information
found.
Loaded 'ntdll.dll', no matching symbolic information found.
Loaded 'C:\WINDOWS\system32\kernel32.dll', no matching symbolic information
found.
Loaded 'C:\WINDOWS\system32\msvcrt.dll', no matching symbolic information
found.
Loaded 'C:\WINDOWS\system32\advapi32.dll', no matching symbolic information
found.
Loaded 'C:\WINDOWS\system32\rpcrt4.dll', no matching symbolic information
found.
Loaded 'C:\WINDOWS\system32\user32.dll', no matching symbolic information
found.
Loaded 'C:\WINDOWS\system32\gdi32.dll', no matching symbolic information
found.
Loaded 'C:\WINDOWS\system32\ole32.dll', no matching symbolic information
found.
Loaded 'C:\WINDOWS\system32\shimeng.dll', no matching symbolic information
found.
Loaded 'C:\WINDOWS\AppPatch\AcGenral.dll', no matching symbolic information
found.
Loaded 'C:\WINDOWS\system32\winmm.dll', no matching symbolic information
found.
Loaded 'C:\WINDOWS\system32\oleaut32.dll', no matching symbolic information
found.
Loaded 'C:\WINDOWS\system32\msacm32.dll', no matching symbolic information
found.
Loaded 'C:\WINDOWS\system32\version.dll', no matching symbolic information
found.
Loaded 'C:\WINDOWS\system32\shell32.dll', no matching symbolic information
found.
Loaded 'C:\WINDOWS\system32\shlwapi.dll', no matching symbolic information
found.
Loaded 'C:\WINDOWS\system32\userenv.dll', no matching symbolic information
found.
Loaded 'C:\WINDOWS\system32\uxtheme.dll', no matching symbolic information
found.
Loaded
'C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.2180_x-ww_a84f1ff9\comctl32.dll', no matching symbolic information found.
Loaded 'C:\WINDOWS\system32\comctl32.dll', no matching symbolic information
found.
Loaded symbols for 'C:\WINDOWS\system32\MSVCRTD.DLL'
First-chance exception in regsvr32.exe (CVD.DLL): 0xC0000005: Access
Violation.
The thread 0xEF0 has exited with code 3 (0x3).
The program 'C:\windows\system32\regsvr32.exe' has exited with code 3 (0x3).

Thanks in advance for any suggestions,
Steve

[Pavel Lebedinsky]

1. Install latest debuggers from
http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx

2. Run regsvr32 under debugger like this:

c:\debuggers>cdb.exe -g -G regsvr32 your.dll

3. When you get access violation, type these commands:

0:000> .symfix+ c:\symbols
0:000> .reload
0:000> r
0:000> kb
0:000> !analyze -v

If you still can't tell where the problem is, post the output here.

[Stephen Ing]

Pavel,
Thanks for your response. As it turns out we discovered that a the OpenCV
function cvLoadPrimitives() was getting called before the DllMain(). The
cvLoadPrimitives() eventually called a function to identify the CPU which was
executing some machine code that was stored in a static array. I changed it
to an inline assembly block and now it works fine. On a side note, I see
that the CPU id code is extremely outdated and didn't even work properly on
my CPU but that's beside the point. At least it doesn't cause a DEP failure.
Steve