Problem using MapViewOfFile on Windows 2003 x64 edition in x86 pro
alunw
I have written database software which uses file mappings to share memory
between clients and the DBMS. The code that creates the mapping is failing on
Windows 2003 R2 X64 edition at the point where MapViewOfFile() is called
GetLastError() return 0x3e6 (error 998 "invalid access to memory location")
NB At this point I am just running the ordinary X86 programs though I fully
intend to port it to X64 when time permits. But I need this code to work from
32 bit client programwhich might not get ported.
The code works fine on all other Windows versions (even Windows CE) but it
will doubt turn out that I am doing something I'm not supposed to, but I have
no idea what it can be.
Here is a code snippet:
/* ensure the GetLastError() does not return a random value */
SetLastError(0);
if ((master_handle = CreateFileMapping(INVALID_HANDLE_VALUE,SA,
PAGE_READWRITE,0,master_size,make_ipc_name("SM",gmem_ptr))) == 0)
{
printf("attach master failed 1\n");
return 1;
}
created = GetLastError()!=ERROR_ALREADY_EXISTS;
printf("GetLastError() is now %d\n",GetLastError());
if ((master = MapViewOfFile(master_handle,FILE_MAP_WRITE,0,0,0)) == 0)
{
printf("attach master failed 2 %x\n",GetLastError());
return 1;
}
The code is failing in the call to MapViewOfFile() GetLastError() returns 0
before it and 998 afterwards. The call to CreateFileMapping() succeeds.
make_ipc_name() is a utility function to generate a name such as "Local\SM"
or "Global\SM" or "SM" depending on Windows version and how SW is configured.
SA is created like this:
#ifndef UNDER_CE
static char c[SECURITY_DESCRIPTOR_MIN_LENGTH];
static PSECURITY_DESCRIPTOR psd = (PSECURITY_DESCRIPTOR)c;
static SECURITY_ATTRIBUTES sa =
{sizeof(SECURITY_ATTRIBUTES),(PSECURITY_DESCRIPTOR) c,0};
#define SA &sa
#else
#define SA 0
#endif
Alexander Grigoriev
backed by the paging file (hFile is INVALID_HANDLE_VALUE), the paging file
must be large enough to hold the entire mapping. If it is not, MapViewOfFile
fails.
"alunw" <al...@discussions.microsoft.com> wrote in message
news:36A09FA8-5F37-4E25...@microsoft.com...
alunw
"Alexander Grigoriev" wrote:
> Do you have enough space in the pagefile for it? If a file mapping object is
> backed by the paging file (hFile is INVALID_HANDLE_VALUE), the paging file
> must be large enough to hold the entire mapping. If it is not, MapViewOfFile
> fails.
>
The object is only 12512 bytes in size, and I already specified the size
when I called CreateFileMapping(). I've tried rounding it up to 16384 bytes
in case there is some kind of page size issue. I presume the OS would grow
the paging file automatically if by some bizarre turn of events it was
initially too small. The documentation of CreateFileMapping() states
"If an application specifies a size for the file-mapping object that is
larger than the size of the actual named file on disk, the file on disk is
grown to match the specified size of the file-mapping object. If the file
cannot be grown, this results in a failure to create the file-mapping object.
GetLastError will return ERROR_DISK_FULL."
I could understand an error like that or an access denied type error, but
not the error code I am actually getting. This is a show stopping problem for
me. If I can't fix it I'll be reduced to using a shared file and really
reading and writing data into it on the X64 platform.
alunw
I think the failure at MapViewOfFile() may be a red herring:
the executable that is encountering the problem has anti-hacking code built
into it and modifies itself abd descrypts itself as it is loaded
intentionally generating and catching a very large number of exceptions.
Another executable using the same code that is not protected does not fail
here.
Is it possible that the OS is deciding to fail my call to MapViewOfFile()
because it has decided my program may be a security threat? I am not
encountering any problem during the loading itself - the code has loaded
properly and decrypted itself correctly, but then gets this strange error
when it tries to call MapViewOfFile()
Volodymyr Shcherbyna
alunw wrote:
> I think the failure at MapViewOfFile() may be a red herring:
> the executable that is encountering the problem has anti-hacking code
> built into it and modifies itself abd descrypts itself as it is loaded
> intentionally generating and catching a very large number of exceptions.
> Another executable using the same code that is not protected does not fail
> here.
I don't think OS is caring about it. Most likely, the protection mechanism
is buggy. Usually these applications wrap import table with own handlers to
make it hard to analyze and debug the file by other people. The
MapViewOfFile and other memory management functions are of special care, as
the person who will be trying to dump the file (to remove protection
mechanism and just dump the pure executable), will be putting breakpoints
to memory management functions, and thus, inside these functions they
implement "smart" logics.
--
Volodymyr M. Shcherbyna, blog: http://www.shcherbyna.com/
(This posting is provided "AS IS" with no warranties, and confers no rights)
Alexander Grigoriev
"Volodymyr Shcherbyna" <v_sch...@online.mvps.org> wrote in message
news:O$yx3bBAK...@TK2MSFTNGP03.phx.gbl...
alunw
is protected has been loaded OK - certainly if anything had gone wrong it
would have malfunctioned in a horrible way, because the code area is garbage
before hand. Only the code itself is encrypted - I did not do anything to
modify DLL import sections of the .exe file . I'm not worried about these
because all the important imports are via ordinal from DLLs that don't export
function names and have no debugging information.
In fact the code that is failing is in another DLL that does not have any
special features and the call to MapViewOfFile is using data generated
entirely within that DLL. So there is no reason for it to fail.
I would not have been too surprised if the executable had completely failed
to load or had gone horribly wrong because the decryption code switches
exception handlers round using the TIB directly, generates hundreds of
exceptions and makes calls to VirtualProtect() to change the protection on
the code sections to make it writeable and then back again. All that seems to
work OK (I had to change that code before - when MS introduced address space
randomisation). Afterwards everything seems OK, except for some KERNEL32
APIs, and just the ones I would think of breaking if I wanted to stop a
malicious program from harming a machine.
For the time being I am going to a build an unprotected version of the
problem executable files that will only run in the WOW64 environment, so that
they will be useless to anyone trying to pirate the software who does not
have a 64bit OS. I probably won't bother to try to puzzle out why this code
should be fine on every other Windows version from Windows95 to Windows 2008
in 32 bit mode but wrong in the 64 bit OS.
m
that works along these lines can be defeated by anyone with a copy of your
binary who is willing to spend enough effort to do it.
With a debugger, and a knowledge of assembly language, no program that the
CPU can execute successfully can be immune to decompilation on Windows
"alunw" <al...@discussions.microsoft.com> wrote in message
news:415FF2CE-EE9B-49DF...@microsoft.com...
Paul Baker [MVP, Windows Desktop Experience]
I was hacking like that in the '80s on an 8-bit computer in order to copy
games from tape to disk! Just so I didn't have to wait forever for them to
load, I might add.
There was a device that attached to the RS232C interface that could somehow
step through the instructions and make it easy to PEEK and POKE or just dump
memory or something. But I didn't have one of those :(
A typical copy protection mechanism decoded the next block of code using the
Z80 R register which would effectively be random. In order to see the code I
wanted in memory, I had to use a tool that came with my assembler to POKE
something into memory at the end of the current block of code, so I would
see then next one and do the same thing with that before I allowed it to
execute. In the end I got was the code I wanted. Which told me how it read
the data from tape. Usually the kind of information I needed was the address
it was loaded to in memory (no relocation!), if there were multiple parts
(usually there was a sort of a splash screen and sometimes there was level
data), what the wavelength for a bit was expected to be (I think in CPU
ticks) and any other decoding or fiddling around it did. Then I just had to
put something on disk that would load the same data into the same place in
memory somehow. Then I would look for cheats. It's interesting how few times
you will find it decrementing a register and comparing it to 0 in a program
that can at most be 64 KB. Could it be decrementing number of lives?
Probably! Put a NOOP in there :)
So, this was pretty complicated, but I put in the effort and I know others
did because there were magazine articles about it.
All that would have been a lot easier with the RS232C device, but I couldn't
afford it at the time. And your software options are limited if there is
only 64 KB of memory that the program is already using most of and it could
crap on you at any time. Although somehow the assembler worked. Hmm. I can't
remember anything about how these programs cooexisted without issue.
Paul
"m" <m@b.c> wrote in message news:%234Mzx5m...@TK2MSFTNGP04.phx.gbl...
Volodymyr M. Shcherbyna
> In fact the code that is failing is in another DLL that does not have any
> special features and the call to MapViewOfFile is using data generated
> entirely within that DLL. So there is no reason for it to fail.
Have you tried breaking in code when MapViewOfFile fails and looking at
stack? The first thing I would do if this happens on my side is to run
unprotected software, dump stack, and run protected software and dump stack
after MapViewOfFiles returns. Now you can analyze the differences. You
could also make small application, i.e. a tiny example to repro the problem
and analyze it in more details in OllyDbg or WinDbg. Protect it, and put it
here, maybe someone will be interested in analysis.
I would not suggest you to leave this problem unsolved, as it could be a
hidden bug of your protector in x32 as-well.
Volodymyr M. Shcherbyna
> For completeness, I feel compelled to mention that any protection
> mechanism that works along these lines can be defeated by anyone with a
> copy of your binary who is willing to spend enough effort to do it.
>
> With a debugger, and a knowledge of assembly language, no program that the
> CPU can execute successfully can be immune to decompilation on Windows
Yes, but the general rule is:
if the time spent (which is money these days) on breaking code is more than
the price of a product, there is no need to break it, just buy.
I saw nice protection mechanism which are impossible to break with the rule
above. The code is encrypted via asymmetric algorithm, you enter the proper
key and the address for jmp is calculated properly. Otherwise the jmp goes
to nowhere. The file is big. Very big. And obfuscated. You can spend weeks
doing that, analyzing subroutines.
Taking into account usual salary of software developer or security engineer
in EU or USA, the cost of breaking becomes thusands of EUR & USD. How many
products you know which cost thousands? Well, a few, but they are usually
big and they can have even more serious protection and you spend even more
time ...
alunw
hidden because programs protected with it work well on all 32 versions of
Windows up to and including Windows 7. So I have other more important things
to worry about.
As I think you said in one of my posts the purpose of the protection is to
make hacking the software uneconomic, not impossible. Actually it will be
difficult for someone who is not very able to defeat - you cannot even trace
into the executable inside Visual Studio for example, and the starting
address of the program and various other things in the executable header are
part of the decryption key, so it would be very hard to either modify or
remove the protection. In any case the protection does not stop users from
running the program so they very little reason to want to do this. The worst
that will happen is that they get restricted functionality if they don't have
a proper licence.
Mathmatician Manqué
find a solution.
My programs use an entry point called premainCRTStartup() which is in
a separate code section. When this has decrypted the executable
residing in the second code section and decided nobody is debugging
it, then it calls mainCRTStartup().
I put a similar shell round a completely ordinary program, and removed
all the fancy stuff my program did. That worked OK, so I then put back
some of the code bit by bit to see what it took to break file
mappings. In the end I determined that the simple action of calling
VirtualQuery() using the starting address of the code (which I do to
work out how to call VirtualProtect() so that I can decrypt the code
in place) is sufficient to stop file mappings working later on. Co-
incidentally I have discovered that there is a hot fix for a similar
though not identical problem with VirtualQuery() on x64 Windows XP and
2003. So I suspect I am falling over something similar. I'm going to
see if VirtualQueryEx behaves any differently. If not I'll try to work
out if I can work out what the size of the region must be from the
other information in the section headers, and get by without calling
it.
Mathmatician Manqué
I have now been able to fix my code protection to work on X64 windows.
There were two problems specific to this platform:
1) Calling VirtualQuery() seems to have a bad effect on later file
mappings. In fact I do not need to call this function at all, as
VirtualProtect() works with any old pointer as the start pointer. The
area I am changing the protection on is certainly all part of one
section.
2) There was a second completely unrelated problem that was stopping
my protected software working. The code protection is intended to be
difficult to debug. One of the things it was doing was using the
"Arbitrary Data" area in the TIB (The dword at FS:0x14). It would
appear that this causes a problem for the OLEACC DLL on X64. I'm not
sure why this DLL is being loaded at all, or why it should need to use
this area of the TIB. I've stopped writing to this area now, and I can
now use my protection mechanism with no problems.