Regarding CloseHandle() on Memory Map File
Pawan.
I am facing a problem related to Memory map files. I have an application
which creates a file mapping and other applications map that file in its
virtual memory. And unmaps the view as soon as the work is over.
The creator of the Mapping file unmaps it from its view and call the
CloseHandle().
The problem is if the same memory map file name is used to create the file
it sometimes gives error that the map file already exists
(ERROR_ALREADY_EXISTS). My question is what is the time difference between
CloseHandle() and actually closing the mapfile (reclaiming the memory). And
is there a way to tune it or instruct the OS to immediately reclaim the
memory.
I am using Win2000 and MSVC7.0.
Thanks in Advance,
Pawan Waghalkar,
Software Engg.
India.
Patrick Philippot
CloseHandle just indicates that you no longer need to use the object in
the thread calling CloseHandle. However, another thread might have an
opened handle on the same object so there's no reason for the system to
immediately destroy the mapping object after you call CloseHandle. It
would be a very bad decision.
Since you obviously have at least two threads using the same mapping
object, just use OpenFileMapping instead of CreateFileMapping. If the
object already exists you'll get a handle on it, if it doesn't exist,
the call will fail and you can then use CreateFileMapping to actually
create it.
Since at least two threads can simultaneously get a handle on the same
mapping object, you also have to handle synchronization (for example by
using a mutex). You have to eliminate the risk of simultaneously writing
to / reading from the mapping object from multiple threads
simultaneously.
--
Patrick Philippot - Microsoft MVP
MainSoft Consulting Services
www.mainsoft.fr
Pawan.
thanks for your reply.
I am aware of this fact that even if the creator of Mapping file calls a
CloseHandle(), the OS will only reclaim that memory when all other
processes/thread atached to it get detached. I have taken care of the same
thing.
And for threads synchronization I am already using shared mutex.
I am facing this problem, when no nobody is connected to that file mapping
and the creator has already unmapped it from its virtual memory and called
CloseHandle() on it. I have confirmed it.
One thing I have observed that, this only happens on a very fast machine and
it is intermittent, FYI I have a two CPU machine.
Thats why, I wanted to know that is there any specfic time delay in between
CloseHandle() call and the OS actually reclaiming the memory, and is it
tunable?
TIA,
Pawan.
Patrick Philippot
If I understand well, you're calling UnmapViewOfFile and then
CloseHandle on the file mapping object, and even if no other thread is
using the object, a subsequent call to CreateFileMapping fails because
the system "thinks" that the file mapping still exists?
Do you have the same problem if you also call CloseHandle against the
file itself (of course you'll have to reopen the file before creating
the file mapping object again)?
UnMapViewOfFile()
CloseHandle() on the file mapping object
CloseHandle() on the actual file
I don't think you have any possibility to tell the system that it shoudl
release the object faster.
Since you are using a dual processor system, it seems that there's a
race condition between the thread cleaning up the object (that is, I
guess the problem is probably with the object namespace, not with the
object itself) and the thread that is trying to recreate a new mapping
with the same name.
Do you also have the problem if you Sleep() for a while before trying to
recreate the file mapping object?
Pawan.
Yeah you are right, It is the same case here.
Actually I am creating memory mapping (shared memory) so, I am not actually
using createFile() in my program (I am just passing a invalid handle i.e
0xffffffff instead, as a first argument to CreateFileMapping()).
Even though it is a dual processor system, I am executing the applications(
not threads) sequentially. After the first application compeletes i.e.
unmapping and closing the handle, then the second application comes in
picture (trying to create the File mapping with the same name used in the
first application).
Yeah you have raised a very good point, when I call sleep() for some seconds
in between the first application and second application call, the second
application sucessfully creates the file mapping and proceeds further. But it
is not a good practice or a full proof solution to avoid this problem, if you
are calling these two applications from a third application.
Anyways thanks for the reply. I just wanted to know if there a way exists to
tell the OS to release or reclaim the memory faster or immediately.
Thanks,
Pawan.
Patrick Philippot
> Yeah you have raised a very good point, when I call sleep() for some
> seconds in between the first application and second application
> call, the second application sucessfully creates the file mapping and
> proceeds further. But it is not a good practice or a full proof
> solution to avoid this problem, if you are calling these two
> applications from a third application.
Interesting. I'll try to find more information about this issue.
There's one thing I'd like you to try: create the object in the global
namespace just to see if this makes a difference. You just have to
prefix the object's name with "Global\"
Patrick Philippot
Question: I'm wondering about the usefulness of using a name ofr the
mapping object if the mapping is not done agaisnt a disk file and if
only one thread is using it at a time.
Patrick Philippot
Another question: are you absolutely sure that there's no other handle
opened on the same mapping in the first process? This could explain that
the second process can only create a mapping with the same name after
the first process has shut down completely.
Pawan.
Hi Patrick,
The answer of your first Q:
Actually the application was designed to run on a Solaris platform and
entirely developed in ANSI C. There I am using shared memory for IPC purpose.
Now I have ported the same application on Windows. There is no concept of
shared memory on Windows, so I am using memory mapped file. A process can
identify the memory map file with its name only. And in my product it is a
predefined name and known to all other applications programs (or utility
programs). So, using this name the other utility program can attach to the
memory map file.
It is not like that, only one application is going to use memory map file,
there are a number of utilities applications which can connect to it and can
read/write the information from it. It is a huge product with lots of utiltiy
processes doing there specified job.
Answer of your second Q:
Yeah, I have taken care of, when the creator of the Memory map file, unmaps
and calls CloseHandle(), there is no other process/ thread has mapped the
memory map file in its virtual memory.
One thing I have to confirm probably by writing a simple program, if I kill
a process which is attached to the memory map file, it will not get a chance
to unmap the view, then what happens to the no_attach count (number of
processes attached to it) maintained for the memory map file? Will OS
decrement it by one immediately or it will be done after some time?
Because in some cases the creator kills a utility which is attached to the
memory map file it has created, to ensure that no one has mapped the file
before calling CloseHandle().
This can be the possible reason, why the second application sees that the
memory map file still exists even after calling CloseHandle() on it.
What is your view regarding this?
TIA,
Pawan.
Patrick Philippot
> Yeah, I have taken care of, when the creator of the Memory map file,
> unmaps and calls CloseHandle(), there is no other process/ thread has
> mapped the memory map file in its virtual memory.
So there is something I don't understand. What are you sharing exactly
in that case? If no other thread or process has a handle on the mapping
object when you call Closehandle in that process, the mapping will be
destroyed when the process terminates. The data in the file mapping are
lost and there's no longer anything to share.
> I kill a process which is attached to the memory map file, it will
> not get a chance to unmap the view, then what happens to the
> no_attach count (number of processes attached to it) maintained for
> the memory map file? Will OS decrement it by one immediately or it
> will be done after some time?
When you terminate a Win32 process, all resources that were allocated to
this process are released, unless another process has a handle on them.
This is why there's generally no memory leak when a process terminates
under Win32. So again, if no other process has opened a handle on your
mapping, it is destroyed when you terminate the process even if you have
not explicitly called CloseHandle.
> Because in some cases the creator kills a utility which is attached
> to the memory map file it has created, to ensure that no one has
> mapped the file before calling CloseHandle().
> This can be the possible reason, why the second application sees that
> the memory map file still exists even after calling CloseHandle() on
> it.
No, this is not possible. The mapping can only survive if a running
process has a handle on it. You can check this with a utility named
WinObj that you'll find there:
http://www.sysinternals.com/SystemInformationUtilities.html .
Once you have created the named mapping, launch WinObj and look in the
BaseNamedObjects tree. You'll find your mapping there. Close your
process and you'll see that the mapping object disappears immediately
(refresh the WinObj display with F5).
This is why your problem is very strange. Normally, the object's name
disappears from the namespace as soon as the object is released. If
another process or thread cannot create a mapping with the same name,
this indicates that there is still a handle on it somewhere else.
Slava M. Usov
news:16648BEE-9B9F-4759...@microsoft.com...
[...]
> There I am using shared memory for IPC purpose.
[...]
> Yeah, I have taken care of, when the creator of the Memory map file,
> unmaps and calls CloseHandle(), there is no other process/ thread has
> mapped the memory map file in its virtual memory.
If "no process has mapped" means no process has a _handle_ opened to that
object _and_ the object mapped, then you cannot use it as IPC -- simply
because it is not shared at all; so this scenario does not make sense.
If it means that there _are_ processes with a handle opened, then
ERROR_ALREADY_EXIST is an _expected_ error code from CreateFileMapping() --
you should just use the handle returned and disregard the error, as
specified in the synopsis for the routine; this is a normal scenario for
_shared_ memory IPC.
> One thing I have to confirm probably by writing a simple program, if I
> kill a process which is attached to the memory map file, it will not get a
> chance to unmap the view, then what happens to the no_attach count (number
> of processes attached to it) maintained for the memory map file?
'no_attach' is something that you just invented; Windows has no such
counter.
Windows does keep a number of handles and references [they're two different
things] to each object, and it cleans the counters if a process exits
without proper cleanup.
[...]
> immediately or it will be done after some time?
Everything takes time; nothing is done immediately; this is a property of
this world and is not OS specific. You can talk about certain serializing
events, but you need to define them; so far you haven't explained how that
is done in your case.
S
Patrick Philippot
1. Put a breakpoint in your code just after the line that creates the
mapping object.
2. When this breakpoint is reached, launch WinObj and check the prešence
of your object in the namespace.
3. Put another breakpoint just after the call to Closehandle. If the
object is still visible in WinObj at this time (after a refresh), so
there's another valid handle on it.
4. Right-click on the object in WinObj, select Properties and look at
the reference count.
Patrick Philippot
1. Put a breakpoint in your code just after the line that creates the
mapping object.
2. When this breakpoint is reached, launch WinObj and check the prešence
of your object in the namespace.
3. Put another breakpoint just after the call to Closehandle. If the
object is still visible in WinObj at this time (after a refresh), so
there's another valid handle on it.
4. Right-click on the object in WinObj, select Properties and look at
the reference count.
--
Patrick Philippot
> Pawan. wrote:
>> Yeah, I have taken care of, when the creator of the Memory map file,
>> unmaps and calls CloseHandle(), there is no other process/ thread has
>> mapped the memory map file in its virtual memory.
>
> So there is something I don't understand. What are you sharing exactly
> in that case? If no other thread or process has a handle on the
> mapping object when you call Closehandle in that process, the mapping
I meant: "the mapping will be destroyed immediately [when you call
CloseHandle]".
Alexander Grigoriev
"Pawan." <Pa...@discussions.microsoft.com> wrote in message
news:8F5E34ED-363A-40B1...@microsoft.com...
Patrick Philippot
> Is there by any chance, antivirus running and scanning the files on
> access?
Good idea, Alexander. This should be checked. However, the problem would
then occur even more often on slow single processor systems. Pawan tells
us that the problem occurs only on fast, dual processor systems.
The more I think about it, the more I'm convinced there's a
hidden/forgotten reference somewhere in the code that is destroyed only
when the process terminates. But it's just a guess.
I'm using a dual processor system for development because some
synchronization bugs never appear on single processor systems. Pawan
certainly has a similar problem. Otherwise, this would mean that the
object destruction process triggered by a reference count returning to
zero after a call to CloseHandle is not synchronous with that call. I'm
not 100% sure that it's the case but I would be very surprised.
Hector Santos
I didn't read the entire thread so I don't know if anything of my comments
here was already touched on, but here is my experience here.
For us, the end result was that there was OS networking drivers that created
erroneous ERROR_ALREADY_EXISTS, especially going across the network
depending on the File Attributes for the CreateFile() function.
This was an old issue dating back to 2000 when it was found. A logical work
around was added, so I don't know if the errors still persist today and
would be surprise if it still does. I do know it was reported to Microsoft
when it was first found.
For our mapfile in this case, the first time the file is created, a header
is written to it, but only if it didn't exist. So once the file was open
with OpenAlways, it was important to know if the file already existed.
For example:
BOOL TMyMapFileClass::Open(const char *fn)
{
DWORD nFlags = 0;
nFlags |= FILE_ATTRIBUTE_NORMAL;
nFlags |= FILE_FLAG_RANDOM_ACCESS,
hFile = CreateFile(fn, GENERIC_READ|GENERIC_WRITE,
0,
NULL,
OPEN_ALWAYS,
nFlags, 0);
int err = GetLastError();
if (hFile == INVALID_HANDLE_VALUE) {
return FALSE;
}
ZeroMemory(&HashTable, sizeof(HashTable));
// v5.6.450.2, HLS, 12/20/00
// don't check error - fixes cross network bug
if (/* err != ERROR_ALREADY_EXISTS || */
GetFileSize(hFile, NULL) == 0) {
WriteAt(0, HashTable, sizeof(HashTable));
}
ReadAt(0, HashTable, sizeof(HashTable));
return TRUE;
}
Now, I can't 100% recall, but I pretty such when it was finally figure out,
it was one of the nFlags file attributes that caused the erroneous error and
only happen, in this case, when the file was across the network.
So if all possible try to see why the ERROR_ALREADY_EXIST is created for
you. There is no "randomness" to this. No Timing issue. Its a bug
somewhere. Either you expect this error or you don't.
See if the file attribute is causing it for you.
I'll be interested to know what you find.
--
Hector Santos, Santronics Software, Inc.
http://www.santronics.com
"Pawan." <Pa...@discussions.microsoft.com> wrote in message
news:2319A458-D293-41B8...@microsoft.com...
Hector Santos
Now that I am thinking about this, I am pretty sure I recall some reference
to a Microsoft service pack fix that addressed some network write/flushing
caching optimization issues. What I don't recall it specifically mentioned
the file attributes such as:
FILE_FLAG_RANDOM_ACCESS
FILE_FLAG_SEQUENTIAL_SCAN
This caused two issues; A) an errorneous ERROR_ALREADY_EXIST error when you
didn't expect it or B) no ERROR_ALREADY_EXIST when you did expect it.
In other words, a bug. :-)
I just wrote a quick program to repeat the problem.. I had no problem going
across drives to XP and 2000 machnes, but on run across to an legacy NT 4.0
machine, I was able to reproduce it.
With this program below, you would be able to see the issue if you have the
same problem on your OS network. It always works fine on local machine
files.
Testing Goal:
1) If the file exist, you MUST get ERROR_ALREADY_EXIST.
2) If the file does not exist, you MUST NOT get ERROR_ALREADY_EXIST
3) If you get the unexpected error, try removing the file attibute.
Source code:
// FileName: TestExistBug.cpp
// Compile: cl testexistbug.cpp /W3 /GX /MD /D "_AFXDLL"
#include <stdio.h>
#include <afx.h>
void main(char argc, char *argv[])
{
char *szfile = "q:\\testing\\foobar1.dat"; // network file
// DeleteFile(szfile);
DWORD nFlags = 0;
nFlags |= FILE_ATTRIBUTE_NORMAL;
nFlags |= FILE_FLAG_RANDOM_ACCESS;
HANDLE hFile = CreateFile(szfile, GENERIC_READ|GENERIC_WRITE,
0,
NULL,
OPEN_ALWAYS,
nFlags,
NULL);
int err = GetLastError();
printf("hFile: %d Error: %d\n",hFile, err);
if (hFile != INVALID_HANDLE_VALUE) {
CloseHandle(hFile);
}
}
Let us know what you find on your setup.
--
Hector Santos, Santronics Software, Inc.
http://www.santronics.com
"Hector Santos" <nospa...@nospamhere.com> wrote in message
news:uPyktLEg...@TK2MSFTNGP15.phx.gbl...
Hector Santos
> 'no_attach' is something that you just invented; Windows has no such
> counter.
Slava, give the guy a break - he is thinking out of the box. :-)
> Everything takes time; nothing is done immediately; this is a property of
> this world and is not OS specific. You can talk about certain serializing
> events, but you need to define them; so far you haven't explained how that
> is done in your case.
What is more important to instill is that there is no randomness the
"computer world." One can hide what seems to be a random timing issue with
a Sleep() or delays, but that's just it - it hides the real
(synchronization) bug in the program.
But we came across this ERROR_ALREADY_EXISTS error or no error (when
expected) with atleast NT 4.0 since the last time we worked around what I
recall to be a cross network file open with caching bug. Retesting the
issue moments ago, it is solid on XP and 2000 but I can repeat it on a
legacy machine NT SP6 machine. This dated back to 12/2000 when I
encountered the bug.
Patrick Philippot
> For us, the end result was that there was OS networking drivers that
> created erroneous ERROR_ALREADY_EXISTS, especially going across the
> network depending on the File Attributes for the CreateFile()
> function.
Hi Hector,
Please note that Pawan is not mapping a disk file (no file opened) but
using the MMF to share memory between processes. So these interesting
comments (I should note this somewhere) are not relevant in that case.
Pawan.
Thanks a lot for the information. It will really help me in my future
projects.
But actually here my case is different, I am not using any disk file for
mapping, actually I am using memory mapped files for IPC purpose.
thanks again,
Pawan.
Hector Santos
From: "Pawan." <Pa...@discussions.microsoft.com>
> Hi Hector,
>
> Thanks a lot for the information. It will really help me
> in my future projects. But actually here my case is
> different, I am not using any disk file for mapping,
> actually I am using memory mapped files for
> IPC purpose.
>
> thanks again,
> Pawan.
>
No problem.
We have a widely deployed product too and memory maps is a fundamental part
of the framework. Never in the 10 year history of this Windows product did
we get a report or issues related memory maps.
Looking over your messages in the thread, it seems you have a CSB issue.
The indication you had to inject sleeps to provide the illusion of a fix,
says it all. The indication that the CSB exist only a fast (and/or
multi-cpus) machine just means the CSB was always there and it took a faster
machine to rear its ugly head - a CSB!
Oh CSB? - Classic Synchronization Bug. <g>
How are you opening the memmap? Attributes, size? PAGE_XXXX? SEC_XXXX?
Are you using WIn32 functions or some higher level library?
Anyway, divide and conquer! :-)
Pawan.
I liked the term CSB, I have added it to my dictionary :)
I am using the standard Win32 api's :
CreateFileMapping()
OpenFileMapping()
MapViewOfFile() etc...
I am passing null initalized security attributes to CreateFileMapping(), to
initialize the security atributes I am calling,
InitializeSecurityDescriptor(mysd, SECURITY_DESCRIPTOR_REVISION);
SetSecurityDescriptorDacl(mysd, TRUE, (PACL) NULL, FALSE);
here mysd is of type PSECURITY_DESCRIPTOR.
And as far as the fprotect flag is concerned (If I am not wrong, you were
talking about the same) the third parameter to CreateFileMapping(), I am
passing it as PAGE_READWRITE.
And passing a specific size (in dwMaximumSizeLow) , which is in multiple of
64KB.
Yeah, after so much of discussion, I also think adding a sleep() is the only
workaround :)
Pawan.
Patrick Philippot
> Yeah, after so much of discussion, I also think adding a sleep() is
> the only workaround :)
Pawan,
This was suggested as a way of detecting a synchronization problem, not
as a fix. Did you try what I suggested with WinObj? Again, if you still
see the object in the namespace *after* you have called CloseHandle,
this clearly means that there's still another reference on it somewhere
else. You have to find out which process holds this reference. This is
not difficult by using Process Explorer for example:
http://www.sysinternals.com/Utilities/ProcessExplorer.html .
If you do have synchronization problem, the Sleep() trick will not fix
anything. The problem will re-appear in another form on a different
system. You should diagnose the problem thoroughly *now*.
Pawan.
I will try with the WinObj, to find out. As I am sure there is no other
process/thread holding a reference on the Memory Map file, still it is a good
idea to double check with this utility.
As the problem appears once in a while, so it will be a bit difficult task
But again thanks for suggesting me this utility.
Pawan.
Patrick Philippot
> I will try with the WinObj, to find out. As I am sure there is no
> other process/thread holding a reference on the Memory Map file,
> still it is a good idea to double check with this utility.
> As the problem appears once in a while, so it will be a bit difficult
> task But again thanks for suggesting me this utility.
Process Explorer has a search utility. So, given the name of the object,
you'll be able to quickly find which process is holding the reference.
Alexander Grigoriev
You say that other applications open the handles to it.
If you expect it to be gone when you reopen the MM object in your "creator"
application, your expectation may be unreasonable. You DON'T have control
nor information whether other apps still hold a handle to it.
If you care to actually wait untill all other applications are done with the
file object, you need to be very careful. Check that UnmapViewOfFile
actually succeeds there (that you're not passing a bogus pointer to it).
Check that you _actually_ wait until other apps are done with that MM obj
and have closed its handle. If the app is closing a bogus handle, then exits
immediately, you may get lucky and not get a CPU before an OS closes the
correct handle for you. But on an SMP machine you may get awaken too early.
Hector Santos
You know, this error 183 (ERROR_ALREADY_EXIST) is easily repeatable at
will. It is all about synchronizing access to your map.
While WinObj might give you some insight, ultimately you need to control
client access from screwing up the integrity of your server application. So
who cares who could holding access to your memory map handle(s)? You need
to stop this from happening in the first place if you plan to expose it for
3rd party utilities, including your own utilities..
You might wish to have a control environment using an wrapper API, client
DLL concept, maybe using a Reader/Writer locking method with a named kernel
event.
What you publish would be the API, not the map itself.
By doing this, you remove all issues about how utilities would be accessing
your server memory map with two basic functions:
BOOL WINAPI ReadMemoryMap(TMyMemData *pMap);
BOOL WINAPI WriteMemoryMap(TMyMemData *pMap);
That is all the world (clients) need to know about your server map. They
need to know nothing more. They don't need to know how the access is done.
Your API will synchronize it with the server and when all your utilities use
the same API, you have consistency and integrity and if you need to change
or fix logic in the API, by single sourcing it this way, the fix is applied
across the board.
Pretty standard stuff.
Hope this helps
--
Hector Santos, Santronics Software, Inc.
http://www.santronics.com
"Pawan." <Pa...@discussions.microsoft.com> wrote in message
news:A4B302F5-1459-4E06...@microsoft.com...