Lazy write threads and the filesystem

[Joseph Bruno]

If I open a memory-mapped file, use MapViewOfFile, and alter data in the
mapped memory, then the file gets written when I close it or when I do
FlushViewOfFile. If I do neither of these things then my alterations remain
in memory only and are not flushed to disk unless Windows needs the memory
for other things.

In other words, there is no automatic "lazy write" thread for mapped memory.
The sequence:
1. CreateFile.
2. MapViewOfFile.
3. Make changes to the mapped memory.
4. Go out to lunch.
5. Come back.
6. Switch off the computer by pulling out the plug.
will result in all my changes being lost... unless at some time between [3]
and [6] Windows is asked to do something that requires a lot of physical
memory and therefore needs to page out my mapped-memory changes to my file.
(Or, of course, unless I call FlushViewOfFile or simply close the file).

My question is: what happens if I do not memory-map, but use WriteFile? Is
it guaranteed that any changes that I make using WriteFile will be flushed
out to the disk automatically within a reasonable timeframe, or could they
sit around in buffers indefinitely?

[Scherbina Vladimir]

"Joseph Bruno" <nob...@nothing.com> wrote in message
news:%23l387SB...@TK2MSFTNGP11.phx.gbl...

When you call WriteFile, system might cache data and its not actually
written to physical media. Use FILE_FLAG_NO_BUFFERING in CreateFile to
ensure that you'r data is written. BTW, you should also remember that HDD
might also have it's own cache, and FILE_FLAG_NO_BUFFERING does not
influence it.
As an alternative, you can use FlushFileBuffers before invoking CloseHandle
on file's handle.

--
Vladimir

[Joseph Bruno]

Thank you for that. I don't really want to force FILE_FLAG_NO_BUFFERING
because of the performance penalties (10 times slower operation, in some
cases).

I am perfectly happy for Windows to make its own decisions about what to
cache and when to flush. I just want to know whether a file left open (and
idle) after WriteFile is guaranteed to have its data flushed to disk
eventually even if I don't call FlushFileBuffers.

[Tom Widmer [VC++ MVP]]

It isn't, no. You'll have to set up a periodic call to FlushFileBuffers
if you need it, or call FlushFileBuffers after performing your writes.
Once you've done that, the data will eventually be written to the HDD,
though due to HDD caching, it may not happen instantaneously depending
on the current IO load.

Tom

[Joseph Bruno]

Thank you for the warning. What, out of interest, does the lazy write thread
do? I thought it wrote lazily, but your reply seems to imply that it does
nothing at all.

"Tom Widmer [VC++ MVP]" <tom_u...@hotmail.com> wrote in message
news:%23JsOpeD...@TK2MSFTNGP15.phx.gbl...

[Pavel Lebedinsky [MSFT]]

The lazy writer will eventually write modified pages to disk. How
long it takes depends on what else is going on in the system
but I believe the typical life time of a dirty cache page is
measured in seconds.

To compare, for modified mapped pages it could be several
minutes, though this will probably change in Vista - see page 23
of this presentation:

http://download.microsoft.com/download/f/0/5/f05a42ce-575b-4c60-82d6-208d3754b2d6/MemoryManagerInWindows.ppt

Windows Internals by Russinovich & Solomon has more details
about the cache manager.

--
This posting is provided "AS IS" with no warranties, and confers no
rights.

[Tom Widmer [VC++ MVP]]

Joseph Bruno wrote:
> Thank you for the warning. What, out of interest, does the lazy write thread
> do? I thought it wrote lazily, but your reply seems to imply that it does
> nothing at all.

This is the best I could find on the issue:

http://www.microsoft.com/technet/prodtechnol/windows2000serv/maintain/optimize/wperfch7.mspx

It seems that your data should actually be autoflushed if the disk is
idle for at least a second or so, or if you've changed enough of the file.

Tom

[Alexander Grigoriev]

Thanks, that PPT is very informative!

"Pavel Lebedinsky [MSFT]" <pa...@online.microsoft.com> wrote in message
news:en8PXGOH...@TK2MSFTNGP09.phx.gbl...

[Joseph Bruno]

What the Powerpoint presentation says is untrue - or else something strange
is happening on my system!

I have run the following program:

#define NBYTES 1000000
HANDLE
h=CreateFile("C:\\Test.txt",GENERIC_READ|GENERIC_WRITE,0,NULL,CREATE_ALWAYS,FILE_FLAG_RANDOM_ACCESS,NULL);
SetFilePointer(h,NBYTES,NULL,FILE_BEGIN);
SetEndOfFile(h);
HANDLE hMap=CreateFileMapping(h,NULL,PAGE_READWRITE,0,NBYTES,NULL);
BYTE *p=(BYTE *)MapViewOfFile(hMap,FILE_MAP_WRITE,0,0,NBYTES);
memset(p,'A',NBYTES);
FlushViewOfFile(p,NBYTES);
memset(p,'B',NBYTES);
Sleep(60*60*1000); // Wait an hour
UnmapViewOfFile(p);
CloseHandle(hMap);
CloseHandle(h);

This program takes an hour to run. If, after it has finished, I open
TEST.TXT in Notepad, I see a million occurrences of the letter B.

But if I start the program, wait 20 minutes, and then switch off the power
to my computer, then when I restart the computer and open TEST.TXT I see a
million occurrences of the letter A. In other words, the memory-mapped pages
were never flushed.

Thus when page 23 of the referenced Powerpoint presentation states that in
Vista "Data is written out much sooner than the prior 5 minute "flush
everything" model", implying that in earlier versions of Windows you could
rely on memory-mapped pages being written out within 5-10 minutes, this is
false. Anyone who constructs a program relying on this information risks
massive data loss. This happened to one of our users, who had a system crash
some hours after the last modification was made to a memory-mapped page,
only to find that the changes had not been written to disk.

I did my tests on Windows XP Home Edition, SP2 (some earlier tests were done
on the Professional Edition). It would be worth trying my program on Vista
to whether TEST.TXT ends up containing the letter A or the letter B.

"Pavel Lebedinsky [MSFT]" <pa...@online.microsoft.com> wrote in message
news:en8PXGOH...@TK2MSFTNGP09.phx.gbl...

> To compare, for modified mapped pages the delay before writing to disk

[Pavel Lebedinsky [MSFT]]

Yes, the only reliable way to write modified mapped pages to disk
is FlushViewOfFile, and that's what we recommend the apps should
be doing.

The presentation talks about the behavior of the mapped page
writer, which writes out pages from the modified list (see Windows
Internals more info). To get to the modified list, pages first have
to be unmapped by the application (or trimmed from its working
set). So if an app modifies some pages and doesn't unmap them
(and doesn't call FlushViewOfFile) it could easily take hours or
even longer before the changes are written to disk.