Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Why doesn't NT Overlapped WriteFile(Ex) overlap ???

1 view
Skip to first unread message

Graeme Gill

unread,
Jun 19, 1996, 3:00:00 AM6/19/96
to

In an attempt to get good file write performance, I have been investigating
Windows NT overlapped I/O. Microsoft make a big song and dance about NT
and overlapped I/O, but experiments so far indicate that it is not very
useful when writing files.

When writing to a fresh (e.g. new, or not previously written to) file, it
appears that NT doesn't overlap writes (NTFS). When writing over an
existing file, it does do overlapped writes. It makes no difference
whether one use WriteFile or WriteFileEx().

Is there any good reason for this, or some trick to get around this
problem ?? (Just allocating the file space using SetFilePointer()
SetEndOfFile(), or CreateFileMapping() doesn't work).

Does one have to revert to using threads to get overlapped I/O under NT ?
(MS make a big point of recommending overlapped I/O to avoid threads!)

Graeme Gill.


Edmond

unread,
Jun 19, 1996, 3:00:00 AM6/19/96
to


How are you overlapping what isn't there? If you have a zero length file, then
your going to have trouble overlapping (reading or writing). Your absolutely
right about it being somewhat synchronous in behavior (as it should be). Any time
your doing straight sequential I/O, your going to get that type of behavior. I
think threading your writes will only cause trouble. One way you could improve
upon overlapping I/O to sequential files is to have 2 overlapping data structures.
That way you don't have to wait for the previous data structure to set a file
pointer to the end of the file to append the data to it. You must calculate how
much your writing from the previous data structure though and set your events up
to signal at the proper times (so that one doesn't get stalled).


--
Edmond Underwood
Bench32 1.20 - The new threshhold in system benchmarking
Bench32 web site: http://www.rmii.com/~underwoe/bench32.html

George P. Dake

unread,
Jun 19, 1996, 3:00:00 AM6/19/96
to

gra...@wallaby.digideas.com.au (Graeme Gill) wrote in article
<4q7ion$j...@wallaby.digideas.com.au>...

> In an attempt to get good file write performance, I have been
investigating
> Windows NT overlapped I/O. Microsoft make a big song and dance about NT
> and overlapped I/O, but experiments so far indicate that it is not very
> useful when writing files.
>
> When writing to a fresh (e.g. new, or not previously written to) file,
it
> appears that NT doesn't overlap writes (NTFS). When writing over an
> existing file, it does do overlapped writes. It makes no difference
> whether one use WriteFile or WriteFileEx().
>
> Is there any good reason for this, or some trick to get around this
> problem ?? (Just allocating the file space using SetFilePointer()
> SetEndOfFile(), or CreateFileMapping() doesn't work).
>
> Does one have to revert to using threads to get overlapped I/O under NT
?
> (MS make a big point of recommending overlapped I/O to avoid threads!)
>

I don't know for sure, but think that even threads won't help you. When
you write to a file but don't change its size, then the file system data
structures for the file allocation don't have to change, nor do the free
space data structures for the volume. But, when you write past end of
file, you do change the of the file. NTFS probably has a thread get
exclusive access to the file in order to change the allocation. It would
get hairy indeed if multiple threads were extending the same file,
allocating space from the volume free space, etc. concurrently.

Also, I'm pretty sure NTFS allows for "file holes". Assume the end of
file for a file is 2GB, but only 512 bytes were written at the beginning
and another 512 bytes were written at the end. The rest of the file was
never written. A file system could just allocate 1024 bytes and remember
that the rest was unwritten. Then if an application ever asked to read
the other ~2GB, it would just return 0's. Several UNIX file systems and
NetWare's file system do this. Note that DOS's FAT doesn't do this; in
DOS or Windows 3.1 setting end of file does allocate space.

I'm not a Win32 expert but maybe there is a call to allocate space to a
file.

HTH,
Geo


--
------------------------------------+------------------------------------
George P. Dake | email: da...@mdi.com
Micro Design International, Inc. | phone: 800-920-8205 or 407-677-8333


Graeme Gill

unread,
Jun 20, 1996, 3:00:00 AM6/20/96
to

Edmond (unde...@Colorado.Edu) wrote:

: How are you overlapping what isn't there? If you have a zero length file, then


: your going to have trouble overlapping (reading or writing).

Why should there be any problems ovrelapping writes ? I do a write to the file,
the system call should return immediately. I am then free to do other things,
including doing more writes.

: Your absolutely


: right about it being somewhat synchronous in behavior (as it should be). Any time
: your doing straight sequential I/O, your going to get that type of behavior. I
: think threading your writes will only cause trouble.

The Win32 documentation claims that overlapped I/O allows non-syncronous behaviour.
It works OK for reads, it works for writes only when overwriting an existing file.
Why doesn't it work when writing a new file ? Is there a problem with NT, the Win32
documentation, or my code ?

: One way you could improve


: upon overlapping I/O to sequential files is to have 2 overlapping data structures.
: That way you don't have to wait for the previous data structure to set a file
: pointer to the end of the file to append the data to it. You must calculate how
: much your writing from the previous data structure though and set your events up
: to signal at the proper times (so that one doesn't get stalled).

Of course I am using two buffers, and two completion data structures. Of course
I am telling each write where to write the data into the file (not relying
on the system calls to track the current file size).

What follows is my test code (C++). I would be gratefull if anyone can point out
the blunder I am making.

Graeme Gill.


// Un-buffered, Overlapped I/O benchmark code.
// The CPU work routines aprox balance the I/O rate on a 90 MHz pentium
// with a 4Mb/sec SCSI drive.
// Creates test file test.xxx of aprox 100 Mbytes
// Try running the program twice in a row, after deleting test.xxx

#include <stdio.h>
#include <windows.h>

#define USE_OVERLAP
#define USE_UNBUFF

void usage();
void error(char *fmt, ...);
void warning(char *fmt, ...);

typedef unsigned char uchar;

int main()
{
int i,j,k, l = 0 ;
unsigned long stime,etime;

#define BSZ (400 * 1024) // Buffer size
#define MBYTES 100 // Total MBytes to test
#define REPS ((MBYTES * 1000000)/(BSZ * 2))

unsigned long fo; // file offset
int now = 0; // Non overlapped writes

uchar *buf1;
OVERLAPPED os1;
HANDLE ose1;

uchar *buf2;
OVERLAPPED os2;
HANDLE ose2;

unsigned long bwritten;
HANDLE fh;
SECURITY_ATTRIBUTES fhs = {
sizeof(SECURITY_ATTRIBUTES),
NULL, // default security
TRUE }; // Let handle be inherited

if ((buf1 = (uchar *)VirtualAlloc(
NULL, // address of region to reserve or commit
BSZ, // size of region
MEM_COMMIT, // type of allocation
PAGE_READWRITE // type of access protection
)) == NULL)
error("Alloc of buffer 1 failed");

if ((ose1 = CreateEvent(
NULL, // no security attr.
TRUE, // manual-reset event
TRUE, // initial state = signaled
NULL // unnamed event object
)) == NULL) // unnamed event object
error("Create event 1 failed");
os1.hEvent = ose1;

if ((buf2 = (uchar *)VirtualAlloc(
NULL, // address of region to reserve or commit
BSZ, // size of region
MEM_COMMIT, // type of allocation
PAGE_READWRITE // type of access protection
)) == NULL)
error("Alloc of buffer 2 failed");

if ((ose2 = CreateEvent(
NULL, // no security attr.
TRUE, // manual-reset event
TRUE, // initial state = signaled
NULL // unnamed event object
)) == NULL) // unnamed event object
error("Create event 2 failed");
os2.hEvent = ose2;

////////////////////////////
stime = GetTickCount();

fh = CreateFile(
"test.xxx", // address of name of the file
GENERIC_READ | GENERIC_WRITE, // access (read-write) mode
0, // share mode
&fhs, // address of security descriptor
OPEN_ALWAYS /* CREATE_ALWAYS */, // how to create
#if !defined(USE_UNBUFF) && !defined(USE_OVERLAP)
FILE_ATTRIBUTE_NORMAL
#else
0
#endif
#ifdef USE_UNBUFF
| FILE_FLAG_NO_BUFFERING
#endif
#ifdef USE_OVERLAP
| FILE_FLAG_OVERLAPPED
#endif
| FILE_FLAG_SEQUENTIAL_SCAN
, // file attributes
(HANDLE)NULL ); // handle of file with attributes to copy

if (fh == (HANDLE)NULL)
error("Create file failed\n");


#ifdef NEVER
// Try pre-allocating file by creating a file mapping
HANDLE mfh;
mfh = CreateFileMapping(
fh, // handle of file to map
&fhs, // optional security attributes
PAGE_READWRITE | SEC_COMMIT, // protection for mapping object
0, // high-order 32 bits of object size
REPS * BSZ * 2, // low-order 32 bits of object size
NULL // name of file-mapping object
);
if (mfh == (HANDLE)NULL)
error("Create map file failed, lasterror = 0x%x", GetLastError());
if (!CloseHandle(mfh))
error("Close map file failed, lasterror = 0x%x", GetLastError());
#endif

#ifdef NEVER
// Try pre-allocating file using setEOF
if (SetFilePointer(
fh, // handle of file
REPS * BSZ * 2, // number of bytes to move file pointer
NULL, // address of high-order word of distance to move
FILE_BEGIN // how to move
) != (REPS * BSZ * 2)) {
error("Seek failed");
}
if (!SetEndOfFile(
fh // handle of file whose EOF is to be set
)) {
error("Set end of file failed");
}
if (SetFilePointer(
fh, // handle of file
0, // number of bytes to move file pointer
NULL, // address of high-order word of distance to move
FILE_BEGIN // how to move
) != 0) {
error("Rewind failed");
}
#endif

////////////////////////////////////
fo = 0;
os1.OffsetHigh =
os2.OffsetHigh = 0; // Won't use over 4 Gigs
for (i = 0; i < REPS; i++) {
if (i != 0) {
if (!GetOverlappedResult(
fh, // handle of file, pipe, or communications device
&os1, // address of overlapped structure
&bwritten, // address of actual bytes count
TRUE // wait flag
) || bwritten != BSZ)
error("Wait for result 1 failed");
}

// Do some work - fill the buffer up
for (j = 0; j < BSZ; j += 1)
buf1[j] = j * j + i;
// for (j = 0; j < BSZ; j += 4)
// *((unsigned long *)&buf1[j]) = (2 * i * BSZ) + j;

os1.Offset = fo;
fo += BSZ;
if (!WriteFile(
fh, // handle of file to write to
buf1, // address of data to write to file
BSZ, // number of bytes to write
&bwritten, // address of number of bytes written
&os1 // addr. of structure needed for overlapped I/O
)) {
if (GetLastError() != ERROR_IO_PENDING)
error("Write 1 failed, lasterror = 0x%x", GetLastError());
}
#ifdef USE_OVERLAP
else
now++;
#endif

if (i != 0) {
if (!GetOverlappedResult(
fh, // handle of file, pipe, or communications device
&os2, // address of overlapped structure
&bwritten, // address of actual bytes count
TRUE // wait flag
) || bwritten != BSZ)
error("Wait for result 2 failed");
}

// Do some work - fill the buffer up
for (j = 0; j < BSZ; j += 1)
buf2[j] = j * j + i;
// for (j = 0; j < BSZ; j += 4)
// *((unsigned long *)&buf2[j]) = (2 * i + 1) * BSZ + j;

os2.Offset = fo;
fo += BSZ;
if (!WriteFile(
fh, // handle of file to write to
buf2, // address of data to write to file
BSZ, // number of bytes to write
&bwritten, // address of number of bytes written
&os2 // addr. of structure needed for overlapped I/O
)) {
if (GetLastError() != ERROR_IO_PENDING)
error("Write 2 failed, lasterror = 0x%x", GetLastError());
}
#ifdef USE_OVERLAP
else
now++;
#endif
}

if (!GetOverlappedResult(
fh, // handle of file, pipe, or communications device
&os1, // address of overlapped structure
&bwritten, // address of actual bytes count
TRUE // wait flag
) || bwritten != BSZ)
error("Wait for result 2 failed");

if (!GetOverlappedResult(
fh, // handle of file, pipe, or communications device
&os2, // address of overlapped structure
&bwritten, // address of actual bytes count
TRUE // wait flag
) || bwritten != BSZ)
error("Wait for result 1 failed");

if (!CloseHandle(fh))
error("File close failed\n");

etime = GetTickCount();

if (now != 0)
warning("Got %d Non-overlapped writes out of %d",now,REPS * 2);

fprintf(stdout,"%f Mbytes in %f secs = Rate = %f Mbytes/sec\n",
1e-6 * ((double)REPS * BSZ * 2), ((etime-stime) * 1e-3),
1e-6 * ((double)REPS * BSZ * 2)/((etime-stime) * 1e-3));

////////////////////////////
// READ Test
stime = GetTickCount();
now = 0;

fh = CreateFile(
"test.xxx", // address of name of the file
GENERIC_READ, // access (read-write) mode
0, // share mode
&fhs, // address of security descriptor
OPEN_EXISTING, // how to create
#if !defined(USE_UNBUFF) && !defined(USE_OVERLAP)
FILE_ATTRIBUTE_NORMAL
#else
0
#endif
#ifdef USE_UNBUFF
| FILE_FLAG_NO_BUFFERING
#endif
#ifdef USE_OVERLAP
| FILE_FLAG_OVERLAPPED
#endif
, // file attributes
(HANDLE)NULL ); // handle of file with attributes to copy

if (fh == (HANDLE)NULL)
error("Open file for read failed\n");

////////////////////////////////////
fo = 0;
os1.OffsetHigh =
os2.OffsetHigh = 0; // Won't use over 4 Gigs
for (i = 0; i < REPS; i++) {
if (i != 0) {
if (!GetOverlappedResult(
fh, // handle of file, pipe, or communications device
&os1, // address of overlapped structure
&bwritten, // address of actual bytes count
TRUE // wait flag
) || bwritten != BSZ)
error("Wait for result 1 failed");

// Do some work - add the buffer up
for (j = k = 0; j < BSZ; j += 1)
k += j * *((char *)&buf1[j]);
l += k;
}

os1.Offset = fo;
fo += BSZ;
if (!ReadFile(
fh, // handle of file to write to
buf1, // address of data to write to file
BSZ, // number of bytes to write
&bwritten, // address of number of bytes written
&os1 // addr. of structure needed for overlapped I/O
)) {
if (GetLastError() != ERROR_IO_PENDING)
error("Read 1 failed, lasterror = 0x%x", GetLastError());
}
#ifdef USE_OVERLAP
else
now++;
#endif

if (i != 0) {
if (!GetOverlappedResult(
fh, // handle of file, pipe, or communications device
&os2, // address of overlapped structure
&bwritten, // address of actual bytes count
TRUE // wait flag
) || bwritten != BSZ)
error("Wait for result 2 failed");

// Do some work - add the buffer up
for (j = k = 0; j < BSZ; j += 1)
k += j * *((char *)&buf2[j]);
l += k;
}
os2.Offset = fo;
fo += BSZ;
if (!ReadFile(
fh, // handle of file to write to
buf2, // address of data to write to file
BSZ, // number of bytes to write
&bwritten, // address of number of bytes written
&os2 // addr. of structure needed for overlapped I/O
)) {
if (GetLastError() != ERROR_IO_PENDING)
error("Read 2 failed, lasterror = 0x%x", GetLastError());
}
#ifdef USE_OVERLAP
else
now++;
#endif
}

if (!GetOverlappedResult(
fh, // handle of file, pipe, or communications device
&os1, // address of overlapped structure
&bwritten, // address of actual bytes count
TRUE // wait flag
) || bwritten != BSZ)
error("Wait for result 2 failed");
// Do some work - add the buffer up
for (j = k = 0; j < BSZ; j += 1)
k += j * *((char *)&buf1[j]);
l += k;

if (!GetOverlappedResult(
fh, // handle of file, pipe, or communications device
&os2, // address of overlapped structure
&bwritten, // address of actual bytes count
TRUE // wait flag
) || bwritten != BSZ)
error("Wait for result 1 failed");
// Do some work - add the buffer up
for (j = k = 0; j < BSZ; j += 1)
k += j * *((char *)&buf2[j]);
l += k;

if (!CloseHandle(fh))
error("File close failed\n");

etime = GetTickCount();

if (now != 0)
warning("Got %d Non-overlapped reads out of %d",now,REPS * 2);

fprintf(stdout,"%f Mbytes in %f secs = Rate = %f Mbytes/sec, l = %d\n",
1e-6 * ((double)REPS * BSZ * 2), ((etime-stime) * 1e-3),
1e-6 * ((double)REPS * BSZ * 2)/((etime-stime) * 1e-3),
l);

if (!VirtualFree(
buf1, // address of region of committed pages
0, // size of region
MEM_RELEASE // type of free operation
))
error("Free of buf1 failed");

if (!VirtualFree(
buf2, // address of region of committed pages
0, // size of region
MEM_RELEASE // type of free operation
))
error("Free of buf2 failed");

return 0;
}

void
error(char *fmt, ...)
{
va_list args;

fprintf(stderr,"test3: Error - ");
va_start(args, fmt);
vfprintf(stderr, fmt, args);
va_end(args);
fprintf(stderr, "\n");
exit (-1);
}

void
warning(char *fmt, ...)
{
va_list args;

fprintf(stderr,"test3: Warning - ");
va_start(args, fmt);
vfprintf(stderr, fmt, args);
va_end(args);
fprintf(stderr, "\n");
}

//////////////////////////////////////////////////////////////////

dave porter

unread,
Jun 20, 1996, 3:00:00 AM6/20/96
to

In article <4q7ion$j...@wallaby.digideas.com.au>
gra...@wallaby.digideas.com.au (Graeme Gill) wrote:

> In an attempt to get good file write performance, I have been investigating
> Windows NT overlapped I/O. Microsoft make a big song and dance about NT
> and overlapped I/O, but experiments so far indicate that it is not very
> useful when writing files.

Just an uninformed guess - but I presume that if NT can complete
the I/O request in its fast path (=copies data into cache immediately),
then the return from the call will be synchronous. This is goodness.

In other words, maybe the I/Os aren't overlapping because
NT is able to absorb the data as fast as you're throwing it.
Overlapped I/O is only needed for handling "backlogged" requests.

Ignoring the matter of whether the I/Os are actually overlapping
or not, how does the speed of writing a new file compare
with writing an existing file?


Jamie Hanrahan, Kernel Mode Systems

unread,
Jun 20, 1996, 3:00:00 AM6/20/96
to

In article <4q7ion$j...@wallaby.digideas.com.au>, gra...@wallaby.digideas.com.au (Graeme Gill) writes:
> In an attempt to get good file write performance, I have been investigating
> Windows NT overlapped I/O. Microsoft make a big song and dance about NT
> and overlapped I/O, but experiments so far indicate that it is not very
> useful when writing files.

Works for me.

> When writing to a fresh (e.g. new, or not previously written to) file, it
> appears that NT doesn't overlap writes (NTFS). When writing over an
> existing file, it does do overlapped writes. It makes no difference
> whether one use WriteFile or WriteFileEx().
>
> Is there any good reason for this, or some trick to get around this
> problem ??

Try opening the file with FILE_FLAG_NO_BUFFERING, which tells NT not
to cache it.

If the file in question is mapped into the cache, a "write" consists
of copying your buffer to the virtual address space of the cache, all
within the context of the requesting thread. (The "dirty" page in the
cache will be queued for writing within about four seconds; the
writing is done by a modified page writer thread in the system
process.)

For a zero-length file the first such attempt will incur a page fault,
but since you don't need the old file contents you'll get a page full
of zeroes from the zero page list, still all done within requesting
thread context.

In short, you aren't seeing overlapped writes because there's no place
in this path that it would ever wait for completion if you *hadn't*
asked for overlapped IO.

Anyway, try it with FILE_FLAG_NO_BUFFERING as well as
FILE_FLAG_OVERLAPPED.

> Does one have to revert to using threads to get overlapped I/O under NT ?
> (MS make a big point of recommending overlapped I/O to avoid threads!)

Actually they make a big point of using IO completion ports to cut
down on the number of threads, but they don't recommend "avoiding
threads".

--- Jamie Hanrahan, Kernel Mode Systems, San Diego CA
drivers, internals, networks, applications, and training for VMS and Windows NT
NT driver seminars: email to semi...@solsem.com, or see http://www.solsem.com
Internet: j...@cmkrnl.com (JH645) CompuServe: 74140,2055

Paul Crowley

unread,
Jun 20, 1996, 3:00:00 AM6/20/96
to

In article <4qcpas$r...@wallaby.digideas.com.au>,
gra...@wallaby.digideas.com.au says...

>
> The test code clearly indicates that performance is poorer when it
> is not overlapping I/O.
>
> : Ignoring the matter of whether the I/Os are actually overlapping


> : or not, how does the speed of writing a new file compare
> : with writing an existing file?
>

> In my benchmark, 1.8 Mbytes/sec and 3.3 Mbytes/sec respectively.
>
> Graeme Gill.

Another supposition here - from what I have read, I suspect the implementation
of unbuffered I/O (uncached) is done in a way so as to provide a forced commit
of written data. Therefore, overlap and unbuffered are mutually exclusive.

Could this be what is going on?

--
---------------------------------------***
pcro...@mcs.com - Paul Crowley
CD-ROM Productions - Get a CD-ROM with YOUR data on it
---------------------------------------***

Graeme Gill

unread,
Jun 21, 1996, 3:00:00 AM6/21/96
to

dave porter (por...@ultranet.com) wrote:
: In article <4q7ion$j...@wallaby.digideas.com.au>
: gra...@wallaby.digideas.com.au (Graeme Gill) wrote:

: Just an uninformed guess - but I presume that if NT can complete


: the I/O request in its fast path (=copies data into cache immediately),
: then the return from the call will be synchronous. This is goodness.

It shouldn't be caching anything - I'm using un-buffered I/O.
Making the calling code sit in the idle loop while the disk
operation happen is not goodness - those are wasted CPU cycles
that could be used for generating the data for the next write!

Syncronous I/O forces the serialization of operations that can be
overlapped - cpu processing of data, and the DMA operations/write
time of the disk.

: In other words, maybe the I/Os aren't overlapping because


: NT is able to absorb the data as fast as you're throwing it.
: Overlapped I/O is only needed for handling "backlogged" requests.

The test code clearly indicates that performance is poorer when it

Carl J. Appellof

unread,
Jun 24, 1996, 3:00:00 AM6/24/96
to

"George P. Dake" <da...@microdes.com> wrote:

>Also, I'm pretty sure NTFS allows for "file holes". Assume the end of
>file for a file is 2GB, but only 512 bytes were written at the beginning
>and another 512 bytes were written at the end. The rest of the file was
>never written. A file system could just allocate 1024 bytes and remember
>that the rest was unwritten. Then if an application ever asked to read
>the other ~2GB, it would just return 0's. Several UNIX file systems and
>NetWare's file system do this. Note that DOS's FAT doesn't do this; in
>DOS or Windows 3.1 setting end of file does allocate space.
>

I'm not sure, but experiments where I've done a CreateFile() followed by
SetFilePointer() and SetEndOfFile() to "preallocate" a file, then doing a
write to a random block somewhere in the middle of the file seems to
indicate that NTFS fills up the intervening blocks with 0s instead of
creating a true holey file like some flavors of UNIX do.

I'd love to be proven wrong on this one.
--
Carl Appellof
not speaking for
Digital Equipment Corporation

0 new messages