WriteFile causes massive file fragmentation -- how to prevent this??

[Jim]

I have an application written in Win32 MASM, that creates large files -
typically several hundred megabytes, and sometimes as much as 3 or 4
gigabytes. File writes are performed by calls to the Win32 function
WriteFile, and are usually in very small segments, like 40 bytes at a time.
However, when the process is finished, the files show as very fragmented
(using Norton Speed Disk). Even when the application starts out with the
disk less than 20% full and with no fragments on the disk, the files show up
as very fragmented (as much as 5,000 fragments in a file) after the
application is run. The application is being run under Windows NT and
Windows 2000.

I suspected that the heavy fragmentation may be due to the small disk
writes, but Microsoft says that Windows buffers disk writes before actually
committing them to disk, so I don't understand why that would cause the
problem. I could buffer these writes in a large buffer created by the
application before writing to disk, but that would complicate my application
significantly. Since Microsoft says Windows buffers before writing, I don't
see why that should be necessary.

I may be able to reduce fragmentation by pre-allocating a large amount of
disk space at the start of the application, using SetFilePointer and
SetEndOfFile, and truncating it down at the end of the application.

Does anybody have any other ideas on how I could solve this problem? Thanks
in advance for any information.

Jim.

[Andy Lutomirski]

Windows will buffer writes, but will AFAIK write as soon as a full segment
(512) bytes is available. This allows windows to mark that cache segment
clean, so it can be dropped at no cost in the future. Hence, unless you
write very fast (faster than the disk) and consistantly the small writes
will hurt. What's happening in any case is that each cluster is allocated
as it is needed. This means that Windows makes no attempt to find large
blocks of contiguous clusters (if it did it would actually _worsen_
free-space fragmentation, so Windows may actually make an effort to allocate
from small free-space blocks). As a result, if you have fragmented
freespace the file will be fragmented.

The solution is to SetFilePointer/SetEndOfFile to grow the file in large
chunks, or even all at once if the file size is known in advance. Also, run
Diskeeper to defrag your freespace.

Andy

"Jim" <ji...@noemailplease.com> wrote in message
news:iJ7A5.50959$XT1.7...@news5.giganews.com...

[Alex Blekhman]

"Andy Lutomirski" <Lu...@mailandnews.nospam.com> wrote in message
news:HA9A5.184$GS6....@newsread1.prod.itd.earthlink.net...

> Windows will buffer writes, but will AFAIK write as soon as a full
segment
> (512) bytes is available. This allows windows to mark that cache
segment
> clean, so it can be dropped at no cost in the future. Hence, unless
you
> write very fast (faster than the disk) and consistantly the small
writes
> will hurt. What's happening in any case is that each cluster is
allocated
> as it is needed. This means that Windows makes no attempt to find
large
> blocks of contiguous clusters (if it did it would actually _worsen_
> free-space fragmentation, so Windows may actually make an effort to
allocate
> from small free-space blocks). As a result, if you have fragmented
> freespace the file will be fragmented.
>
> The solution is to SetFilePointer/SetEndOfFile to grow the file in
large
> chunks, or even all at once if the file size is known in advance.
Also, run
> Diskeeper to defrag your freespace.

I would disagree with you about Diskeeper. This app uses defrag API
provided by system. Defrag functions can move file only in chunks = 16
cluster. If file size is not exactly aliquot 16 clusters it will
occupy rest of clusters anyway because next file will be moved at 16
cls boundary. So after free space defragmentation you'll have one big
hole at the end of the disk and many < 16 clusters holes at the
beginning. At the first write after defragmentation system will fill
this small holes because, as you correctly pointed, NTFS doesn't
search for large free hole, but rather fill first available free
space. So first file written after defrag will be extremely
fragmented. Then you'll want to run Diskeeper again and it again will
leave small holes after ends of files. And so on.
The only defragmenter for NTFS that not use defragment API is Norton
Speedisk 5.0 for NT4. Probably it does raw disk read/write. This app
performs true defragmentation. Unfortunately it's still unavailable
for 2K. Usually it's better to leave NTFS as it is (equally fragmented
all over the disk), than to use Diskeeper.