Is it possible to use TFileStream with FILE_FLAG_NO_BUFFERING flag?

[A]

To write a file to a disk without buffering you need to use:
CreateFile with flags: FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH
As noted here:
http://msdn.microsoft.com/en-us/library/aa364439%28v=vs.85%29.aspx

Is it possible to set somehow these flags when using TFileStream?

Any other solution that includes Delphi wrapper functions for
Windows API that include these flags?

[A]

I think I found my answer here:
http://stackoverflow.com/questions/787019/how-to-flush-a-tfilestream

so no answer needed I guess...

[Arivald]

W dniu 2011-09-05 17:24, A pisze:

You need explicitly TFileStream class? Or it is enough to have its
functionality? If latter one, use this:

uses
windows, consts, Classes;

type
type
TFileStreamEx = class(THandleStream)
public
constructor Create(const FileName: string; Mode: Word; flags: DWORD
= FILE_ATTRIBUTE_NORMAL);
destructor Destroy; override;
end;

constructor TFileStreamEx.Create(const FileName: string; Mode: Word;
flags: DWORD = FILE_ATTRIBUTE_NORMAL);
const
AccessMode: array[0..2] of LongWord = (
GENERIC_READ,
GENERIC_WRITE,
GENERIC_READ or GENERIC_WRITE);
ShareMode: array[0..4] of LongWord = (
0,
0,
FILE_SHARE_READ,
FILE_SHARE_WRITE,
FILE_SHARE_READ or FILE_SHARE_WRITE);

begin
if Mode = fmCreate then
begin
inherited Create(
CreateFile(
PChar(FileName),
GENERIC_READ or GENERIC_WRITE,
0,
nil,
CREATE_ALWAYS,
flags,
0
)
);
if Handle < 0 then
raise EFCreateError.CreateFmt(SFCreateError, [FileName]);
end
else
begin
inherited Create(
CreateFile(
PChar(FileName),
AccessMode[Mode and 3],
ShareMode[(Mode and $F0) shr 4],
nil,
OPEN_EXISTING,
flags,
0
)
);
if Handle < 0 then
raise EFOpenError.CreateFmt(SFOpenError, [FileName]);
end;
end;

destructor TFileStreamEx.Destroy;
begin
if Handle >= 0 then CloseHandle(THandle(Handle));
end;

///////////////////////////////////////////////////

This class basically works like TFileStream, but allow You to specify
flags to CreateFile(). If You leave "flags" param at its default value
(FILE_ATTRIBUTE_NORMAL) it will work exactly like original TFileStream.

You can use it like this:

... := TFileStreamEx.Create('filename', fmCreate, FILE_ATTRIBUTE_NORMAL
or FILE_FLAG_NO_BUFFERING or FILE_FLAG_WRITE_THROUGH).

BTW, You may also want to use FILE_ATTRIBUTE_NORMAL or
FILE_FLAG_SEQUENTIAL_SCAN hints flags to optimize performance.

Note: if some function require TFileStream, You may cast TFileStreamEx
to TFileStream. All properties and methods are shared, and objects are
binary compatible. But if function explicitly check object class at
runtime(using operator "as" for example), check will fail.

--
Arivald

[A]

thanks for this piece of code but it doesn't seem right. I had to add
RTLConsts to uses section and while compiling it still reports
"Comparison always evaluates to False" for
if (Handle < 0)
raise ...

is it a piece of larger code?

A guy on the link above says that TFileStream is already overloaded which
accepts file handle...
FS := TFileStream.Create( CreateFile( PChar(FileName), ... ,
FILE_FLAG_WRITE_THROUGH, ... ) );

but it doesn't seem to work for me?

[Jamie]

Can't you do this after each successive Tfilestream write operation

FlushFileBuffers(MyStream.Handle);

For that matter, you could create an inherited class of your own that
would call that each time a "WRITE" function was executed.

Btw, the file handle should be 0 if not valid.. you are testing for
- values here, I don't think that is correct. If there is a problem
during opening, the TFileStream.Create should return an error, also,
if you are building your own class you must override the Create
(constructor) and call the inherited first before checking the handle
other wise, you're going to get wild stuff!

Jamie

[Arivald]

W dniu 2011-09-07 23:28, A pisze:

> thanks for this piece of code but it doesn't seem right. I had to add
> RTLConsts to uses section and while compiling it still reports
> "Comparison always evaluates to False" for
> if (Handle< 0)
> raise ...
>
> is it a piece of larger code?

No. But it is for Delphi 4, where some units and variables may be a bit
different than in You version.
Which version You use?


Basically I get TFileStream class from Delphi source, and do required
modifications.

RTLConsts is required for error strings in exceptions. in My delphi they
are in unit "consts". If You put Your own strings, RTLConsts will be not
reqired.

> "Comparison always evaluates to False" for
> if (Handle< 0)
> raise ...

This error probably is related to THandleStream.Handle type being
unsigned on Your Delphi. On my Delphi it is just integer. In Delphi XE
it seems to have type THandle, which maps to unsigned 32-bit integer.

Anyway, WinAPI documentation says that CreateFile returns
INVALID_HANDLE_VALUE (which bits maps for -1 for signed, and $FFFFFFFF
for unsigned integer) if opening of file fails.

So I modify code:

if Handle <> INVALID_HANDLE_VALUE then

raise EFOpenError.CreateFmt(SFOpenError, [FileName]);
end;
end;

destructor TFileStreamEx.Destroy;
begin

if Handle <> INVALID_HANDLE_VALUE then CloseHandle(THandle(Handle));
end;

> A guy on the link above says that TFileStream is already overloaded which
> accepts file handle...
> FS := TFileStream.Create( CreateFile( PChar(FileName), ... ,
> FILE_FLAG_WRITE_THROUGH, ... ) );
>
> but it doesn't seem to work for me?

As I checked, Delphi 2006 have not this overload. Maybe they add it in
in some new version... Although online documentation did not list such
overload:
http://docwiki.embarcadero.com/VCL/en/Classes.TFileStream.Create


--
Arivald

[Arivald]

W dniu 2011-09-08 09:40, Arivald pisze:

Lack of coffe... of course it should be

if Handle = INVALID_HANDLE_VALUE then

[A]

> As I checked, Delphi 2006 have not this overload. Maybe they add it in in
> some new version... Although online documentation did not list such
> overload:
> http://docwiki.embarcadero.com/VCL/en/Classes.TFileStream.Create

It is not in the docs. But there is third overloaded version. I am using
2010 version at the moment.
In Delphi code insight just says "Handle". In C++ Builder it says
"TFileStream &" so I assume it is TFileStream handle if that is any
different than CreateFile handle.

I think it is a bit easier to use FlushFileBuffers(MyStream.Handle) rather
than fiddle with another overloaded class.
There may be difference in performance but that difference is small for my
needs (I haven't really seen any difference).

Simply FlushFileBuffers after a large enough chunk of buffer has been sent
to disk - 16 kb or more. That way it will be quite fast.

Thanks for the sample code anyway - perhaps it helps someone - I'll keep a
bookmark on google groups just in case if I need it at later time.

[Arivald]

W dniu 2011-09-08 13:52, A pisze:

>> As I checked, Delphi 2006 have not this overload. Maybe they add it in in
>> some new version... Although online documentation did not list such
>> overload:
>> http://docwiki.embarcadero.com/VCL/en/Classes.TFileStream.Create
>
> It is not in the docs. But there is third overloaded version. I am using
> 2010 version at the moment.
> In Delphi code insight just says "Handle". In C++ Builder it says

> "TFileStream&" so I assume it is TFileStream handle if that is any

> different than CreateFile handle.
>
> I think it is a bit easier to use FlushFileBuffers(MyStream.Handle) rather
> than fiddle with another overloaded class.

But it is not same functionality.

> There may be difference in performance but that difference is small for my
> needs (I haven't really seen any difference).
>
> Simply FlushFileBuffers after a large enough chunk of buffer has been sent
> to disk - 16 kb or more. That way it will be quite fast.

Calling FlushFileBuffers after 16 kb is basically useless. Windows file
cache manager use 4 kb pages, so after 16 kb write buffer should be
already scheduled to write. Calling FlushFileBuffers only stop your
program execution until it really happens, and data are really send to
disk controller.

BTW, what You need this option for?

For me, I used it only to log data in some debug builds, when
application crash may happen. This way I'm sure that all data are logged.

--
Arivald

[Jamie]

I want to know where you get this idea about pages not being written
until a 16Kb is completed? It's not even documented at $MS.

We are not talking about Disk hardware IO here, we are talking about
the OS not using its own buffering.

On Top of that, there are issues when back traversing your file
pointers, how ever, this will not happen if you are just doing contiguous
streams.

Read here.
http://msdn.microsoft.com/en-us/library/aa364439%28v=vs.85%29.aspx

On top of all that, using the optional flag "FILE_FLAG_NO_BUFFERING"
in the createfile comes with a heavy price of over head, you need to manage
your writes in sector sizes of what the IO system is on a particular
system, each one not always being the same. For this this information,
you need to use the "GetDiskFreeSpace" which gives you this information
so your app can manage this properly, which is most likely one reason
why "FlushFileBUffers" may not behave properly at times.

Jamie

[Arivald]

W dniu 2011-09-08 16:03, Jamie pisze:

I do not said it is not written until 16 kb. It may be written or not.

Basically I had a problem with log file when application dies. Not even
AV message, just process was killed. In this case I get a few random
characters at end of log file.

> We are not talking about Disk hardware IO here, we are talking about the
> OS not using its own buffering.

Yes.
But there is big performance penalty: OS can read from device only whole
blocks. So to append 100 bytes of log entry, it read 4KB block, (some
bytes of block are unused, yet drive send them to OS), update 100 bytes,
then write block back.
And bigger loss of time: wait until disk rotate once, to be able to save
block back.

Note that FILE_FLAG_WRITE_THROUGH can disable buffering at device level.
It depends only on device driver.

> On Top of that, there are issues when back traversing your file
> pointers, how ever, this will not happen if you are just doing contiguous
> streams.

Luckily for me, I was only appending ;-)

> On top of all that, using the optional flag "FILE_FLAG_NO_BUFFERING" in
> the createfile comes with a heavy price of over head,

Yes, I know. As I said, I used it only for logging, and event there it
must be enabled manually when I suspect crash and want to know last
logged entry.

> you need to manage
> your writes in sector sizes of what the IO system is on a particular
> system, each one not always being the same.

Not really... You may do it for performance, but performance was not
issue for me. But such actions defeat purpose of not buffered file (you
make buffer in app, instead of buffer in OS).

For pure performance I will use OS buffers and multithreading
(Overlapped I/O).


--
Arivald

[A]

> I want to know where you get this idea about pages not being written until
> a 16Kb is completed? It's not even documented at $MS.

It was an arbitrary number not a hard rule (I wrote "large enough chunk of
buffer"). You can use any number instead. OK, maybe it wasn't explicit
enough.
16kb is large enough not to be flushed too often. If you flush 1 byte it
will slow down terribly disk I/O.

> Read here.
> http://msdn.microsoft.com/en-us/library/aa364439%28v=vs.85%29.aspx
> On top of all that, using the optional flag "FILE_FLAG_NO_BUFFERING" in
> the createfile comes with a heavy price of over head, you need to manage
> your writes in sector sizes of what the IO system is on a particular
> system, each one not always being the same. For this this information,

Exactly the reason why I prefer now FlushFileBuffers ofer the NO_BUFFERING
flat... I don't want to mess around even further with aligning to sector
sizes and similar stuff.

> you need to use the "GetDiskFreeSpace" which gives you this information so
> your app can manage this properly, which is most likely one reason why
> "FlushFileBUffers" may not behave properly at times.

From what I could see FlushFileBuffers is not recommended only because it
may behave less efficiently if you call it too often and for every possible
write. If you call it on some buffered data it will work quite fast. If you
need to write 1000 kb of data, you can do it in blocks of 16 kb and it
should work good enough but flushing after every byte is definitely not a
good idea just like a documentation says.

[A]

> But it is not same functionality.

Yes, but I never said it is the same.

> BTW, what You need this option for?

For ensuring that the critical data hits the disk before application
continues in an event of a crash or so. Yes, it may pause application a bit
I am aware of that.
But it can be also put in a thread so it works more efficiently.

[Arivald]

W dniu 2011-09-08 16:29, A pisze:

Efficiency defeats security, even in other thread. You will wait for
thread to complete write to be sure.

Anyway in Your case just flushing buffer is enough, You did not need
unbuffered file. Writes will be reasonably fast, You will wait only
while flushing.


--
Arivald

[A]

> Efficiency defeats security, even in other thread. You will wait for
> thread to complete write to be sure.

yes but i can queue more stuff to write that way and only wait when i really
need to wait.
in my particular case this is not really needed because writes are small
(50-300 kb on average each) so can probably be flushed right after the
write.

> Anyway in Your case just flushing buffer is enough, You did not need
> unbuffered file. Writes will be reasonably fast, You will wait only while
> flushing.

exactly what i needed to see :)