std::fstream and buffer sizes
Rune Allnor
I'm pretty sure I have both asked this question before,
and recieved an answer, but I can't find it in the archives.
How does one adjust the buffer sizes for fstreams?
Rune
--
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
Bo Persson
> Hi all.
>
> I'm pretty sure I have both asked this question before,
> and recieved an answer, but I can't find it in the archives.
>
> How does one adjust the buffer sizes for fstreams?
>
You cannot request a different size, but you CAN supply you own buffer
area to the contained filebuf. See pubsetbuf in basic_streambuf.
Bo Persson
Rune Allnor
> Rune Allnor wrote:
> > Hi all.
>
> > I'm pretty sure I have both asked this question before,
> > and recieved an answer, but I can't find it in the archives.
>
> > How does one adjust the buffer sizes for fstreams?
>
> You cannot request a different size, but you CAN supply you own buffer
> area to the contained filebuf. See pubsetbuf in basic_streambuf.
Thanks.
Stroustrup isn't too clear on memory managment
responibilities.
Do I need to do anything to ensure that the original
buffer is destructed? Or does the basic_streambuf object
handle the original buffer, even if it is set to use
some other buffer supplied by me for data IO?
Rune
Bo Persson
> On 4 Jul, 08:30, "Bo Persson" <b...@gmb.dk> wrote:
>> Rune Allnor wrote:
>>> Hi all.
>>
>>> I'm pretty sure I have both asked this question before,
>>> and recieved an answer, but I can't find it in the archives.
>>
>>> How does one adjust the buffer sizes for fstreams?
>>
>> You cannot request a different size, but you CAN supply you own
>> buffer area to the contained filebuf. See pubsetbuf in
>> basic_streambuf.
>
> Thanks.
>
> Stroustrup isn't too clear on memory managment
> responibilities.
>
> Do I need to do anything to ensure that the original
> buffer is destructed? Or does the basic_streambuf object
> handle the original buffer, even if it is set to use
> some other buffer supplied by me for data IO?
>
It doesn't say that you have to to anything, so just supply a new
buffer. You will have to keep the buffer alive, of course, as long as
it is used.
Bo Persson
Howard Hinnant
> Hi all.
>
> I'm pretty sure I have both asked this question before,
> and recieved an answer, but I can't find it in the archives.
>
> How does one adjust the buffer sizes for fstreams?
In contradiction to other replies you have received, you'll need to
read your implementation's documentation. Here is what the standard
says:
Effects: If setbuf(0,0) is called on a stream before any I/O has
occurred on that stream, the stream
becomes unbuffered. Otherwise the results are implementation-defined.
“Unbuffered” means that
pbase() and pptr() always return null and output to the file should
appear as soon as possible.
I.e. the only thing you can count on from the standard is fs.rdbuf()-
>pubsetbuf(0,0).
-Howard
Rune Allnor
> On Jul 2, 9:59 pm, Rune Allnor <all...@tele.ntnu.no> wrote:
>
> > Hi all.
>
> > I'm pretty sure I have both asked this question before,
> > and recieved an answer, but I can't find it in the archives.
>
> > How does one adjust the buffer sizes for fstreams?
>
> In contradiction to other replies you have received, you'll need to
> read your implementation's documentation. Here is what the standard
> says:
>
> Effects: If setbuf(0,0) is called on a stream before any I/O has
> occurred on that stream, the stream
> becomes unbuffered. Otherwise the results are implementation-defined.
> “Unbuffered” means that
> pbase() and pptr() always return null and output to the file should
> appear as soon as possible.
>
> I.e. the only thing you can count on from the standard is fs.rdbuf()-
>
> >pubsetbuf(0,0).
Ouch!
So iostream buffer manipulations are non-portable?
Rune
Bo Persson
> On 6 Jul, 05:14, Howard Hinnant <howard.hinn...@gmail.com> wrote:
>> On Jul 2, 9:59 pm, Rune Allnor <all...@tele.ntnu.no> wrote:
>>
>>> Hi all.
>>
>>> I'm pretty sure I have both asked this question before,
>>> and recieved an answer, but I can't find it in the archives.
>>
>>> How does one adjust the buffer sizes for fstreams?
>>
>> In contradiction to other replies you have received, you'll need to
>> read your implementation's documentation. Here is what the
>> standard says:
>>
>> Effects: If setbuf(0,0) is called on a stream before any I/O has
>> occurred on that stream, the stream
>> becomes unbuffered. Otherwise the results are
>> pbase() and pptr() always return null and output to the ?le should
>> appear as soon as possible.
>>
>> I.e. the only thing you can count on from the standard is
>> fs.rdbuf()-
>>
>>> pubsetbuf(0,0).
>
> Ouch!
>
> So iostream buffer manipulations are non-portable?
>
Well, how do you intend to select an "optimal" buffer size in a
system-independent way? This has to be identified as a problem, and
tuned for each platform anyway.
Like Howard says, in the general case the language standard makes no
guarantees, so you will have to check that you are using an underlying
device that will make use of the buffer.
One implementation
(http://msdn.microsoft.com/en-us/library/2bzc2y2z%28VS.80%29.aspx)
describes this as:
"...offer the array of _Count bytes beginning at _Buffer as a buffer
for the stream".
If the stream rejects the buffer, you are out of luck.
If the fstream interfaces to a file on some of the more widely used
operating systems, it will probably not reject the buffer. You still
don't know what size is universally optimal though - and why that
isn't the one already used.
Bo Persson
Rune Allnor
True, the numbers are system-dependent, but it seems
the C++ interface to the stream buffers (or lack thereof)
means I as programmer have no safe way of manipulating
the buffer size.
> If the fstream interfaces to a file on some of the more widely used
> operating systems, it will probably not reject the buffer. You still
> don't know what size is universally optimal though - and why that
> isn't the one already used.
The applications I have in mind will need to access
several GBytes of data, fast. I will need buffers
in the order of tens, maybe hundreds, of MBytes,
or disk access during file IO ops will slow the
program down too much.
Rune
George Neuner
wrote:
>The applications I have in mind will need to access
>several GBytes of data, fast. I will need [file] buffers
>in the order of tens, maybe hundreds, of MBytes,
>or disk access during file IO ops will slow the
>program down too much.
MB file buffers are probably impossible - I've never met a file system
implementation that allowed a buffer that big.
If the data format is amenable you might be better off memory mapping
the files and accessing them directly as arrays.
Otherwise I would suggest you get familiar with non-blocking I/O and
read in the new data chunks while you process the current one.
>Rune
George