Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
fstream Buffers
93 views
Skip to first unread message
Mike Copeland
May 17, 2012, 4:34:40 PM5/17/12
to
I have the following routine that determines the time required to
copy a large text file to another. I'm comparing this code to a class I
wrote (too large to post here) that does text file I/o with explicit
buffer size designation (4096 character buffers). The code below runs
12-13 times slower, although doing exactly the same work.
I tried to google "fstream buffer" with great confusion and little
understanding of what I'd need to do to change the code below to use
explicit buffers: hoping to get a better performance match to my old
processing.
I'd like to use C++ strings and I/o streams if possible. but I wonder
if the overhead to use these functions can match older C processing
(fgets, setbuf, etc.) at all. The following seems probable:
1. The overhead to size and allocate/release a string variable for each
logical record will always be slower than use of an explicit char
[some_size] for the text string being read from the input file.
2. The "<<" operators are probably much slower than "fputs".
Are my assumptions valid? Are there options or function calls I can
use to specify fstream buffers (and how are they used?)? Please advise.
TIA
//////////////////// Code ////////////////////
string line;
fstream fVar1, fVar2;
fVar1.open("pat12.n00", fstream::in);
if(fVar1.is_open())
{
fVar2.open("test.txt", fstream::out);
while(getline(fVar1, line))
{
fVar2 << line << endl;
}
fVar1.close(), fVar2.close();
}
else cout << "Unable to open file";
//////////////////////////////////////////////
copy a large text file to another. I'm comparing this code to a class I
wrote (too large to post here) that does text file I/o with explicit
buffer size designation (4096 character buffers). The code below runs
12-13 times slower, although doing exactly the same work.
I tried to google "fstream buffer" with great confusion and little
understanding of what I'd need to do to change the code below to use
explicit buffers: hoping to get a better performance match to my old
processing.
I'd like to use C++ strings and I/o streams if possible. but I wonder
if the overhead to use these functions can match older C processing
(fgets, setbuf, etc.) at all. The following seems probable:
1. The overhead to size and allocate/release a string variable for each
logical record will always be slower than use of an explicit char
[some_size] for the text string being read from the input file.
2. The "<<" operators are probably much slower than "fputs".
Are my assumptions valid? Are there options or function calls I can
use to specify fstream buffers (and how are they used?)? Please advise.
TIA
//////////////////// Code ////////////////////
string line;
fstream fVar1, fVar2;
fVar1.open("pat12.n00", fstream::in);
if(fVar1.is_open())
{
fVar2.open("test.txt", fstream::out);
while(getline(fVar1, line))
{
fVar2 << line << endl;
}
fVar1.close(), fVar2.close();
}
else cout << "Unable to open file";
//////////////////////////////////////////////
Joshua Maurice
May 17, 2012, 4:40:57 PM5/17/12
to
What compiler?
cart...@gmail.com
May 17, 2012, 4:52:52 PM5/17/12
to
On Thursday, May 17, 2012 3:34:40 PM UTC-5, Mike Copeland wrote:
> while(getline(fVar1, line))
> {
> fVar2 << line << endl;
> }
> fVar1.close(), fVar2.close();
std::endl sends a '\n' to the stream and flushes it. flushing after each and every line of output will likely kill performance.
> while(getline(fVar1, line))
> {
> fVar2 << line << endl;
> }
> fVar1.close(), fVar2.close();
Joshua Maurice
May 17, 2012, 5:09:05 PM5/17/12
to
I usually won't look at code in such questions until I get which
compiler and which compiler options. It's always annoying to debug
performance problems in code when the code was being run without
optimizations.
Jorgen Grahn
May 17, 2012, 5:19:48 PM5/17/12
to
On Thu, 2012-05-17, Mike Copeland wrote:
> I have the following routine that determines the time required to
> copy a large text file to another. I'm comparing this code to a class I
> wrote (too large to post here) that does text file I/o with explicit
> buffer size designation (4096 character buffers). The code below runs
> 12-13 times slower, although doing exactly the same work.
...
> I have the following routine that determines the time required to
> copy a large text file to another. I'm comparing this code to a class I
> wrote (too large to post here) that does text file I/o with explicit
> buffer size designation (4096 character buffers). The code below runs
> 12-13 times slower, although doing exactly the same work.
> string line;
> fstream fVar1, fVar2;
> fVar1.open("pat12.n00", fstream::in);
> if(fVar1.is_open())
> {
> fVar2.open("test.txt", fstream::out);
> while(getline(fVar1, line))
> {
> fVar2 << line << endl;
You're flushing the stream for every line. That almost certainly is
the cause of your 12x slowdown. Fix that and measure again.
Remember, the C++ line terminator is the same as in C, namely '\n'.
/Jorgen
--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
Scott Lurndal
May 17, 2012, 5:28:53 PM5/17/12
to
mrc...@cox.net (Mike Copeland) writes:
> I have the following routine that determines the time required to
>copy a large text file to another. I'm comparing this code to a class I
>wrote (too large to post here) that does text file I/o with explicit
>buffer size designation (4096 character buffers). The code below runs
>12-13 times slower, although doing exactly the same work.
Can you invoke the operating system utility that is optimized for
> I have the following routine that determines the time required to
>copy a large text file to another. I'm comparing this code to a class I
>wrote (too large to post here) that does text file I/o with explicit
>buffer size designation (4096 character buffers). The code below runs
>12-13 times slower, although doing exactly the same work.
copying files (e.g. system("cp pat12.n00 test.text");)?
You'll never be able to approach optimal performance using buffering
in the application (either via C++ streams or using vanilla <stdio.h>).
Using pread/pwrite with 1MB transfers is probably the best you'll be able
to do. And you may want to open the files with O_DIRECT to avoid the kernel
buffer/file cache, which may help avoid filling the OS file cache with
"access once" data. On POSIX systems anyway. Windows may have similar
facilities.
System copy utilities will generally preserve 'holes' in the file as well, where
a less sophisticated copy algorithm will fill the holes (and the resulting output
file will consume more disk space). For example, if a file had been created
by writing 10 bytes at the start, and 100 bytes at offset 1,024,000 the file
would appear to be 1,024,100 bytes in length, but will take only two file system blocks
in real storage. Copying that file using your method will result in the destination
file using many more file system blocks (since read will return all zeros for the unwritten
data in the original file and you'll blindly write them to the destination file).
scott
Scott Lurndal
May 17, 2012, 5:32:41 PM5/17/12
to
once in the kernel file/buffer cache (at one or more page (typ. 4096 byte) granularity)
once in the streams (or stdio) buffer (at streams buffer/setvbuf (1024/2408 byte) granularity)
once in the application buffer. (at line size granularity)
That's a lot of cpu cycles wasted moving data between buffers, and more system calls than is
necessary.
scott
Jorgen Grahn
May 17, 2012, 6:01:19 PM5/17/12
to
On Thu, 2012-05-17, Scott Lurndal wrote:
> cart...@gmail.com writes:
>>On Thursday, May 17, 2012 3:34:40 PM UTC-5, Mike Copeland wrote:
>>> while(getline(fVar1, line))
>>> {
>>> fVar2 << line << endl;
>>> }
>>> fVar1.close(), fVar2.close();
>>
>>std::endl sends a '\n' to the stream and flushes it. flushing after
>>each and every line of output will likely kill performance.
>
> Not to mention buffering the data _three_ times:
...
> cart...@gmail.com writes:
>>On Thursday, May 17, 2012 3:34:40 PM UTC-5, Mike Copeland wrote:
>>> while(getline(fVar1, line))
>>> {
>>> fVar2 << line << endl;
>>> }
>>> fVar1.close(), fVar2.close();
>>
>>std::endl sends a '\n' to the stream and flushes it. flushing after
>>each and every line of output will likely kill performance.
>
> Not to mention buffering the data _three_ times:
> That's a lot of cpu cycles wasted moving data between buffers,
> and more system calls than is
> necessary.
Probably insignificant compared to the constant flushing, though.
> and more system calls than is
> necessary.
Mike Copeland
May 17, 2012, 6:13:39 PM5/17/12
to
In article <a328e357-9312-4971-99ba-1b18c2f21d86
@s9g2000pbc.googlegroups.com>, joshua...@gmail.com says...
> > =A0 =A0I have the following routine that determines the time required to
> > copy a large text file to another. =A0I'm comparing this code to a class =
> > =A01. The overhead to size and allocate/release a string variable for each
> > Are my assumptions valid? =A0Are there options or function calls I
> > fstream =A0fVar1, fVar2;
> > =A0 while(getline(fVar1, line))
> > =A0 {
> > =A0 =A0 fVar2 << line << endl;
> > =A0 }
> > =A0 fVar1.close(), fVar2.close();}
and/or efficiencies are involved (don't know how to find out...), and
it's VS6.0 (which everyone here condemns, but it's all I have.
The real point is that this code is in linear sequence in a program,
the old calls followed by the posted code. I also reversed the order of
the tests, to assure that there's not a "caching" advantage that the
second process has versus the first. Whether the above code is first or
second, the time it takes is 12-13 times as long as the old
code/technique (fgets/fputs/setbuf). 8<{{
@s9g2000pbc.googlegroups.com>, joshua...@gmail.com says...
> > =A0 =A0I have the following routine that determines the time required to
> > copy a large text file to another. =A0I'm comparing this code to a class =
> I
> > wrote (too large to post here) that does text file I/o with explicit
> > buffer size designation (4096 character buffers). =A0The code below runs
> > wrote (too large to post here) that does text file I/o with explicit
> > 12-13 times slower, although doing exactly the same work.
> > =A0 =A0I tried to google "fstream buffer" with great confusion and little
> > understanding of what I'd need to do to change the code below to use
> > explicit buffers: hoping to get a better performance match to my old
> > processing.
> > =A0 =A0I'd like to use C++ strings and I/o streams if possible. but I won=
> > explicit buffers: hoping to get a better performance match to my old
> > processing.
> der
> > if the overhead to use these functions can match older C processing
> > (fgets, setbuf, etc.) at all. =A0The following seems probable:
> > if the overhead to use these functions can match older C processing
> > =A01. The overhead to size and allocate/release a string variable for each
> > logical record will always be slower than use of an explicit char
> > [some_size] for the text string being read from the input file.
> > =A02. The "<<" operators are probably much slower than "fputs".
> > [some_size] for the text string being read from the input file.
> > Are my assumptions valid? =A0Are there options or function calls I
> can
> > use to specify fstream buffers (and how are they used?)? Please advise.
> > TIA
> >
> > //////////////////// Code ////////////////////
> > string =A0 line;
> > use to specify fstream buffers (and how are they used?)? Please advise.
> > TIA
> >
> > //////////////////// Code ////////////////////
> > fstream =A0fVar1, fVar2;
> > fVar1.open("pat12.n00", fstream::in);
> > if(fVar1.is_open())
> > {
> > =A0 fVar2.open("test.txt", fstream::out);
> > if(fVar1.is_open())
> > {
> > =A0 while(getline(fVar1, line))
> > =A0 {
> > =A0 =A0 fVar2 << line << endl;
> > =A0 }
> > =A0 fVar1.close(), fVar2.close();}
> >
> > else cout << "Unable to open file";
> > //////////////////////////////////////////////
>
> First silly question: what compiler flags? Are optimizations enabled?
> What compiler?
>
Unfortunately, a "silly answer": I don't know what compiler flags
> > else cout << "Unable to open file";
> > //////////////////////////////////////////////
>
> First silly question: what compiler flags? Are optimizations enabled?
> What compiler?
>
and/or efficiencies are involved (don't know how to find out...), and
it's VS6.0 (which everyone here condemns, but it's all I have.
The real point is that this code is in linear sequence in a program,
the old calls followed by the posted code. I also reversed the order of
the tests, to assure that there's not a "caching" advantage that the
second process has versus the first. Whether the above code is first or
second, the time it takes is 12-13 times as long as the old
code/technique (fgets/fputs/setbuf). 8<{{
Mike Copeland
May 17, 2012, 6:16:37 PM5/17/12
to
In article <d6c93c71-68ed-42a0...@googlegroups.com>,
cart...@gmail.com says...
line prior to the "<<" operator. 8<}}
cart...@gmail.com says...
> > while(getline(fVar1, line))
> > {
> > fVar2 << line << endl;
> > }
> > fVar1.close(), fVar2.close();
>
> std::endl sends a '\n' to the stream and flushes it. flushing after each and every line of output will likely kill performance.
>
Okay, that's useful. Guess I'll have to append a '\n' to the data
> > {
> > fVar2 << line << endl;
> > }
> > fVar1.close(), fVar2.close();
>
> std::endl sends a '\n' to the stream and flushes it. flushing after each and every line of output will likely kill performance.
>
line prior to the "<<" operator. 8<}}
Mike Copeland
May 17, 2012, 6:24:27 PM5/17/12
to
line +='\n';
fVar2 << line;
Mike Copeland
May 17, 2012, 6:28:08 PM5/17/12
to
In article <slrnjraqrk.r...@frailea.sa.invalid>,
grahn...@snipabacken.se says...
no effect. I.e.
while(getline(fVar1, line))
{
fgets/fputs/setbuf. 8<{{
grahn...@snipabacken.se says...
> > I have the following routine that determines the time required to
> > copy a large text file to another. I'm comparing this code to a class I
> > wrote (too large to post here) that does text file I/o with explicit
> > buffer size designation (4096 character buffers). The code below runs
> > 12-13 times slower, although doing exactly the same work.
> ...
>
> > string line;
> > fstream fVar1, fVar2;
>
> Not very good names. 'src' and 'dest' would be better.
>
> > fVar1.open("pat12.n00", fstream::in);
> > if(fVar1.is_open())
> > {
> > fVar2.open("test.txt", fstream::out);
> > while(getline(fVar1, line))
> > {
> > fVar2 << line << endl;
> ^^^^
> You're flushing the stream for every line. That almost certainly is
> the cause of your 12x slowdown. Fix that and measure again.
>
> Remember, the C++ line terminator is the same as in C, namely '\n'.
>
And doing so (appending the '\n' to "line" prior to writing it) has
> > copy a large text file to another. I'm comparing this code to a class I
> > wrote (too large to post here) that does text file I/o with explicit
> > buffer size designation (4096 character buffers). The code below runs
> > 12-13 times slower, although doing exactly the same work.
> ...
>
> > string line;
> > fstream fVar1, fVar2;
>
> Not very good names. 'src' and 'dest' would be better.
>
> > fVar1.open("pat12.n00", fstream::in);
> > if(fVar1.is_open())
> > {
> > fVar2.open("test.txt", fstream::out);
> > while(getline(fVar1, line))
> > {
> > fVar2 << line << endl;
> ^^^^
> You're flushing the stream for every line. That almost certainly is
> the cause of your 12x slowdown. Fix that and measure again.
>
> Remember, the C++ line terminator is the same as in C, namely '\n'.
>
no effect. I.e.
while(getline(fVar1, line))
{
line +='\n';
fVar2 << line;
}
Same 12x-13x performance degradation with this code versus
fVar2 << line;
}
fgets/fputs/setbuf. 8<{{
Mike Copeland
May 17, 2012, 6:38:11 PM5/17/12
to
In article <p4etr.3$lT...@news.usenetserver.com>, sc...@slp53.sl.home
says...
text file I/o and am trying to convert fgets/fputs/setbuf logic (with C-
style strings) to fstream/string processing. My first tests of such
modifications seemed to show great slowdown; I wrote some simple "file
copy" testing to see if that's so; I found an enormous performance
"hit" with the new logic.
My guess was that the absence of a user-defined I/o buffer was the
major source of the degradation, and I sought a way to achieve the
"setbuf" functionality with something that fstream might provide.
Searching the Web for information hasn't led to an answer, so I asked in
this NG.
I don't much care about actual "file copying", per se.
My goal here is to (greatly) improve the performance of
fstream/string usage in applications I write, as it's a deal-killer for
now... 8<{{
says...
> Can you invoke the operating system utility that is optimized for
> copying files (e.g. system("cp pat12.n00 test.text");)?
>
> You'll never be able to approach optimal performance using buffering
> in the application (either via C++ streams or using vanilla <stdio.h>).
That's not the point here: I'm writing applications that do a lot of
> copying files (e.g. system("cp pat12.n00 test.text");)?
>
> You'll never be able to approach optimal performance using buffering
> in the application (either via C++ streams or using vanilla <stdio.h>).
text file I/o and am trying to convert fgets/fputs/setbuf logic (with C-
style strings) to fstream/string processing. My first tests of such
modifications seemed to show great slowdown; I wrote some simple "file
copy" testing to see if that's so; I found an enormous performance
"hit" with the new logic.
My guess was that the absence of a user-defined I/o buffer was the
major source of the degradation, and I sought a way to achieve the
"setbuf" functionality with something that fstream might provide.
Searching the Web for information hasn't led to an answer, so I asked in
this NG.
I don't much care about actual "file copying", per se.
My goal here is to (greatly) improve the performance of
fstream/string usage in applications I write, as it's a deal-killer for
now... 8<{{
Joshua Maurice
May 17, 2012, 6:39:57 PM5/17/12
to
On May 17, 3:13 pm, mrc2...@cox.net (Mike Copeland) wrote:
> In article <a328e357-9312-4971-99ba-1b18c2f21d86
> @s9g2000pbc.googlegroups.com>, joshuamaur...@gmail.com says...
> In article <a328e357-9312-4971-99ba-1b18c2f21d86
> > First silly question: what compiler flags? Are optimizations enabled?
> > What compiler?
>
> Unfortunately, a "silly answer": I don't know what compiler flags
> and/or efficiencies are involved (don't know how to find out...),
You need to learn your tools. Determine if it's an optimized build or
> > What compiler?
>
> Unfortunately, a "silly answer": I don't know what compiler flags
> and/or efficiencies are involved (don't know how to find out...),
not, and get back to us. If you're using default settings, there's
probably a toolbar with a couple dropdowns, one for "config" and one
for "platform". Currently, the toolbar in my IDE reads; "Debug"
"Win32". Change "Debug" to "Release", compile, run, and see if that
helps.
> and
> it's VS6.0 (which everyone here condemns, but it's all I have.
express versions at least, and AFAIK they're all you need unless
you're doing some microsoft specific coding (beyond the basic win32
API), and I know nothing about such things. It's sufficient for my
uses.
> The real point is that this code is in linear sequence in a program,
> the old calls followed by the posted code. I also reversed the order of
> the tests, to assure that there's not a "caching" advantage that the
> second process has versus the first. Whether the above code is first or
> second, the time it takes is 12-13 times as long as the old
> code/technique (fgets/fputs/setbuf). 8<{{
general, has its efficiency depend heavily on good optimization,
including inline-expansion of functions. If you don't have the basic
optimization flags on, chances are that the C++ standard library
fstream solution will be much slower.
In addition to what was noted else-thread: std::endl writes a '\n' and
does a flush. You probably don't want to flush that often. You
probably want to flush only at the end.
Joshua Maurice
May 17, 2012, 6:43:29 PM5/17/12
to
On May 17, 3:24 pm, mrc2...@cox.net (Mike Copeland) wrote:
> > In article <d6c93c71-68ed-42a0...@googlegroups.com>,
> > carte...@gmail.com says...
fVar2 << line << '\n';
Also, fstream exposes a read() member function. You can do the same
fixed-size buffer approach, and that may get you better results. As
is, you're reading in potentially very small lines, resulting in more
work than necessary.
Also, are you doing a text file copy or another kind of file? Look up
std::ios::binary if you don't want garbled binary files.
http://cplusplus.com/reference/iostream/ios_base/openmode/
> > In article <d6c93c71-68ed-42a0...@googlegroups.com>,
> > carte...@gmail.com says...
> > > std::endl sends a '\n' to the stream and flushes it. flushing after each and every line of output will likely kill performance.
>
> > Okay, that's useful. Guess I'll have to append a '\n' to the data
> > line prior to the "<<" operator. 8<}}
>
> Doing so has no effect whatsoever; code follows:
>
> line +='\n';
> fVar2 << line;
Instead, for this context I'd suggest:
>
> > Okay, that's useful. Guess I'll have to append a '\n' to the data
> > line prior to the "<<" operator. 8<}}
>
> Doing so has no effect whatsoever; code follows:
>
> line +='\n';
> fVar2 << line;
fVar2 << line << '\n';
Also, fstream exposes a read() member function. You can do the same
fixed-size buffer approach, and that may get you better results. As
is, you're reading in potentially very small lines, resulting in more
work than necessary.
Also, are you doing a text file copy or another kind of file? Look up
std::ios::binary if you don't want garbled binary files.
http://cplusplus.com/reference/iostream/ios_base/openmode/
Jorgen Grahn
May 17, 2012, 7:03:00 PM5/17/12
to
On Thu, 2012-05-17, Mike Copeland wrote:
fVar2 << line << endl;
./filecopy 0.89s user 2.81s system 98% cpu 3.774 total
./filecopy 0.78s user 2.77s system 97% cpu 3.623 total
./filecopy 0.68s user 2.68s system 98% cpu 3.421 total
fVar2 << line << '\n';
./filecopy 0.36s user 0.19s system 97% cpu 0.565 total
./filecopy 0.35s user 0.18s system 70% cpu 0.750 total
./filecopy 0.34s user 0.22s system 56% cpu 0.996 total
line +='\n'; fVar2 << line;
./filecopy 0.32s user 0.15s system 95% cpu 0.488 total
./filecopy 0.31s user 0.18s system 68% cpu 0.711 total
cp a b # standard Unix tool
cp a b 0.00s user 0.16s system 96% cpu 0.166 total
cp a b 0.00s user 0.15s system 95% cpu 0.155 total
cp a b 0.00s user 0.16s system 94% cpu 0.166 total
The version with the flushing is wedged in a myriad system calls. The
others read and write (on my system) 8 kB at a time. 'cp' does the
same system calls, but 32 kB at a time. It doesn't try to find lines
either, so it's not surprising it's quite a bit faster. You normally
read line-by-line because you want to parse the text line by line.
If you're not seeing any difference, perhaps it's the very old
compiler you're using (mentioned elsethread) or with Windows. I have
no experience with either of them.
Scott Lurndal
May 17, 2012, 7:10:31 PM5/17/12
to
mrc...@cox.net (Mike Copeland) writes:
>In article <p4etr.3$lT...@news.usenetserver.com>, sc...@slp53.sl.home
>says...
>> Can you invoke the operating system utility that is optimized for
>> copying files (e.g. system("cp pat12.n00 test.text");)?
>>
>> You'll never be able to approach optimal performance using buffering
>> in the application (either via C++ streams or using vanilla <stdio.h>).
>
> That's not the point here: I'm writing applications that do a lot of
>text file I/o and am trying to convert fgets/fputs/setbuf logic (with C-
>style strings) to fstream/string processing. My first tests of such
>modifications seemed to show great slowdown; I wrote some simple "file
>copy" testing to see if that's so; I found an enormous performance
>"hit" with the new logic.
> My guess was that the absence of a user-defined I/o buffer was the
>major source of the degradation, and I sought a way to achieve the
>"setbuf" functionality with something that fstream might provide.
Look at the public methods of the filebuf object in the fstream to set
>In article <p4etr.3$lT...@news.usenetserver.com>, sc...@slp53.sl.home
>says...
>> Can you invoke the operating system utility that is optimized for
>> copying files (e.g. system("cp pat12.n00 test.text");)?
>>
>> You'll never be able to approach optimal performance using buffering
>> in the application (either via C++ streams or using vanilla <stdio.h>).
>
> That's not the point here: I'm writing applications that do a lot of
>text file I/o and am trying to convert fgets/fputs/setbuf logic (with C-
>style strings) to fstream/string processing. My first tests of such
>modifications seemed to show great slowdown; I wrote some simple "file
>copy" testing to see if that's so; I found an enormous performance
>"hit" with the new logic.
> My guess was that the absence of a user-defined I/o buffer was the
>major source of the degradation, and I sought a way to achieve the
>"setbuf" functionality with something that fstream might provide.
you own buffer.
scott
Robert Wessel
May 17, 2012, 10:08:28 PM5/17/12
to
On 17 May 2012 21:28:53 GMT, sc...@slp53.sl.home (Scott Lurndal)
wrote:
If he was going to do something that system dependent, it would be far
better to just use the Windows CopyFile() API.
wrote:
better to just use the Windows CopyFile() API.
Dombo
May 18, 2012, 6:07:38 AM5/18/12
to
Op 18-May-12 0:13, Mike Copeland schreef:
> Unfortunately, a "silly answer": I don't know what compiler flags
> and/or efficiencies are involved (don't know how to find out...), and
> it's VS6.0 (which everyone here condemns, but it's all I have.
I vaguely remember there was an issue with the iostream implementation
that came with Visual Studio 6.0 which made it very slow. I believe it
had to do something with constantly syncing with the stdio, and that it
could be disabled.
> Unfortunately, a "silly answer": I don't know what compiler flags
> and/or efficiencies are involved (don't know how to find out...), and
> it's VS6.0 (which everyone here condemns, but it's all I have.
that came with Visual Studio 6.0 which made it very slow. I believe it
had to do something with constantly syncing with the stdio, and that it
could be disabled.
Adam Skutt
May 18, 2012, 8:49:58 AM5/18/12
to
On May 17, 5:32 pm, sc...@slp53.sl.home (Scott Lurndal) wrote:
>
> Not to mention buffering the data _three_ times:
>
> once in the kernel file/buffer cache (at one or more page (typ. 4096 byte) granularity)
You're done once the data reaches here. There's no guaranteed way to
>
> Not to mention buffering the data _three_ times:
>
> once in the kernel file/buffer cache (at one or more page (typ. 4096 byte) granularity)
bypass this caching, nor any value in doing so for most applications
(certainly any application using fstream). We can pretend it does not
exist.
> once in the streams (or stdio) buffer (at streams buffer/setvbuf (1024/2408 byte) granularity)
> once in the application buffer. (at line size granularity)
>
> That's a lot of cpu cycles wasted moving data between buffers, and more system calls than is
> necessary.
compared to making system calls. However, if more system calls than
necessary are occurring, it's a bug in iostream implementation, not in
his code. I suspect while not strictly optimal, it's probably close
enough.
Adam
Adam Skutt
May 18, 2012, 9:07:05 AM5/18/12
to
On May 17, 5:28 pm, sc...@slp53.sl.home (Scott Lurndal) wrote:
the system() function securely--in a few cases it simply cannot be
used securely. Secondly, for all the effort involved, you could just
copy the code from a BSD implementation or something similar.
>
> You'll never be able to approach optimal performance using buffering
> in the application (either via C++ streams or using vanilla <stdio.h>).
Actually, one can, but it depends on the iostream implementation.
>
> Using pread/pwrite with 1MB transfers is probably the best you'll be able
> to do.
I'm not sure why you think 1MB [sic] is the right transfer size. I'm
not aware of any cp(1) implementation that uses a buffer that large,
because it's pointless.
> And you may want to open the files with O_DIRECT to avoid the kernel
> buffer/file cache, which may help avoid filling the OS file cache with
> "access once" data. On POSIX systems anyway.
Nope. There's no guarantee that it does any of those things, and it
has terrible semantics for correct usage. Again, why are you
recommending things that cp(1) doesn't do?
> System copy utilities will generally preserve 'holes' in the file
as well, where
> a less sophisticated copy algorithm will fill the holes (and the resulting output
> file will consume more disk space).
This is not relevant here and plainly apparent from the OP.
Adam
> mrc2...@cox.net (Mike Copeland) writes:
> > I have the following routine that determines the time required to
> >copy a large text file to another. I'm comparing this code to a class I
> >wrote (too large to post here) that does text file I/o with explicit
> >buffer size designation (4096 character buffers). The code below runs
> >12-13 times slower, although doing exactly the same work.
>
> Can you invoke the operating system utility that is optimized for
> copying files (e.g. system("cp pat12.n00 test.text");)?
Please don't do this. For starters, it's incredibly difficult to use
> > I have the following routine that determines the time required to
> >copy a large text file to another. I'm comparing this code to a class I
> >wrote (too large to post here) that does text file I/o with explicit
> >buffer size designation (4096 character buffers). The code below runs
> >12-13 times slower, although doing exactly the same work.
>
> Can you invoke the operating system utility that is optimized for
> copying files (e.g. system("cp pat12.n00 test.text");)?
the system() function securely--in a few cases it simply cannot be
used securely. Secondly, for all the effort involved, you could just
copy the code from a BSD implementation or something similar.
>
> You'll never be able to approach optimal performance using buffering
> in the application (either via C++ streams or using vanilla <stdio.h>).
>
> Using pread/pwrite with 1MB transfers is probably the best you'll be able
> to do.
not aware of any cp(1) implementation that uses a buffer that large,
because it's pointless.
> And you may want to open the files with O_DIRECT to avoid the kernel
> buffer/file cache, which may help avoid filling the OS file cache with
> "access once" data. On POSIX systems anyway.
has terrible semantics for correct usage. Again, why are you
recommending things that cp(1) doesn't do?
> System copy utilities will generally preserve 'holes' in the file
as well, where
> a less sophisticated copy algorithm will fill the holes (and the resulting output
> file will consume more disk space).
Adam
Scott Lurndal
May 18, 2012, 1:24:03 PM5/18/12
to
Adam Skutt <ask...@gmail.com> writes:
>> Using pread/pwrite with 1MB transfers is probably the best you'll be able
>> to do.
>
>I'm not sure why you think 1MB [sic] is the right transfer size. I'm
>not aware of any cp(1) implementation that uses a buffer that large,
>because it's pointless.
>
Twenty years of measuring and optimizing disk performance, particularly
>> Using pread/pwrite with 1MB transfers is probably the best you'll be able
>> to do.
>
>I'm not sure why you think 1MB [sic] is the right transfer size. I'm
>not aware of any cp(1) implementation that uses a buffer that large,
>because it's pointless.
>
on large Oracle installations. 1MB (no sic) has been measured to be
good, given quality host bus adapter sg support.
>>=A0And you may want to open the files with O_DIRECT to avoid the kernel
>> buffer/file cache, which may help avoid filling the OS file cache with
>> "access once" data. =A0On POSIX =A0systems anyway.
>
>Nope. There's no guarantee that it does any of those things, and it
>has terrible semantics for correct usage. Again, why are you
>recommending things that cp(1) doesn't do?
Optimal performance, of course. cp is hardly optimal. dd(1) or xdd(1)
>Nope. There's no guarantee that it does any of those things, and it
>has terrible semantics for correct usage. Again, why are you
>recommending things that cp(1) doesn't do?
are much better from a performance standpoint.
O_DIRECT is guaranteed to DMA the data from the device into the
application buffer, completely bypassing any kernel or glibc buffers.
Prior to O_DIRECT, the traditional unix raw disk devices had the same semantic.
Linux does not support traditional raw unix disk devices.
scott
Adam Skutt
May 18, 2012, 2:33:17 PM5/18/12
to
On May 18, 1:24 pm, sc...@slp53.sl.home (Scott Lurndal) wrote:
> Adam Skutt <ask...@gmail.com> writes:
> >> Using pread/pwrite with 1MB transfers is probably the best you'll be able
> >> to do.
>
> >I'm not sure why you think 1MB [sic] is the right transfer size. I'm
> >not aware of any cp(1) implementation that uses a buffer that large,
> >because it's pointless.
>
> Twenty years of measuring and optimizing disk performance, particularly
> on large Oracle installations.
I'm not sure why you think that is remotely relevant here. The OP is
> Adam Skutt <ask...@gmail.com> writes:
> >> Using pread/pwrite with 1MB transfers is probably the best you'll be able
> >> to do.
>
> >I'm not sure why you think 1MB [sic] is the right transfer size. I'm
> >not aware of any cp(1) implementation that uses a buffer that large,
> >because it's pointless.
>
> Twenty years of measuring and optimizing disk performance, particularly
> on large Oracle installations.
plainly not writing a part of, nor a replacement for, Oracle.
> >>=A0And you may want to open the files with O_DIRECT to avoid the kernel
> >> buffer/file cache, which may help avoid filling the OS file cache with
> >> "access once" data. =A0On POSIX =A0systems anyway.
>
> >Nope. There's no guarantee that it does any of those things, and it
> >has terrible semantics for correct usage. Again, why are you
> >recommending things that cp(1) doesn't do?
>
> Optimal performance, of course. cp is hardly optimal. dd(1) or xdd(1)
> are much better from a performance standpoint.
manipulating text, so optimal performance in the I/O stack is very
likely to be pointless anyway. His largest source of overhead is
likely to be in whatever processing his application is actually
doing. Your suggestions are both factually incorrect and
irrelevant. dd(1) also doesn't do direct I/O unless asked, FWIW.
>
> O_DIRECT is guaranteed to DMA the data from the device into the
> application buffer, completely bypassing any kernel or glibc buffers.
Nope. All it does is disable any _caching_ done by the kernel, if
> O_DIRECT is guaranteed to DMA the data from the device into the
> application buffer, completely bypassing any kernel or glibc buffers.
possible. On modern Linux, that means that I/O skips the page cache
for the file data only. It doesn't necessarily result in direct DMA
because that's simply not possible on some systems. If the I/O
requires bounce buffers, then there still will be a copy operation in
the kernel. If the file is open anywhere else, then you still get
manipulate of the page cache in order to guarantee cache coherency.
If the filesystem needs to do some magic to ensure consistency (e.g.,
block-level checksums, ordered writes) then those will still happen,
and they may or may not involve a copy operation or some other I/O
"performance killing" behavior.
That's ignoring all the other implementation requirements for using
O_DIRECT:
* You're not guaranteed it's supported. Some file systems simply do
not support it, and the open(2) call will fail. Some operating
systems simply do not support the flag. There's no portable way to
determine whether the flag is supported.
* You have to align everything correctly, and there is no portable way
to find out the correct alignment, nevermind the optimal one.
All of that for a rather dubious performance benefit, and direct I/O
as you describe it is essentially impossible on some modern
filesystems (e.g., ZFS, BTRFS). Moreover, I would suspect the VM
gymnastics required to allow the FS to work correctly would kill a lot
of the performance, especially when compared to just copying the
buffer. TLB flushes and messing with page tables is expensive.
Making copies is not expensive up to a pretty large point.
Adam
Ian Collins
May 19, 2012, 12:14:47 AM5/19/12
to
a meaningful answer.
What is the code that runs so much faster?
Is the data format line or record based?
--
Ian Collins
Mike Copeland
May 20, 2012, 12:07:12 AM5/20/12
to
In article <a1ol1n...@mid.individual.net>, ian-...@hotmail.com
says...
The code above _is_ the code that runs much slower, and the code that
I'm basing it on uses fgets/fputs and setbuf with a size of 4096
characters/bytes. That is the only distinction between the 2 code
fragments, but including the I/o class definitions I wrote to prepare
and handle the fgets/fputs with *FILE and open/close logic is too much
to post here.
Bottom line: the executing code is only comparing *FILE
fgets/fputs/setbuf with the fstream getline/<< code above. The only
difference I can see if that the absence of a "setbuf" capability with
fstreams is an enormous performance hit...or getline/<< is a terrible
way to to text file I/o. 8<{{
says...
I'm basing it on uses fgets/fputs and setbuf with a size of 4096
characters/bytes. That is the only distinction between the 2 code
fragments, but including the I/o class definitions I wrote to prepare
and handle the fgets/fputs with *FILE and open/close logic is too much
to post here.
Bottom line: the executing code is only comparing *FILE
fgets/fputs/setbuf with the fstream getline/<< code above. The only
difference I can see if that the absence of a "setbuf" capability with
fstreams is an enormous performance hit...or getline/<< is a terrible
way to to text file I/o. 8<{{
Ian Collins
May 20, 2012, 2:31:06 AM5/20/12
to
your results.
These things are tricky to to test. For instance, I have to keep using
new files, or use a file > 16GB to avoid the OS file cache. So any code
that compares reading the same file will be biased to the second test.
By the way, on my Solaris box, comparing
while( (fgets( buf, bufSize, from ) ) )
{
lines++;
fputs( buf, to );
}
to
while( std::getline(in, line) )
{
lines++;
out << line << '\n';
}
shows the iostream version to take a little under twice as long as the C
version. Using a custom streambuf with a 32K buffer narrows this slightly.
--
Ian Collins
Paavo Helde
May 20, 2012, 2:42:27 AM5/20/12
to
mrc...@cox.net (Mike Copeland) wrote in
news:MPG.2a221bd7d...@news.eternal-september.org:
Sorry, no, you haven't provide enough information. The fact that you
report 12x difference shows that either you or your prehistoric compiler
is doing something seriously wrong, and the code fragments do not show
clearly what this is. Please post compilable examples for both cases,
together with main() and the timing results, and all the compiler
switches (you can copy them from the C++ project properties Command Line
page.
Cheers
Paavo
news:MPG.2a221bd7d...@news.eternal-september.org:
report 12x difference shows that either you or your prehistoric compiler
is doing something seriously wrong, and the code fragments do not show
clearly what this is. Please post compilable examples for both cases,
together with main() and the timing results, and all the compiler
switches (you can copy them from the C++ project properties Command Line
page.
Cheers
Paavo
jamin....@googlemail.com
May 21, 2012, 11:15:03 AM5/21/12
to
On Sunday, May 20, 2012 5:07:12 AM UTC+1, Mike Copeland wrote:
> Bottom line: the executing code is only comparing *FILE
> fgets/fputs/setbuf with the fstream getline/<< code above. The only
> difference I can see if that the absence of a "setbuf" capability with
> fstreams is an enormous performance hit...or getline/<< is a terrible
> way to to text file I/o. 8<{{
OK, here is my opening gambit for provoking more focused discussion. I wondered the same thing and the subject has come up a couple of times by users of my library (http://www.benhanson.net/lexertl.html) Personally, I've only needed to use read() and gcount() on istream derived classes, so the following code is based around that assumption:
> Bottom line: the executing code is only comparing *FILE
> fgets/fputs/setbuf with the fstream getline/<< code above. The only
> difference I can see if that the absence of a "setbuf" capability with
> fstreams is an enormous performance hit...or getline/<< is a terrible
> way to to text file I/o. 8<{{
template<typename CharT, class Traits>
class basic_fast_filebuf : public std::basic_streambuf<CharT, Traits>
{
public:
basic_fast_filebuf (const char *filename_) :
_fp (0)
{
_fp = ::fopen(filename_, "r");
}
virtual ~basic_fast_filebuf()
{
::fclose(_fp);
_fp = 0;
}
protected:
FILE *_fp;
virtual std::streamsize xsgetn (CharT *ptr_, std::streamsize count_)
{
return ::fread (ptr_, sizeof(CharT), static_cast<std::size_t>(count_), _fp);
}
};
typedef basic_fast_filebuf<char, std::char_traits<char> > fast_filebuf;
typedef basic_fast_filebuf<wchar_t, std::char_traits<wchar_t> > wfast_filebuf;
I used the code like so:
fast_filebuf buf ("Unicode/PropList.txt");
std::istream if_(&buf);
etc.
I haven't tested to see if this really is faster, but presumably it couldn't get a lot simpler than that without foregoing C++ streams entirely.
Anyway, does that help at all? :-)
Regards,
Ben
jamin....@googlemail.com
May 22, 2012, 7:35:23 AM5/22/12
to
Regards,
Ben
jamin....@googlemail.com
May 27, 2012, 9:23:40 AM5/27/12
to
If anyone is still following this thread, this looks like the ultimate way to access files that will fit in memory: http://en.wikibooks.org/wiki/Optimizing_C%2B%2B/General_optimization_techniques/Input/Output#Memory-mapped_file
Regards,
Ben
Regards,
Ben
Luca Risolia
May 27, 2012, 12:06:08 PM5/27/12
to
On 17/05/2012 22:34, Mike Copeland wrote:
> I have the following routine that determines the time required to
> copy a large text file to another. I'm comparing this code to a class I
> wrote (too large to post here) that does text file I/o with explicit
> buffer size designation (4096 character buffers). The code below runs
> 12-13 times slower, although doing exactly the same work.
> I have the following routine that determines the time required to
> copy a large text file to another. I'm comparing this code to a class I
> wrote (too large to post here) that does text file I/o with explicit
> buffer size designation (4096 character buffers). The code below runs
> 12-13 times slower, although doing exactly the same work.
> fVar1.open("pat12.n00", fstream::in);
> if(fVar1.is_open())
> {
> fVar2.open("test.txt", fstream::out);
> while(getline(fVar1, line))
> {
> fVar2<< line<< endl;
> }
> fVar1.close(), fVar2.close();
> }
It's not really clear why you need the string in the above example, but
> if(fVar1.is_open())
> {
> fVar2.open("test.txt", fstream::out);
> while(getline(fVar1, line))
> {
> fVar2<< line<< endl;
> }
> fVar1.close(), fVar2.close();
> }
if you just want to copy large text files efficiently using fstream
buffers, do this:
ifstream in("file1");
ofstream out("file2");
out << in.rdbuf();
It performs better than my system cp, which is written in C:
$ time cp file1 file2
real 1m36.491s
user 0m0.040s
sys 0m4.220s
$ rm file2; time ./fstreamcp
real 1m25.223s
user 0m0.164s
sys 0m5.252s
$ ls -l file*
-rw-rw-r-- 1 luca luca 1073741824 mag 27 17:35 file1
-rw-rw-r-- 1 luca luca 1073741824 mag 27 17:57 file2
fstreamcp has been compiled with g++ -O3.
Scott Lurndal
May 27, 2012, 7:39:15 PM5/27/12
to
It is likely that file1 was completely in memory when you ran your
fstreamcp, since the cp would have pre-loaded it as part of the copy.
scott
Luca Risolia
May 27, 2012, 8:24:10 PM5/27/12
to
On 28/05/2012 01:39, Scott Lurndal wrote:
> how did you flush file1 from the OS file cache before you ran fstreamcp?
with
> how did you flush file1 from the OS file cache before you ran fstreamcp?
$ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
Pavel
May 27, 2012, 10:12:14 PM5/27/12
to
on that. Now, to measure the difference in performance of user-space code, you
probably *want* cache warmed-up (in both cases) to get file system i/o (it's
equally page-based at kernel level and device-dependent-buffer-based at device
level anyway).
On the other hand, analysing "user" time in your result (which includes all C++
stream code in question and its 'cp' counterpart) you can see that 'cp' takes
more than 4 times less 'user' time than 'fstreamcp'.
-Pavel
0 new messages