newbie's slow file i/o
ga...@oak.circa.ufl.edu
** Hi y'all, A Question from Newbieville
** ... the following proggie seems _real slow_. Is it because I
** added "printf("Processing file...);" down yonder _OR_ is it
** because I have several flags && therefore several ints to check
** each time a new character is read???
** A user might run this on one file, or they might run it on an
** entire directory ala "detab *.aml *.menu"
** Comments and criticism well appreciated. Flames too, if you must ;-)
** One more thingy: if someone wants to _teach_ me what would be the
** minimum necessary parts to post, I'll gladly learn to do so and
** help enhance the signal to noise ratio.
*/
/* detab.c */
/*
** Gary M. Greenberg. April 3, 1995
** detab.c = (Copyright == _NULL_ ) ? NoGuarantee : You_Not_Use;
**
** Program for De-Tabbing AMLs; Arc/Info Macro Language.
** AML barfs if a "\t" is in the code, but not in comments.
** My solution:
** copy input to new file and replace tabs with two spaces.
** overwrite original file with tab-less file.
** Skip tabs in text following comments.
** A comment in AML is "/*" <sans quotes> and ends at newline.
** That is, it doesn't have closure "*/" like C comments.
*/
#include <stdio.h>
#include <string.h>
#include <ctype.h>
int main( int argc, char *argv[100])
{
int i, tabcount=0, c=0, b=0;
int c_one=0, c_two=0;
FILE *rfp, *wfp;
if(argc<2) {
printf("Usage: <%s> <file_name>.\n",argv[0]);
printf("Note: %s accepts *.xyz for Workstations.\n",argv[0]);
exit (1);
}
/* Note ALL further error checking omitted for post */
rfp=fopen(argv[i],"r");
wfp=fopen("write_temp","w");
for (i=1;i<argc;++i) { /* Outer Loop */
while((c=getc(rfp))!=EOF) {
if (c=='/') /* potential comment, set flag 1 */
c_one = 1;
if (c=='\n' || c=='\r' || c==' ') /* Nope, reset flag */
c_one = 0;
if (c=='*' && c_one == 1)
c_two = 1;
if (c_two == 1 && (c=='\n' || c=='\r'))
c_two = 0;
/* Now, if c_two is still 0, it's not an AML comment */
if (c=='\t' && c_two == 0) {
/* got a tab, replace it with 2 spaces */
++tabcount;
fprintf(wfp," ");
continue;
printf("Currently removing tabs from %s.\n",argv[i]);
}
else
putc(c,wfp);
/* Totally inane graphic. HEY YOU, quit laughing ;-) */
/* There must be a _correct_ way to do this */
printf("Processing file\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b");
}
fclose(wfp);
fclose(rfp);
printf("%d Tabs replaced in file %s.\n",tabcount,argv[i]);
tabcount=0;
/* file has been cleared of tabs, replace it. */
rfp=fopen("write_temp","r");
wfp=fopen(argv[i],"w");
while((c=getc(rfp))!=EOF)
putc(c,wfp);
/* all done; clean it up */
fclose(wfp);
fclose(rfp);
}
printf("All thru, All good, Good day.\n");
return 0;
} /* ********* end program ********* */
/* AdvThxAnce, gary */
Ronald F. Guilmette
>/*
>** Hi y'all, A Question from Newbieville
>** ... the following proggie seems _real slow_. Is it because I
>** added "printf("Processing file...);" down yonder _OR_ is it
>** because I have several flags && therefore several ints to check
>** each time a new character is read???
>** A user might run this on one file, or they might run it on an
>** entire directory ala "detab *.aml *.menu"
>** Comments and criticism well appreciated. Flames too, if you must ;-)
OK... here's a mild flame:
Don't post 100+ lines of code and ask us where the bottleneck is. You're
a big boy, and you should be able to profile it yourself. If you haven't
got a profiler and/or a code coverage tool, get one. (I think there are
some free ones floating around the net.)
P.S. Actually, a lot of profilers are next to useless for tracking bottle-
necks down to the statement level, as many of them will only tell you how
much time you spent in each _function_. I invite people to post information
on available STATEMENT LEVEL profilers, as that information would probably
be of value to a lot of the audience here. I know that there are quite
a number of CODE COVERAGE tools out there that will tell you how many
_times_ a each statement was executed (for a given run) but these execu-
tion counts may correlate only poorly to actual TIME SPENT executing each
different statement. (I only mention this as an admission that code
coverage tools are not really the best way to find a bottleneck... but
if it is all you have, it is better than nothing.)
--
-- Ron Guilmette, Sunnyvale, CA ---------- RG Consulting -------------------
---- E-mail: r...@segfault.us.com ----------- Purveyors of Compiler Test ----
-------------------------------------------- Suites and Bullet-Proof Shoes -
Henry C. Schoepp
Think about allocating a large buffer, possibly the size of the file and using a
fread() I/O to get a large block. Then go through the block with a pointer and
write it out to a file. If you can create a block large enough to hold the
entire file, then you no longer need the intermediate file.
You will want to create another block of memory twice the size of the first
(Assumes a line of all tabs) to copy the characters into. This will also save
time as there will be fewer disk access during the write.
Hank
Mike Bandy
There's already a followup regarding the cascaded ifs so I won't comment
on that. But the second part of the program deals with overwriting the
original file and goes like this:
[big snip]
> /* file has been cleared of tabs, replace it. */
> rfp=fopen("write_temp","r");
> wfp=fopen(argv[i],"w");
> while((c=getc(rfp))!=EOF)
> putc(c,wfp);
> /* all done; clean it up */
> fclose(wfp);
> fclose(rfp);
You're assured to get this done faster by using the system()
command to execute operating system commands directly. The actual
commands will vary based on the O/S under which this runs, but might be
something like this under Unix:
char str[100];
sprintf( str, "rm -f %s", argv[i] );
system( str );
sprintf( str, "mv write_temp %s", argv[i] );
system( str );
--
Mike Bandy ba...@aplcomm.jhuapl.edu
Johns Hopkins University / Applied Physics Laboratory
Scott Tringali
: /*
: ** Hi y'all, A Question from Newbieville
: ** ... the following proggie seems _real slow_. Is it because I
: ** added "printf("Processing file...);" down yonder _OR_ is it
: ** because I have several flags && therefore several ints to check
: ** each time a new character is read???
Neither. The offending statement is this:
: while((c=getc(rfp))!=EOF) {
Do a block read (in your case, fread) with a buffer of 8K, and then do
your processing on the buffer. Want some numbers?
Bufsize Clock Time #loops
--------------------------------
1 423.4 1468802
2 215.2 734401
4 107.2 367201
8 54.0 183601
. . .
8192 0.3 180
You get the point. [Data from Stevens, Advanced Programming in the
Unix Environment, p. 57]
-Scott
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Scott J. Tringali trin...@ultranet.com
Raytheon Electronic Systems Division IATC #include <dumb_quote.h>
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Thomas Richard Dibble
g> /*
g> ** Hi y'all, A Question from Newbieville
g> ** ... the following proggie seems _real slow_. Is it because I
g> ** added "printf("Processing file...);" down yonder _OR_ is it
that printf is out of any loop, so it shouldn't slow things down measurably.
g> ** because I have several flags && therefore several ints to check
g> ** each time a new character is read???
That's the one! Well, that and the fact that it's using the C buffered
file access instead of machine-specific assembly, so it's going to be loads
slower than, say, the 'copy' function which is in your OS.
g> **
g> ** Program for De-Tabbing AMLs; Arc/Info Macro Language.
g> ** AML barfs if a "\t" is in the code, but not in comments.
g> ** My solution:
g> ** copy input to new file and replace tabs with two spaces.
g> ** overwrite original file with tab-less file.
If you wanted to be *really* spiffy, you'd indent to tabstops, but that is
a bit more involved than the code below (you have to know where on the line
you are) so I won't go into it.
g> /* Note ALL further error checking omitted for post */
g> rfp=fopen(argv[i],"r");
g> wfp=fopen("write_temp","w");
Okay, this is where I'd re-organize things ...
g> for (i=1;i<argc;++i) { /* Outer Loop */
First off, the above file-opening code should be inside the loop I assume
(it has to be, in order for the code to do what the comments above say) ...
g> while((c=getc(rfp))!=EOF) {
g> if (c=='/') /* potential comment, set flag 1 */
g> c_one = 1;
g> if (c=='\n' || c=='\r' || c==' ') /* Nope, reset flag */
g> c_one = 0;
g> if (c=='*' && c_one == 1)
g> c_two = 1;
g> if (c_two == 1 && (c=='\n' || c=='\r'))
g> c_two = 0;
g> /* Now, if c_two is still 0, it's not an AML comment */
g> if (c=='\t' && c_two == 0) {
g> /* got a tab, replace it with 2 spaces */
g> ++tabcount;
g> fprintf(wfp," ");
g> continue;
g> printf("Currently removing tabs from %s.\n",argv[i]);
g> }
g> else
g> putc(c,wfp);
g> /* Totally inane graphic. HEY YOU, quit laughing ;-) */
g> /* There must be a _correct_ way to do this */
g> printf("Processing file\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b");
Not 100% portably. But if you have a 'conio' package, or are working with
anything other than a dumb terminal teletype-like thing, you should be able
to save the position of the cursor, puts("Processing file"); and then
return to the last saved cursor location.
g> }
A bit neater: (sorry about the goofy spacing)
while((c=getc(rfp))!=EOF)
{
switch(c)
{
case '/':
poss_comm=1;
break;
case '*':
if(poss_comm)
{
putc(wfp,c);
process_comm(rfp, wfp);
}
continue; /* Don't put c again! */
case '\t':
putc(wfp,' ');
c=' ';
poss_comm=0;
break;
default: poss_comm=0; break;
}
/* Write the character to the file */
putc(wfp,c);
}
And then 'process_comm(FILE *rfp, FILE *wfp)' would do the same thing,
except scan for a '\n' and return after writing it (of course, not doing
anything with comments or tabs).
g> fclose(wfp);
g> fclose(rfp);
g> printf("%d Tabs replaced in file %s.\n",tabcount,argv[i]);
g> tabcount=0;
g> /* file has been cleared of tabs, replace it. */
Should probably close it first.
g> rfp=fopen("write_temp","r");
g> wfp=fopen(argv[i],"w");
g> while((c=getc(rfp))!=EOF)
g> putc(c,wfp);
g> /* all done; clean it up */
This step might be done faster using assembly or calling the system, if you
are worried about speed here a lot. Such code as above is about 8-10 times
as slow as the assembly version (in DOS ... figure from re-writing 'copy'
using such code (in fact exactly like those last five lines) and comparing
execution times with the DOS copy and xcopy functions).
g> fclose(wfp);
g> fclose(rfp);
g> }
g> printf("All thru, All good, Good day.\n");
g> return 0;
g> } /* ********* end program ********* */
g> /* AdvThxAnce, gary */
g>
--
/"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""\
| Tired of lying in the sunshine, \ / tom...@wpi.wpi.edu |
| staying home to watch the rain. / \ the happiest man alive |
\.........................................................................../
Ronald F. Guilmette
Henry C. Schoepp
Mike Bandy
There's already a followup regarding the cascaded ifs so I won't comment
on that. But the second part of the program deals with overwriting the
original file and goes like this:
[big snip]
> /* file has been cleared of tabs, replace it. */
> rfp=fopen("write_temp","r");
> wfp=fopen(argv[i],"w");
> while((c=getc(rfp))!=EOF)
> putc(c,wfp);
> /* all done; clean it up */
Scott Tringali
: /*
: ** Hi y'all, A Question from Newbieville
: ** ... the following proggie seems _real slow_. Is it because I
: ** added "printf("Processing file...);" down yonder _OR_ is it
: ** because I have several flags && therefore several ints to check
: ** each time a new character is read???
Neither. The offending statement is this:
Jari Kokko
> sprintf( str, "rm -f %s", argv[i] );
> system( str );
> sprintf( str, "mv write_temp %s", argv[i] );
> system( str );
Yikes! Are you nuts? Can you spell "zsh -c"? Executing system level with
system() is crazy if all you want to do is overwrite a file. Why not just
remove() and rename(). Both are standard functions.
Jari, who thinks he might be missing a joke somewhere...
Michael Shields
Mike Bandy <ba...@aplcomm.jhuapl.edu> wrote:
> You're assured to get this done faster by using the system()
> command to execute operating system commands directly. The actual
> commands will vary based on the O/S under which this runs, but might be
> something like this under Unix:
>
> char str[100];
> system( str );
> sprintf( str, "mv write_temp %s", argv[i] );
> system( str );
Surely you're joking. Apart from only working on Unix and imposing a
100-character limit on the filename -- both serious flaws -- it has the
overhead of a fork and exec, just to call other C programs. How could
that be faster, once the poster changes the character-at-a-time I/O?
A portable replacement:
remove(argv[i]);
rename("write_temp", argv[i]);
--
Shields.
Thomas Richard Dibble
[ RE using getc() type functions to read/write a file ]
HCS> You're doing things one character at a time. This is slow and disk
HCS> intensive. Think about allocating a large buffer, possibly the size
HCS> of the file and using a fread() I/O to get a large block. Then go
HCS> through the block with a pointer and write it out to a file. If you
HCS> can create a block large enough to hold the entire file, then you no
HCS> longer need the intermediate file.
It was my understanding that the functions which use the (FILE *) type read
large buffers into memory from disk (hence, the large area allocated to
the FILE pointer). Is this incorrect?
Even so, your suggestion would improve matters considerably, especially if
both file buffers could be held in memory entirely at once. Faster 16-,
32- and 64-bit memory copies could also be used in the 'fread' case,
whereas the 'getc' case limits the program to 8-bit copies.
HCS> You will want to create another block of memory twice the size of the
HCS> first (Assumes a line of all tabs) to copy the characters into. This
HCS> will also save time as there will be fewer disk access during the
HCS> write.
Along these lines, another consideration is how one sizes the input block
of memory. Reading the file as a single one-dimensional line (with \n's or
NULL's marking line breaks) rather than in a two-dimensional array makes
this a simpler process, and needing just realloc in addition to
malloc/calloc. But then again, once you have the input file read in you
could do the same for the output buffer (allocate some arbitrary amount and
realloc when that amount is exhausted).
---- Tom Dibble
Edward Hartnett
J> Mike Bandy:
>> sprintf( str, "rm -f %s", argv[i] );
>> system( str );
>> sprintf( str, "mv write_temp %s", argv[i] );
>> system( str );
J> Yikes! Are you nuts? Can you spell "zsh -c"? Executing system level with
J> system() is crazy if all you want to do is overwrite a file. Why not just
J> remove() and rename(). Both are standard functions.
J> Jari, who thinks he might be missing a joke somewhere...
On a slightly related topic, I would be very interested in seeing some
examples of C or C++ programs that do things otherwise usually done in
the shell.
I have a perl 5 program, for example, that does some fairly complex
shelly sort of things. It opens an ftp connection, grabs a bunch of
files, runs various programs on the files, and then renames them.
I've also got tons of C and some C++ code, and I'm wondering if it
would not be more productive to switch from perl 5 to C++ for this
task (much as I lover perl 5, a *fantastic* language). Problem is,
since I've always done this kind of task with shell scripts (first
csh, then bourne) and perl, I've never written any C code for the
purpose.
[Not sure if this is on topic, but I think it would be an interesting
discussion, and related to some common issues faced by C programmers. -mod]
--
Henry C. Schoepp
>large buffers into memory from disk (hence, the large area allocated to
>the FILE pointer). Is this incorrect?
The above is a true statement. The definition of large in the second line is
implementation specific. Sometimes large means 512 bytes, sometimes it may mean
512 megabytes. Generally these buffers are some convenient small number, like
the size of a cluster on a system. Furthermore, it requires that each of the
characters be fetched from that buffer, and all the other internal stuff dealing
with it. By using the fread() function to read a larger block of data at once,
you reduce the number of seeks on the disk (assuming the file does not have 100%
fragmentation!) which is the longest delay in any such access.
Disk seeks are generally in the range of about 10 to 30 ms, depending upon the
age of the hardware. This maybe further reduced by additional software cahces.
However it is not uncommon to have transfer rates around 1 M/s.
Hank
George Swan
Ronald F. Guilmette <r...@rahul.net> wrote:
>In article <3m181e$d...@solutions.solon.com>, <ga...@oak.circa.ufl.edu> wrote:
>>** Hi y'all, A Question from Newbieville
>
>OK... here's a mild flame:
>
>Don't post 100+ lines of code and ask us where the bottleneck is. You're
>a big boy, and you should be able to profile it yourself. If you haven't
>got a profiler and/or a code coverage tool, get one. (I think there are
>some free ones floating around the net.)
Actually, shouldn't Gary really be using expand or col?
That is, existing tools that do the task he had in mind, without
requiring him to do any programming.
[well, depends; not if Gary's on a platform which doesn't have such tools.
-mod]
William Kaufman
h...@eel.ufl.edu (Henry C. Schoepp) writes:
] You're doing things one character at a time. This is slow and disk intensive.
] Think about allocating a large buffer, possibly the size of the file and using a
] fread() I/O to get a large block.
And, in article <3m6ge8$8...@solutions.solon.com>,
trin...@news.ed.ray.com (Scott Tringali) says the same:
] Do a block read (in your case, fread) with a buffer of 8K, and then do
] your processing on the buffer.
But the standard C libraries should be doing block I/O and handing
back single characters, shouldn't it? Buffering (e.g., through
setvbuf()) shouldn't win you much, especially if the file is smaller
than BUFSIZ (or if BUFSIZ is at least 8K).
I'd think that all you gain by this is some function call overhead,
calling fread() once versus N calls to fgetc(). (And maybe not even
that: fgetc() could be implemented inline--say, via a macro--where
fread() almost never can, practically speaking.)
Scott, you quoted (with statistics) Stevens, but I don't have access
to that book. Was he talking about fopen()/fread(), or was he talking
about open()/read()? (The latter definitely suffer from lack of
buffering,...)
In article <3m6ga6$8...@solutions.solon.com>,
ba...@aplcomm.jhuapl.edu (Mike Bandy) writes,
] You're assured to get this done faster by using the system()
] command to execute operating system commands directly.
Wouldn't calling remove() and rename() be faster? At least, you
avoid two calls to system() (which, on Unix, involves fork() and exec(),
plus two invocations of the shell).
But, yes, using either rename() or the OS's rename utility should be
faster than copying it by hand, since the OS probably doesn't do any
copying, esp. if the two files are in the same directory.
-- Bill K.
Bill Kaufman | "I mean ... it's not even been a two-and-two-make-
wkau...@us.oracle.com | five sort of day, it's more like a two-and-two-
| make ... *fish* ..." -- "Cages", Dave McKean
Ray Dunn
>You're doing things one character at a time. This is slow and disk intensive.
>Think about allocating a large buffer, possibly the size of the file and using a
>write it out to a file. If you can create a block large enough to hold the
>entire file, then you no longer need the intermediate file.
The advice about buffersize here and in other responses is good, as is using
remove() and rename() - do *not* use system() for i/o that is provided as library
calls.
The use of an intermediary file is recommended however. When complete the
original file is deleted and the new one renamed to the original name as was
shown. This gives the minimum window for a crash to delete your data. You will
always have either the original or the temporary version except if something goes
wrong during the rename. If you overwrite the original file, you are maximizing
the risk of a crash or power-down etc. destroying your data.
---
Ray Dunn (opinions are my own) | Phone: (514) 954 9050
Montreal | Phax : (514) 954 9057
r...@ultimate-tech.com | Home : (514) 630 3749
Dan Pop
>ga...@oak.circa.ufl.edu wrote:
>: /*
>: ** Hi y'all, A Question from Newbieville
>: ** ... the following proggie seems _real slow_. Is it because I
>: ** added "printf("Processing file...);" down yonder _OR_ is it
>: ** because I have several flags && therefore several ints to check
>: ** each time a new character is read???
>
>Neither. The offending statement is this:
>
>: while((c=getc(rfp))!=EOF) {
>
>Do a block read (in your case, fread) with a buffer of 8K, and then do
>your processing on the buffer.
Why? stdio will do the block read anyway. Hint: stdio has its own
internal buffering. And getc() is a macro defined in <stdio.h> in most
implementations, so it doesn't even have the overhead of a function call
(unless the programmer explicitly bypasses the macro and uses the function
from the standard library).
> Want some numbers?
>
>Bufsize Clock Time #loops
>--------------------------------
>1 423.4 1468802
>2 215.2 734401
>4 107.2 367201
>8 54.0 183601
>. . .
>8192 0.3 180
>
>You get the point. [Data from Stevens, Advanced Programming in the
>Unix Environment, p. 57]
He gets the wrong point, unless he has Stevens' book and can read that
page himself and see that this table applies to something completely
different than the subject of our discussion: calling read() (a Unix
_system call_) with different sizes of the user supplied buffer.
When using stdio functions, whether getc() or fread(), read() is called
by stdio, usually with the same buffer size. So, whether the user reads
the file with getc() or with fread(), the same number of read() calls
are issued by stdio. Which means that the above quoted table is
completely irrelevant to our discussion. Showing it to a newbie,
without adequate comments, in order to support a bogus claim/advice,
is extremely unethical.
Reading the file with fread() instead getc() will be somewhat faster
(less stdio overhead), but not 1400+ times faster, as suggested by the
table. And if the processing has to be done on a character by character
basis, the getc() approach is the most natural one, even if marginally
slower.
Dan
--
Dan Pop
CERN, CN Division
Email: dan...@afsmail.cern.ch
Mail: CERN - PPE, Bat. 31 R-004, CH-1211 Geneve 23, Switzerland
Jonathan Chen
ba...@aplcomm.jhuapl.edu (Mike Bandy) writes:
[snip]
>You're assured to get this done faster by using the system()
>command to execute operating system commands directly. The actual
>commands will vary based on the O/S under which this runs, but might be
>something like this under Unix:
>
> char str[100];
> sprintf( str, "rm -f %s", argv[i] );
> system( str );
> sprintf( str, "mv write_temp %s", argv[i] );
> system( str );
>
I don't know what system you're using, but on all the UNIX boxes we have,
unlink(2) and rename(2) are a lot faster (and more efficient) than making
a system(3) call.
[remove(3) is also fast, and is ANSI, too. -mod]
--
-------------------------------------------------------------------
Jonathan Chen <jo...@pinnacle.co.nz>
-------------------------------------------------------------------
Greg Wooledge
>Neither. The offending statement is this:
>: while((c=getc(rfp))!=EOF) {
Most getc() implementations are rather fast. Since he has to process
one character at a time anyway, due to the nature of his problem, the
getc() code is natural.
>Bufsize Clock Time #loops
>--------------------------------
>1 423.4 1468802
>2 215.2 734401
>4 107.2 367201
>8 54.0 183601
>. . .
>8192 0.3 180
>You get the point. [Data from Stevens, Advanced Programming in the
>Unix Environment, p. 57]
I don't have that book (yet). Are those data for getc() and/or fwrite(),
or are they for a Unix read()? (Kernighan & Pike, p. 203, has a similar
table.) For later reference, your worst/best ratio is 1411.
I did some testing with the following code:
#include <stdio.h>
#include <stdlib.h>
int main (int argc, char *argv[])
{
int buflen;
char *buf;
if (argc <= 1)
return EXIT_FAILURE;
if ((buflen = atoi (argv[1])) == 0)
return EXIT_FAILURE;
if (buflen == 1)
{
int c;
while ((c = getchar()) != EOF)
if (putchar (c) == EOF)
return EXIT_FAILURE;
}
else
{
size_t nread;
if ((buf = malloc (buflen)) == NULL)
return EXIT_FAILURE;
while ((nread = fread (buf, 1, buflen, stdin)) > 0)
if (fwrite (buf, 1, nread, stdout) < nread)
return EXIT_FAILURE;
}
return 0;
}
Note that in this case, I am not analyzing each character of the buffer
(unless of course buflen == 1). I am merely copying data. Here is what
I found:
Buffer length real user sys
------------- ---- ---- ----
1 2.25 1.50 0.25
1 1.68 1.19 0.18
2 6.14 4.84 0.13
2 6.91 4.81 0.13
4 3.79 2.42 0.14
4 2.86 2.26 0.10
8 1.35 1.14 0.14
8 2.00 1.44 0.12
32 0.56 0.43 0.11
32 0.52 0.42 0.10
128 0.32 0.11 0.17
128 0.90 0.21 0.14
512 0.21 0.06 0.13
512 0.32 0.12 0.11
1024 0.57 0.14 0.18
1024 0.30 0.13 0.10
2048 0.34 0.11 0.12
2048 0.45 0.17 0.14
4096 0.39 0.07 0.11
4096 0.73 0.04 0.23
8192 0.15 0.06 0.09
8192 0.14 0.06 0.07
Data collected: AIX 3.2.5, IBM POWERstation 220, NFS-mounted filesystems,
compiled with cc -O. Input size 1381854 bytes. Actual invocations:
time timing 1 </usr/local/bin/gdb >/dev/null
etc.
Worst/best ratio (read time) is 49.35, significantly lower than 1411.
To summarize, using a buffer will help minimize the time required for I/O
and accompanying error checking. However, the results are not quite as
dramatic as your post would lead one to believe because stdin and stdout
are already buffered. Moreover, using getchar/putchar instead of
fread/fwrite for single-character processing improves performance quite
noticeably. In an application which must process the data it reads,
instead of merely copying it from one stream to another, the I/O overhead
may be negligible compared to the processing overhead, so the added
complication of using one's own buffers may not be justified.
I would be interested to see timing results of a buffered solution to the
original problem versus those of a getc/getchar solution.
--
# Greg Wooledge /..\
# bh...@summitis.com \--/ Have a day.
# aa1...@freenet.lorain.oberlin.edu (good luck)
Thad Smith
"Ronald F. Guilmette" <r...@rahul.net> wrote:
> P.S. Actually, a lot of profilers are next to useless for tracking bottle-
> necks down to the statement level, as many of them will only tell you how
> much time you spent in each _function_. I invite people to post information
> on available STATEMENT LEVEL profilers, as that information would probably
> be of value to a lot of the audience here.
The profiler which came with Borland's BC++ 3.10 works nicely,
including statement level profiling. It runs interactively to allow
you to select range and granularity of coverage and to determine the
sampling method and rate, which determines how much overhead the
profiler adds. It produces a simple histogram showing execution time
and percentage of execution time. I heard rumors that Borland removed
it from later versions (a pity if so).
--
Thad Smith ---------------\ T3 Systems - embedded software development
Thad...@acm.org \"A lot of kisses and you'll be surprised"
PGP key: finger th...@csn.net \--Box of candy kisses with prize, 1957
----------------------------------------------------------------------
Ray Dunn
pigeons and categorically state that getch() is the _slowest_ way to scan
a file character by character, and this I think has been already been
shown by the timings published in other articles.
This is _not_ just the overhead of getchar() as opposed to a direct
reference through a buffer pointer (which is large in itself), but also
involves the read latency each time you have to go back to the disk.
Even if a subsequent disk read is from the same cylinder as the last
read (you can't even guarantee that on a multi-tasking OS), on average
you will waste half a revolution before the data is under the head. You
want to minimise the number of times the disk is physically read, and
optimally you want to read a cylinder at a time (but optimizing down to
this hardware level is not a good idea - the hardware may change).
Read as large a buffer as possible, scan it using pointers, and if you
are creating output, only copy the input data if you absolutley have to.
If you can create the output by modifying the input in-situ without large
block moves, then do so.
Yes, the high level I/O functions do buffer their input, but in general
the buffers are much smaller than you can make them yourselves. In the
PC world BUFSIZ is 512 bytes. You can create buffers up to UINT_MAX, but
they should be a multiple of BUFSIZ bytes so the the file system can
tranfer an integral number of buffers to your data area.
I work in the graphics arts business and for the huge multi-megabyte
image files we have to deal with, these techniques mean the difference
between viable commercial products and sluggish toys.
Elmar Bartel
[ ... ]
: ** ... the following proggie seems _real slow_. Is it because I
: ** added "printf("Processing file...);" down yonder _OR_ is it
: ** because I have several flags && therefore several ints to check
: ** each time a new character is read???
Nobody in this thread gave Gary the right answer, so here it is:
It is because of the "printf("Processing file....
Especially because of the lot's of \b's, the tty handling code
of the kernel has a lot todo with them:
This printf is done with each character read from file.
This string has 30 characters (15 of them are \b). If your are
running on a normal conifgured tty the kernel will produce for
each \b three characters: backspace blank, backspace. So the
kernel outputs 15+3*15 characters to your tty for each character
you read. Assume you process a file with 1000 characters, the
kernel will write 60K characters to your tty! Probably the kernel
executes much more instructions than your program.
Or worse, on a 9600 connected dumb terminal its not the kernel
which is the bottleneck, but the serial line: it will last for
60000*8/9600 = 50s !!!) Got the point?
Gary also wrote:
[ ... ]
: ** Comments and criticism well appreciated. Flames too, if
: you must ;-)
Here a really SERIOUS suggestion:
Write your code properly formatted, so that matching
braces/blocks etc can be easily recognized!!
Or use a formatting tool like indent.
Otherwise things like this happen:
Thomas Richard Dibble (tom...@WPI.EDU) wrote:
: g> ** ... the following proggie seems _real slow_. Is it because I
: g> ** added "printf("Processing file...);" down yonder _OR_ is it
: that printf is out of any loop, so it shouldn't slow things down
: measurably.
I assume this is due to the completly misleading formatting
of the source!!
But another point, why I'm jumping in in this thread are the lot's of
irritating statements and wrong suggestions, two examples:
=======================================================================
Henry C. Schoepp (h...@eel.ufl.edu) wrote:
(References: <3m181e$d...@solutions.solon.com> <3m6g7l$8...@solutions.solon.com>)
: You're doing things one character at a time.
: This is slow and disk intensive.
: Think about allocating a large buffer, possibly the size of the file
: and using a fread() I/O to get a large block.
[ ... ]
Exactly this done when you use getc/putc. The standard libc allocates
a buffer, fills this buffer with one system call. getc and putc do
their job on this buffer and the system get only called when this
buffer gets empty/full. The influence of the buffersize on the
execution time is marginal if the is about 1K or so.
(See also William Kaufman reply)
Or more wrong:
=======================================================================
Mike Bandy (ba...@aplcomm.jhuapl.edu) wrote:
(References: <3m181e$d...@solutions.solon.com> <3m6ga6$8...@solutions.solon.com>)
: ga...@oak.circa.ufl.edu writes:
: [big snip]
: > /* file has been cleared of tabs, replace it. */
: > rfp=fopen("write_temp","r");
: > wfp=fopen(argv[i],"w");
: > while((c=getc(rfp))!=EOF)
: > putc(c,wfp);
: > /* all done; clean it up */
: > fclose(wfp);
: > fclose(rfp);
: You're assured to get this done faster by using the system()
: command to execute operating system commands directly. The actual
: commands will vary based on the O/S under which this runs, but might
: be something like this under Unix:
: char str[100];
: sprintf( str, "rm -f %s", argv[i] );
: system( str );
: sprintf( str, "mv write_temp %s", argv[i] );
: system( str );
This is already commented on later in this threat (by Jari Kokko,
Michael Shields and others), but to make it VERY clear, Mike is
completly wrong:
1) system(3) is VERY expensive: it calls fork(2), exec(2).
2) rename(3)/remove(3) or link(2)/unlink(2) (these are used
to implement rename/remove in libc on Unix) is the way to
do it in C on Unix-Systems.
[whether system does fork and exec or not is, of course, not part of C,
but in the original case, it's almost true; actually, it does fork, exec,
fork, and exec, because the called shell has to then find the command,
fork it, and exec it. -mod]
Sorry for the critisism... or am I too narrow minded?
Yours,
Elmar Bartel, RBG.
--
preferred: bar...@informatik.tu-muenchen.de
secondary: fax: +49 89 2105-8232 or phone: -2025
Last resort: Institut fuer Informatik, TU Muenchen,
Arcisstrasse 21, D-80333 Muenchen, Germany
... frau kann durch einen deutschen Artikel nicht diskriminiert werden.
Nick Maclaren
|> In <3m7hla$4...@solutions.solon.com> trin...@news.ed.ray.com (Scott Tringali) writes:
|> >ga...@oak.circa.ufl.edu wrote:
|> >: ** added "printf("Processing file...);" down yonder _OR_ is it
|> >: ** because I have several flags && therefore several ints to check
|> >: ** each time a new character is read???
|> >
|> >
|> >: while((c=getc(rfp))!=EOF) {
|> >
|> >your processing on the buffer.
|>
|> Why? stdio will do the block read anyway. Hint: stdio has its own
|> internal buffering. And getc() is a macro defined in <stdio.h> in most
|> implementations, so it doesn't even have the overhead of a function call
|> (unless the programmer explicitly bypasses the macro and uses the function
|> from the standard library).
What is more, some run-time systems implement fread() by a loop of
getc()s anyway! The C standard says that the behaviour should be
"as if" this were the case, and many systems take the easy path to
full conformance ....
|> When using stdio functions, whether getc() or fread(), read() is called
|> by stdio, usually with the same buffer size. So, whether the user reads
|> the file with getc() or with fread(), the same number of read() calls
|> are issued by stdio. Which means that the above quoted table is
|> completely irrelevant to our discussion. Showing it to a newbie,
|> without adequate comments, in order to support a bogus claim/advice,
|> is extremely unethical.
Yes, quite, but remember that read() is the primitive I/O call
for UNIX systems only - POSIX conformance is irrelevant here.
Other systems will have a different underlying call (i.e.
read() will be replaced by get_block() or whatever). But I
100% support your point on getc() and fread() - this will apply
almost whichever system you are using, because all serious C
implementations implement getc() as a macro.
The only real efficiency advantage to reading a block and unpicking
it yourself if if you are the sort of programmer who habitually
writes better code than the authors of <stdio.h>. Very, very few
newbies will fall into this category :-)
Nick Maclaren,
University of Cambridge Computer Laboratory,
New Museums Site, Pembroke Street, Cambridge CB2 3QG, England.
Email: nm...@cam.ac.uk
Tel.: +44 1223 334761 Fax: +44 1223 334679
Anders Juul Munch
>But another point, why I'm jumping in in this thread are the lot's of
>irritating statements and wrong suggestions, two examples:
>=======================================================================
>Henry C. Schoepp (h...@eel.ufl.edu) wrote:
>(References: <3m181e$d...@solutions.solon.com> <3m6g7l$8...@solutions.solon.com>)
>: You're doing things one character at a time.
>: This is slow and disk intensive.
>: Think about allocating a large buffer, possibly the size of the file
>: and using a fread() I/O to get a large block.
>[ ... ]
>Exactly this done when you use getc/putc. The standard libc allocates
>a buffer, fills this buffer with one system call. getc and putc do
>their job on this buffer and the system get only called when this
>buffer gets empty/full. The influence of the buffersize on the
>execution time is marginal if the is about 1K or so.
>(See also William Kaufman reply)
Good thing you pointed out what was really slowing things down in Gary's
program (a printf for each character read). But you are a little too
hasty in putting down a sound piece of advice here.
Is 5 times slower what you would call "marginal" ?
Your description of what getc/putc do is fine, but the bottom line is that
getc/putc is *much* slower than larger-block fread/fwrite. This is my
experience on both ms-dos and unix. You can try it out by writing a file
copying program. Well, actually I did this for you.
File copying using getc/putc, adding setbuf() buffering (or else it would
be even slower):
File SLOWCP.C
#include <stdio.h>
#include <assert.h>
int main(int argc, char** argv)
{
FILE* in;
FILE* out;
int c;
char inbuf[BUFSIZ];
char outbuf[BUFSIZ];
assert(argc == 3);
in = fopen(argv[1], "r");
out = fopen(argv[2], "w");
assert(in != NULL && out != NULL);
setbuf(in, inbuf);
setbuf(out, outbuf);
while((c = getc(in)) != EOF)
{
putc(c, out);
}
fclose(out);
fclose(in);
return 0;
}
Compare it to this fread/fwrite file copier:
File FASTCP.C
#include <stdio.h>
#include <assert.h>
#define BUFSIZE 10000
int main(int argc, char** argv)
{
FILE* in;
FILE* out;
int c;
size_t bytes_read;
char buf[BUFSIZE];
assert(argc == 3);
in = fopen(argv[1], "r");
out = fopen(argv[2], "w");
assert(in != NULL && out != NULL);
while(
(bytes_read = fread(buf, 1, BUFSIZE, in)) != 0)
{
fwrite(buf, 1, bytes_read, out);
}
fclose(out);
fclose(in);
return 0;
}
On the system that I am logged on to right now (a HP-UX system, programs
compiled using gcc with full optimization, and tried out on a 700K file),
SLOWCP.C is about 5 times slower than FASTCP.C. (And the system `cp'
command is even faster than that, for whatever reason that may be.)
Of course once you start doing actual, time-consuming work on the
individual characters read, this speed difference may become
insignificant. But having an entire file in memory as a long string gives
other performance benefits: You can use library functions such as strchr()
and memchr() to scan quickly through the string to find the parts of
interest, and you can access surrounding characters easily using pointer
arithmetic.
One thing to remember on ms-dos is to open files in binary mode, or
fread/fwrite will be as slow as getc/putc.
Anders Munch | Department of Computer Science
ju...@diku.dk | University of Copenhagen, Denmark
Still confused but at a higher level
Chris Torek
<ju...@diku.dk> wrote:
>Your description of what getc/putc do is fine, but the bottom line is that
>getc/putc is *much* slower than larger-block fread/fwrite.
It is hard to tell just how much, actually (and it varies a lot).
>This is my experience on both ms-dos and unix. You can try it out
>by writing a file copying program. Well, actually I did this for you.
[example programs deleted]
% time ./slowcp xxx out
0.2u 0.2s 0:00.50 84.0% 0+0k 0+39io 0pf+0w
% time ./slowcp.nosetbuf xxx out
0.2u 0.1s 0:00.43 79.0% 0+0k 0+34io 0pf+0w
% time ./fastcp xxx out
0.0u 0.0s 0:00.20 70.0% 0+0k 0+34io 0pf+0w
% ls -l xxx
-rw-r--r-- 1 torek bsdi 755039 Apr 22 20:40 xxx
The `fast' version (using fread/fwrite) is definitely faster, but
not `5 times'---the elapsed time goes from about .5 seconds to
about .2, while the time measured by the system goes from `about
none' (anything less than .2 or .3 is suspect) to `about none';
clearly we need to copy a larger file.
Making xxx 4 times larger:
% time ./slowcp.nosetbuf xxx out
0.9u 0.4s 0:01.44 94.4% 0+0k 0+90io 0pf+0w
% time ./fastcp xxx out
0.1u 0.3s 0:00.60 90.0% 0+0k 0+86io 0pf+0w
If anything, it looks like the `fast' version is about 4.25 times
faster on this system, with almost all the savings being in user
time (as one could expect).
More interesting, I think, is this claim:
>File copying using getc/putc, adding setbuf() buffering (or else it would
>be even slower) ...
Note that I sped up the program by removing the setbuf() calls. If
setbuf() calls speed up your tests, you probably have a poor stdio;
stdio should always choose a good buffer size:
% time ./slowcp xxx out
0.6u 1.0s 0:01.74 94.2% 0+0k 0+111io 0pf+0w
Although the user time has been reduced slightly (from .9 to .6
seconds), the system time has increased to more than make up for
this (from .4 to 1.0 seconds).
>On the system that I am logged on to right now (a HP-UX system, programs
>compiled using gcc with full optimization, and tried out on a 700K file),
>SLOWCP.C is about 5 times slower than FASTCP.C. (And the system `cp'
>command is even faster than that, for whatever reason that may be.)
% time cp xxx out
0.0u 0.3s 0:00.45 86.6% 0+0k 0+86io 0pf+0w
As might be expected, the system cp uses the same amount of system
time as a properly-buffered stdio, but avoids some of the user overhead
involved in fread/write.
>Of course once you start doing actual, time-consuming work on the
>individual characters read, this speed difference may become
>insignificant.
This is also true. Optimization requires considering the whole problem;
focusing on one small aspect will lead to surprises.
--
In-Real-Life: Chris Torek, Berkeley Software Design Inc
Berkeley, CA Domain: to...@bsdi.com +1 510 549 1145
Network News
>On a slightly related topic, I would be very interested in seeing some
>examples of C or C++ programs that do things otherwise usually done in
>the shell.
>
> discussion, and related to some common issues faced by C programmers. -mod]
First I'll give my general hints on this kind of issue, then an interesting
example.
The Unix system is designed such that a great deal can be accomplished without
resorting to a general-purpose programming language, and one would be a fool
not to take advantage of that. Furthermore, the pipeline nature of shell
constructs allows complex tasks to be easily performed by combining a number of
simple programs. Therefore:
1. Do it with existing tools if reasonably practicable;
2. If you have to use C, make the C as general as possible, so that it fits
into a pipeline appropriately.
I.e. don't make the C do everything when some tasks are neatly seperable.
Rather than describe a current project, which has a makefile call a shell
script that calls a C program whose output goes into an application program
(being tested), whose output goes into another shel script which filters input
to a display engine which feeds commands to X, I'll describe a very simple
utility. It became necessary at some point to fire off a backgrounded process
for each file found by a find(1) command. find -exec won't do this. It could
be done by calling a small shell script that would background the process and
exit, but it is rather wasteful to start up a shell merely to do this. I came
up with a C one-liner that would fork(2) off a specified program and then
terminate, which could be called by find(1):
main(c,v)int*v;{fork()||execvp(*++v,v);}
This is the minimal form. Admittedly, it relies on sizeof(int)==sizeof(char*),
but making it portable to systems where this is not the case only makes it two
characters longer:
main(c,v)char**v;{fork()||execvp(*++v,v);}
And for those that like a bit off error checking,
main(c,v)char**v;{!*++v?exit(2):(c=fork())?exit(c<0):execvp(*v,v);}
will exit 0 on success, 1 if the fork(2) fails, and 2 if it isn't passed any
command-line arguments. Note that this utility is very general; it will
execute any program, with any arguments, not just the one that I needed when
I wrote this.
[and this is as far out as this one goes. followups to comp.unix.programmer.
-mod]
-zefram
Lawrence Chee
I am looking at all these posts and I am amused. I see
some great suggestions (replacing system calls with the
better remove and rename functions). There are many
ways to improve the efficency of the code at varying
levels of cost. Perhaps, the fastest code would be to
rewrite the whole thing in assembler at the expense of
great programming time. A more cost effective way
would be to reconsider the algorithm, and to reimple-
ment the new algorithm using more efficent function
calls. But this is not my problem and I have very few
ideas about the intentions of the code poster to be
able to read their mind. So I will suggest two things:
1) remove the printf from the loops, sending
output to the screen is costly in terms of time,
2) allocate a large buffer - say 1 Mb, insert
a function call to setvbuf, and free this buffer
after the fclose.
The second suggestion requires the fewest changes to
the code, it should improve the I/O performance, and it
is portable to most operating systems. There are more
optimizations which can be made to the code but without
more information (and motivation) I would hestitate to
suggest more.
Good luck,
Lawrence D. Chee
lc...@csc.com