Count lines in a file
John Siracusa
number of lines in a file. What I'm using now is,believe it
or not, is fseek() and friends...
-John
John Siracusa
: I'm looking for the fastest, most efficient way to count the
: number of lines in a file. What I'm using now is,believe it
: or not, is fseek() and friends...
..that's on AIX, BTW.
-John
David L Evens
Now about read the file character by character and count the number of
EOL markers you get?
--
---------------------------+--------------------------------------------------
Ring around the neutron, | "OK, so he's not terribly fearsome.
A pocket full of positrons,| But he certainly took us by surprise!"
A fission, a fusion, |
We all fall down! |
---------------------------+--------------------------------------------------
Lars Hecking
John Siracusa (maci...@bu.edu) wrote:
: John Siracusa (maci...@bu.edu) wrote:
: : I'm looking for the fastest, most efficient way to count the
: : number of lines in a file. What I'm using now is,believe it
: : or not, is fseek() and friends...
: ..that's on AIX, BTW.
Now about read the file character by character and count the number of
EOL markers you get?
Tedious. I use
FILE *fp;
char inBuf[BUFSIZ];
int lineCount = 0;
...
while (fgets (inBuf, BUFSIZ, fp))
lineCount++;
instead. BUFSIZ should do in most real-life apps.
--
Lars G. Hecking lhec...@nmrc.ucc.ie
GE d--- H+>H--- s:->s++ g+ p0>? !au a- w+ v+>v++ C(+++) US++ P+>+++
L>L++ 3- E+ N+ K- W-->--- M- V- -po+ Y+ t+ 5 !j R G? tv- b+ D--
B? e++++ u++(-) h* f r++ n+ y+*
John Siracusa
: Now about read the file character by character and count the number of
: EOL markers you get?
That's exactly what I'm doing. It's slow...:-/
-John
Greg Limes
you could use fgets().
you could use getchar() and compare with end-of-line.
you could use an efficient system utility :-)
but fseek() is not portably helpful in solving this one.
details of solutions left as an exercise for the student.
--
-- Greg Limes li...@3do.com, li...@netcom.com
"Your reality check is in the E-mail"
Not speaking for my employer, of course
PGP key available on request
Chris Torek
John Bickers <jbic...@templar.actrix.gen.nz> suggests:
>... Ideally, you want the largest [read] size you can get ...
On a system that contains some parallelism---such as a read-ahead
buffer in a disk drive---this is not the case.
Imagine a system in which a `read' of 8192 bytes causes something
to asynchronously read the *next* 8192 bytes, where this applies
only to the magic number 8192. Then the sequence:
main cpu secondary processing
-------- --------------------
wait for 8192 bytes | read 8192 bytes
think about them | read ahead 8192 bytes
ask for 8192 bytes | move pre-read bytes quickly
think about them | read ahead 8192 more bytes
will go considerably faster than:
main cpu secondary processing
-------- --------------------
wait for 16384 bytes | read 16384 bytes
think about them | idle
In the first example, the second 8192 bytes will be obtained
while the `thinking' goes on for the first 8192 bytes. Then the
next `ask for 8192 bytes' will return nearly instantaneously. The
second example will have to wait for all 16384 bytes, then think
about them, without being able to overlap the `think' time with
the second `read' time.
A bit more mathematically, the total time for example 1 is `wait
8192 + think 8192 + move 8192 + think 8192'; the total time for
example 2 is `wait 16384 + think 16384'. If `wait + move' < `2*wait',
example 2 will win.
The number 8192 is not necessarily correct for your system. (If
you have a cheap system, you may not have any read-ahead at all,
in which case overhead will dominate and John Bickers' answer will
work for you.)
--
In-Real-Life: Chris Torek, Berkeley Software Design Inc
Berkeley, CA Domain: to...@bsdi.com +1 510 549 1145
John Bickers
To accelerate this, read the file in larger chunks. Ideally, you
want the largest chunk size you can get, and the lowest overhead
IO functions available on your system. Generally the fastest IO
functions available will not be the ANSI C standard library ones,
but even with those tinkering with functions like setvbuf() can
make a big difference on some systems.
> -John
--
*** Count Templar, ELITE (FM-287) jbic...@templar.actrix.gen.nz ***
*** "Radioactivity - It's in the air, for you and for me" - Kraftwerk ***
David W. Murphy
Lawrence Kirby
>In article <jbicke...@templar.actrix.gen.nz>
>John Bickers <jbic...@templar.actrix.gen.nz> suggests:
>>... Ideally, you want the largest [read] size you can get ...
>
>On a system that contains some parallelism---such as a read-ahead
>buffer in a disk drive---this is not the case.
Also if reading/writing large buffers wipes your processor cache you could
see a big performance hit.
--
-----------------------------------------
Lawrence Kirby | fr...@genesis.demon.co.uk
Wilts, England | 7073...@compuserve.com
-----------------------------------------
John Bickers
> In article <jbicke...@templar.actrix.gen.nz>
> John Bickers <jbic...@templar.actrix.gen.nz> suggests:
> >... Ideally, you want the largest [read] size you can get ...
>
> On a system that contains some parallelism---such as a read-ahead
> buffer in a disk drive---this is not the case.
I use asynchronous IO myself when IO speed is a problem and the OS
provides async. functions easily enough. Then the best buffer size
tends to be a function of the amount of data processing you do and
how long you can afford to wait for an IO request to complete.
However, the only OS I know where it is easy to do this is the
Amiga's, so I tend not to mention it for general cases.
> In-Real-Life: Chris Torek, Berkeley Software Design Inc
Greg Black
The fastest most efficient way is to use the C code that somebody else
has already written for you and run `wc -l ...'
If you must write it yourself, the most sensible way is:
#include <stdio.h>
int main(void)
{
int c, nlines = 0;
while ((c = getchar()) != EOF)
nlines += c == '\n';
printf("%d\n", nlines);
return 0;
}
If you really need better performance than this gives then you should
tell us what you're doing and why it's too slow. Generally the only
people who need to improve on this code know quite well how to go
about it.
--
Greg Black -- g...@gba.oz.au
Stan Milam
: I'm looking for the fastest, most efficient way to count the
: number of lines in a file. What I'm using now is,believe it
: or not, is fseek() and friends...
: -John
The very fastest way is to do only one read of the file. This means you
must determine the size of the file, allocate enough memory to contain the
file, read the file with fread() and zoom through it counting all of the
'\n' characters. Here is an example which is a variation on a method I use
to read a text file with one read and return an array of pointers to the
beginning of each line. It is blindingly fast - provided you have enough
memory.... The example does a *minimum* of error checking.
/***************************************************************/
/* File Id. CHRCOUNT.C. */
/* Author: Stan Milam. */
/* Date Written: 17 Apr. 92. */
/* */
/* This function will count the occurances of a character that */
/* is contained within a string. */
/* */
/* Arguments: char *s1 - String to be searched. */
/* char ch - Character to search for. */
/* */
/* Returns: The number of times ch is in s1. */
/* */
/***************************************************************/
#include <string.h>
unsigned chrcount(char *s1, int ch) {
unsigned count = 0; /* The count to return */
char *wrk = strchr(s1, ch); /* Find first char in s1 */
while (wrk) { /* While we have matches */
count++, wrk++; /* Increment the count & pointer */
wrk = strchr(wrk, ch); /* Search for next occurance */
}
return count; /* Return the count */
}
------------------------------Cut Here--------------------------------
/***************************************************************/
/* File Id. FILELEN.C. */
/* Author: Stan Milam. */
/* Date Written: 5 May 92. */
/* */
/* This function will return the length of a file in bytes. */
/* */
/* Arguments: int fd - A file descriptor. */
/* Returns: long rv - Length of file in bytes. -1L if an */
/* error occurs. */
/* */
/***************************************************************/
#include <stdio.h>
#include <sys/types.h>
#ifdef _AIX
# include <unistd.h>
#else
# ifdef __TURBOC__
# include <io.h>
# endif
#endif
long filelength (int fd) {
long rv, here;
here = lseek(fd, 0L, SEEK_CUR); /* Save current address in file */
rv = lseek(fd, 0L, SEEK_END); /* Seek to the end of the file */
lseek(fd, here, SEEK_SET); /* Get back our orignal position */
return rv; /* Return length of file */
}
-------------------------------Cut Here--------------------------------
#include <stdio.h>
#include <stdlib.h>
long filelength (int fd);
unsigned chrcount(char *s1, int ch);
int main( void ) {
FILE *fp;
char *buffer;
long filesize;
unsigned linecount;
if ((fp = fopen("somefile", "r")) != NULL) {
filesize = filelength( fileno( fp ) );
if ((buffer = calloc( filesize, sizeof(char) )) != NULL) {
fread(buffer, sizeof(char), filesize, fp);
linecount = chrcount( buffer, '\n' );
printf("Lines = %u\n", linecount);
}
fclose(fp);
}
return 0;
}
Ramsey Dow
>John Siracusa (maci...@bu.edu) wrote:
>: I'm looking for the fastest, most efficient way to count the
>The very fastest way is to do only one read of the file. This means you
>must determine the size of the file, allocate enough memory to contain the
>file, read the file with fread() and zoom through it counting all of the
>'\n' characters. [...]
If you are on a UNIX system you may dispense with Stan Milam's FILELEN.C
program. Instead you can use the stat(2) system call to tell you the size
of the file in bytes. Briefly,
struct stat *sbufptr;
if (stat(filenameptr, sbufptr)) {
/* error; use errno to see what happened */
} else {
/* malloc a chunk of memory equal to
sizeof(char)*(int)sbufptr->st_size */
}
This should save you some time, but only if you're on a UNIX system.
Ramsey
--
Ramsey Dow Internet: d...@piedmont.mthcsc.wfu.edu Voicenet: 919.725.5432
GCS/MU -d+ p@ c++ !l u(++@) e+(*) m* s/- !n(---) h f g+ w+++ t(@) r+(@) y++
Nyarlathotep..the crawling chaos..I am the last..I will tell the audient void.
Stan Milam
: struct stat *sbufptr;
Yes, I am aware of the stat(2) call, however I have felt it is a pain to setup
and call when I already have an open file. I probablly should have set up the
filelength() function to be a wrapper for the stat call, but my DOS C compilers
all support filelength(), so I wrote a UNIX version to allow my code to be more
portable. As to speed, I do not think anyone could ever notice a difference!
Ralph Silverman
: maci...@bu.edu (John Siracusa) writes:
: #include <stdio.h>
--
*********begin r.s. response*********
i've written this problem myself too,
(somewhat differently, i'm sure).
i'd just like to say, the single
line body of your 'while' loop
is extremely slick!
*********end r.s. response***********
Ralph Silverman
z007...@bcfreenet.seflin.lib.fl.us