random access reads

19 views
Skip to first unread message

Shea Martin

unread,
Apr 25, 2005, 5:11:21 PM4/25/05
to
We have a software package which read data from (almost random locations) in
huge (100~400GB) files. The software was originally written using
fread/fwrite, because the original developer was concerned that
read(2)/write(2) were not part of the ANSI standard.

By replacing the fread(3) with read(2) we notice a significant performance
increase. I am assuming that this is because due to the random nature of
the reads, fread is buffering data on the assumption that freads are
sequential. While read(2) does not buffer (as much). Does anyone know if
there are any plans for read/write to be deprecated in Solaris (this seems
unlikely). Are they not part of the POSIX standard?

Is there a way to turn off buffering at an application level with either
fread(3) or read(2), in Solaris?

Or perhaps there is there a better tool that we are overlooking?

Thoughts, comments...?


Thanks,

~S

Rich Teer

unread,
Apr 25, 2005, 6:04:19 PM4/25/05
to
On Mon, 25 Apr 2005, Shea Martin wrote:

> By replacing the fread(3) with read(2) we notice a significant performance
> increase. I am assuming that this is because due to the random nature of
> the reads, fread is buffering data on the assumption that freads are
> sequential. While read(2) does not buffer (as much). Does anyone know if
> there are any plans for read/write to be deprecated in Solaris (this seems
> unlikely). Are they not part of the POSIX standard?

I'd say that theodds of read and write being dropped are 0. Yes, they're
mandated by POSIX.

> Is there a way to turn off buffering at an application level with either
> fread(3) or read(2), in Solaris?

Yes, see the setbuf man page. I also deal with the effects of I/O buffering
in my book, Solaris Systems Programming.

--
Rich Teer, SCNA, SCSA, OpenSolaris CAB member

President,
Rite Online Inc.

Voice: +1 (250) 979-1638
URL: http://www.rite-group.com/rich

Roland Mainz

unread,
Apr 25, 2005, 7:47:48 PM4/25/05
to
Shea Martin wrote:
> We have a software package which read data from (almost random locations) in
> huge (100~400GB) files. The software was originally written using
> fread/fwrite, because the original developer was concerned that
> read(2)/write(2) were not part of the ANSI standard.

Uhm... |read()| and |write()| are standard on all Unix/Linux versions
and will definately not go away in the forseeable future. AFAIK this are
the lowest level of userspace access to read+write to files and there is
nothing lower than than except writing your own syscalls... :)

BTW: When you make random read/write then |pread()| and |pwrite()| may
be better as you may save one syscall. An alternative solution may be to
|mmap()| the file (which may be the fasted solution when combied with
directio) or use async I/O (which may lower read/write latency) ...

----

Bye,
Roland

--
__ . . __
(o.\ \/ /.o) roland...@nrubsig.org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 7950090
(;O/ \/ \O;)

Dragan Cvetkovic

unread,
Apr 25, 2005, 7:55:06 PM4/25/05
to
Rich Teer <rich...@rite-group.com> writes:

> On Mon, 25 Apr 2005, Shea Martin wrote:

>> Is there a way to turn off buffering at an application level with either
>> fread(3) or read(2), in Solaris?
>
> Yes, see the setbuf man page. I also deal with the effects of I/O buffering
> in my book, Solaris Systems Programming.

And also man directio

Dragan

--
Dragan Cvetkovic,

To be or not to be is true. G. Boole No it isn't. L. E. J. Brouwer

!!! Sender/From address is bogus. Use reply-to one !!!

Thomas Dickey

unread,
Apr 25, 2005, 8:22:24 PM4/25/05
to
Rich Teer <rich...@rite-group.com> wrote:

> Yes, see the setbuf man page. I also deal with the effects of I/O buffering
> in my book, Solaris Systems Programming.

hmm - Stevens section 5.4 corresponds to Teer section 5.11

(this would make a nice example for side-by-side display on a webpage ;-)

--
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net

Casper H.S. Dik

unread,
Apr 26, 2005, 3:54:19 AM4/26/05
to
Shea Martin <sma...@arcis.com> writes:

>By replacing the fread(3) with read(2) we notice a significant performance
>increase. I am assuming that this is because due to the random nature of
>the reads, fread is buffering data on the assumption that freads are
>sequential. While read(2) does not buffer (as much). Does anyone know if
>there are any plans for read/write to be deprecated in Solaris (this seems
>unlikely). Are they not part of the POSIX standard?

No that will not happen; they're part of the POSIX standard.

Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

Shea Martin

unread,
Apr 26, 2005, 10:24:25 AM4/26/05
to
Thanks for all the suggestions.

~S

Reply all
Reply to author
Forward
0 new messages