random access reads

[Shea Martin]

We have a software package which read data from (almost random locations) in
huge (100~400GB) files. The software was originally written using
fread/fwrite, because the original developer was concerned that
read(2)/write(2) were not part of the ANSI standard.

By replacing the fread(3) with read(2) we notice a significant performance
increase. I am assuming that this is because due to the random nature of
the reads, fread is buffering data on the assumption that freads are
sequential. While read(2) does not buffer (as much). Does anyone know if
there are any plans for read/write to be deprecated in Solaris (this seems
unlikely). Are they not part of the POSIX standard?

Is there a way to turn off buffering at an application level with either
fread(3) or read(2), in Solaris?

Or perhaps there is there a better tool that we are overlooking?

Thoughts, comments...?

Thanks,

~S

[Rich Teer]

On Mon, 25 Apr 2005, Shea Martin wrote:

> By replacing the fread(3) with read(2) we notice a significant performance
> increase. I am assuming that this is because due to the random nature of
> the reads, fread is buffering data on the assumption that freads are
> sequential. While read(2) does not buffer (as much). Does anyone know if
> there are any plans for read/write to be deprecated in Solaris (this seems
> unlikely). Are they not part of the POSIX standard?

I'd say that theodds of read and write being dropped are 0. Yes, they're
mandated by POSIX.

> Is there a way to turn off buffering at an application level with either
> fread(3) or read(2), in Solaris?

Yes, see the setbuf man page. I also deal with the effects of I/O buffering
in my book, Solaris Systems Programming.

--
Rich Teer, SCNA, SCSA, OpenSolaris CAB member

President,
Rite Online Inc.

Voice: +1 (250) 979-1638
URL: http://www.rite-group.com/rich

[Roland Mainz]

Shea Martin wrote:
> We have a software package which read data from (almost random locations) in
> huge (100~400GB) files. The software was originally written using
> fread/fwrite, because the original developer was concerned that
> read(2)/write(2) were not part of the ANSI standard.

Uhm... |read()| and |write()| are standard on all Unix/Linux versions
and will definately not go away in the forseeable future. AFAIK this are
the lowest level of userspace access to read+write to files and there is
nothing lower than than except writing your own syscalls... :)

BTW: When you make random read/write then |pread()| and |pwrite()| may
be better as you may save one syscall. An alternative solution may be to
|mmap()| the file (which may be the fasted solution when combied with
directio) or use async I/O (which may lower read/write latency) ...

----

Bye,
Roland

--
__ . . __
(o.\ \/ /.o) roland...@nrubsig.org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 7950090
(;O/ \/ \O;)

[Dragan Cvetkovic]

Rich Teer <rich...@rite-group.com> writes:

> On Mon, 25 Apr 2005, Shea Martin wrote:

>> Is there a way to turn off buffering at an application level with either
>> fread(3) or read(2), in Solaris?
>
> Yes, see the setbuf man page. I also deal with the effects of I/O buffering
> in my book, Solaris Systems Programming.

And also man directio

Dragan

--
Dragan Cvetkovic,

To be or not to be is true. G. Boole No it isn't. L. E. J. Brouwer

!!! Sender/From address is bogus. Use reply-to one !!!

[Thomas Dickey]

Rich Teer <rich...@rite-group.com> wrote:

> Yes, see the setbuf man page. I also deal with the effects of I/O buffering
> in my book, Solaris Systems Programming.

hmm - Stevens section 5.4 corresponds to Teer section 5.11

(this would make a nice example for side-by-side display on a webpage ;-)

--
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net

[Casper H.S. Dik]

Shea Martin <sma...@arcis.com> writes:

>By replacing the fread(3) with read(2) we notice a significant performance
>increase. I am assuming that this is because due to the random nature of
>the reads, fread is buffering data on the assumption that freads are
>sequential. While read(2) does not buffer (as much). Does anyone know if
>there are any plans for read/write to be deprecated in Solaris (this seems
>unlikely). Are they not part of the POSIX standard?

No that will not happen; they're part of the POSIX standard.

Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

[Shea Martin]

Thanks for all the suggestions.

~S