buffered io.ReaderAt

[Sebastien Binet]

hi there,

I know about bufio.Reader that provides a buffered version of an io.Reader.

Has anybody written a buffered version of an io.ReaderAt ?

I have this binary file format that implicitly forces decoders of said format to fetch data using ReadAt.

In C/C++, that's OK because files are open already buffered.

But when I compare the performances of my pure-Go based decoder with the C++ one, I see:

=== ROOT ===

real=4.27 user=3.63 sys=0.30 CPU=92% MaxRSS=305920 I/O=13656/0

real=4.33 user=3.95 sys=0.30 CPU=98% MaxRSS=306248 I/O=72/0

real=3.99 user=3.72 sys=0.27 CPU=100% MaxRSS=309516 I/O=0/0

real=3.92 user=3.65 sys=0.27 CPU=100% MaxRSS=306340 I/O=0/0

=== go-hep/rootio ===

real=8.37 user=8.11 sys=0.26 CPU=100% MaxRSS=40980 I/O=0/0

real=8.28 user=8.03 sys=0.26 CPU=100% MaxRSS=41004 I/O=0/0

real=8.41 user=8.16 sys=0.25 CPU=100% MaxRSS=40972 I/O=0/0

real=8.37 user=8.12 sys=0.26 CPU=100% MaxRSS=40976 I/O=0/0

and in the profile I see (*os.File).ReadAt (and then the according syscall)

so I'd need:

- either a way to open *os.File in buffered mode

- or tack a buffered io.Reader+io.ReaderAt

cheers,

-s

[Jan Mercl]

On Sat, Sep 16, 2017 at 1:52 PM Sebastien Binet <seb....@gmail.com> wrote:

> Has anybody written a buffered version of an io.ReaderAt ?

IIUC what's needed, you might want to try https://github.com/cznic/file/blob/efffc965e97bbf31b0ed00deb82fe697500f22b5/file.go#L1039

--

-j

[Sebastien Binet]

ah! I thought I had scrutinized your repos for such a thing (you usually have quite the gems over there) but overlooked that one.

thanks.

sadly, the improvement wasn't impressive (probably a testimony to the performances of the linux filesystem layer...)

copying the whole file's content into a []byte and building my own reader+readerat on top of that got me the same perfs than with your libs.

now, my biggest time consumers are:

ROUTINE ======================== go-hep.org/x/hep/rootio.(*LeafD).readBasket in /usr/lib/go/src/encoding/binary/binary.go

     3.25s      3.25s (flat, cum) 18.83% of Total

         .          .    121:}

         .          .    122:

         .          .    123:func (bigEndian) Uint64(b []byte) uint64 {

         .          .    124: _ = b[7] // bounds check hint to compiler; see golang.org/issue/14808

         .          .    125: return uint64(b[7]) | uint64(b[6])<<8 | uint64(b[5])<<16 | uint64(b[4])<<24 |

     3.25s      3.25s    126: uint64(b[3])<<32 | uint64(b[2])<<40 | uint64(b[1])<<48 | uint64(b[0])<<56

         .          .    127:}

which compiles down to:

      70ms       70ms     4f028d: ANDQ SI, BX                             ;go-hep.org/x/hep/rootio.(*LeafD).readBasket rbuffer.go:307

         .          .     4f0290: CMPQ $0x7, DX                           ;binary.go:124

         .          .     4f0294: JBE 0x4f02e1

         .          .     4f0296: MOVQ 0(DI)(BX*1), DX                    ;binary.go:126

     3.25s      3.25s     4f029a: BSWAP DX                                ;go-hep.org/x/hep/rootio.(*LeafD).readBasket binary.go:126

      90ms       90ms     4f029d: MOVQ DX, 0x38(SP)                       ;go-hep.org/x/hep/rootio.(*LeafD).readBasket rbuffer.go:308

so probably not much to be gained here...

the profile is here:

 https://cern.ch/binet/go-hep/rootio.cpu.pprof

 https://cern.ch/binet/go-hep/read-data.go

(if anyone's interested in making that program as fast or even faster than the C++ one... it's for science! :P)

-s