io.Copy buffer size

[Roger Pau Monné]

Hello,

I've looked at io.Copy code, and realized that the buffer used is
always 32KB (it is hard coded). I think this should be some kind of
parameter, since at least in my case, I'm always using 16KB blocks.

Regards, Roger.

[Russ Cox]

> I've looked at io.Copy code, and realized that the buffer used is
> always 32KB (it is hard coded). I think this should be some kind of
> parameter, since at least in my case, I'm always using 16KB blocks.

If the default behavior of io.Copy is not correct for
your particular reader or writer, you have three options:

1. Don't use it. It's a tiny amount of convenience code
and perfectly fine to duplicate if you really need to.

2. Implement the ReadFrom method on your Writer.

3. Implement the WriteTo method on your Reader.

Adding a block size parameter is unnecessary
complexity.

Russ

[Jonathan]

I agree, but splitting out a copy buffer function conveniently
captures a fungible bit of code and gives Roger what he wants without
complicating the copy functions.

Jonathan


func CopyBuffer(dst Writer, src Reader, buf []byte) (written int64,
err os.Error) {
nr, err := src.Read(buf)
if nr == 0 {
return 0, err
}
nw, ew := dst.Write(buf[0:nr])
if ew == nil && nw != nr {
ew = ErrShortWrite
}
if ew != nil {
err = ew
}
return int64(nw), err
}

func Copyn(dst Writer, src Reader, n int64) (written int64, err
os.Error) {
// If the writer has a ReadFrom method, use it to to do the copy.
// Avoids a buffer allocation and a copy.
if rt, ok := dst.(ReaderFrom); ok {
return rt.ReadFrom(LimitReader(src, n))
}
buf := make([]byte, 32*1024)
uncopied := n
var nc int64
for uncopied > int64(len(buf)) && err == nil {
nc, err = CopyBuffer(dst, src, buf)
uncopied -= nc
}
if uncopied > 0 && err == nil {
nc, err = CopyBuffer(dst, src, buf[0:uncopied])
uncopied -= nc
}
return n - uncopied, err
}

func Copy(dst Writer, src Reader) (written int64, err os.Error) {
// If the writer has a ReadFrom method, use it to to do the copy.
// Avoids an allocation and a copy.
if rt, ok := dst.(ReaderFrom); ok {
return rt.ReadFrom(src)
}
// Similarly, if the reader has a WriteTo method, use it to to do
the copy.
if wt, ok := src.(WriterTo); ok {
return wt.WriteTo(dst)
}
buf := make([]byte, 32*1024)
for err == nil {
var nc int64
nc, err = CopyBuffer(dst, src, buf)
written += nc
}
if err == os.EOF {
err = nil
}
return written, err
}

[Rob 'Commander' Pike]

That's a lot of similar not-quite-the-same functions with a lot of overlap. I like the fact that Copy doesn't take a buffer size, just like I like the fact that go doesn't take a stack size. A reasonable implementation is just that, reasonable, but it also frees up brain cells from worrying about arbitrary things made decisive.

Sometimes, though, the trivia matter. It might make sense to have a Copy variant that takes a buffer size, and have Copy just call ThatCopy(32K). Then there's no code duplication and I can continue not to care about my buffer size. You can care all you want, just stay out of my parameter lists.

-rob

[Jonathan]

That's what I am saying: the copy and copyn functions in my mail are
just rewrites of the io package functions of the same names that use
the CopyBuffer function on a buffer they create. Only the CopyBuffer
function is new. Your parameter lists are unsullied :).

Jonathan