Buffer vs. bytes.NewReader vs. bufio.NewReader?

[Jordan Wright]

Pardon the newbie question, but I'm having some difficulty grasping the idea of effective I/O.

In general, I get the idea of everything being just a stream of bytes to be manipulated, read, written, etc. As to how to do that effectively is what's avoiding me.

Is there a list of best practices on how to effectively use buffers vs. readers/writers?

When should I use bytes.NewReader instead of bufio.NewReader?

Is the only benefit of using a bytes.Buffer to be able to treat the underlying slice dynamically? 

Why would I want to use a bytes.Reader instead of a bytes.Buffer?

Quite a few questions, but any help at all would be greatly appreciated! I feel that the design idioms of Go are almost too elegant and straightforward that I'm missing something!

Thanks in advance,

Jordan

[Chris Kastorff]

> Is there a list of best practices on how to effectively use buffers vs.
> readers/writers?

It's really up to what you want to do. Buffering the IO when syscalls
are involved (network, files, pipes) is not a bad way to go though,
but sometimes it's nice to have the obvious semantics without any
buffering. (Do note that some libraries add a buffer for you, like
net/http.)

> When should I use bytes.NewReader instead of bufio.NewReader?

These do two fundamentally different things; bytes.NewReader creates a
Reader from a byte slice (that is, a chunk of memory you already have
in your program.) Useful if you want to pass a byte slice to some
other API that expects a Reader. bufio.NewReader on the other hand is
for wrapping existing Readers (usually ones whose Read method is
relatively expensive, like a TCP connection or a file) to coalesce a
bunch of easy-to-program small Read calls into a few larger ones.

> Is the only benefit of using a bytes.Buffer to be able to treat the
> underlying slice dynamically?

I'm not sure what you mean by this. The main help that a bytes.Buffer
gives you over a []byte is that it implements a bunch of methods, so
you can use it to pass a []byte to other libraries expecting a Reader,
or use it to construct a []byte with a library expecting a Writer.

> Why would I want to use a bytes.Reader instead of a bytes.Buffer?

A bytes.Buffer is a buffer with two ends; you can only read from the
start of it, and you can only write to the end of it. No seeking.
There's no need to use both sides though; I often use it to construct
a chunk of encoded stuff in memory (by passing it to serialization
libraries expecting a Writer) then pull out the []byte at the end
(with (*bytes.Buffer).Bytes()), then stop using the bytes.Buffer and
pass the []byte I got to another layer for more work.

A bytes.Reader is more explicitly only for turning a []byte into
something that's a Reader; since no modifications are possible, it can
also implement Seek and ReadAt without ambiguity, thus being more
useful to pass to other functions expecting a ReaderAt or a ReadSeeker
or whatever.

In general I use bytes.Buffer for writing to byte slices, and
bytes.Reader for reading from byte slices.

-Chris K

[Jordan Wright]

Thanks for the great response, Chris!

I often use it to construct 
a chunk of encoded stuff in memory (by passing it to serialization 
libraries expecting a Writer) then pull out the []byte at the end 
(with (*bytes.Buffer).Bytes()), then stop using the bytes.Buffer and 
pass the []byte I got to another layer for more work. 

Would I be correct in saying that this does not create another byte slice in memory and only keeps on copy of the bytes in memory the whole time? It's to my understanding that when I create a new Buffer using bytes.NewBuffer(b), the internal Buffer.buff is simply a pointer to b, not a copy. Is that the correct way to think about it?

Thanks again - that helped a ton.

[Chris Kastorff]

>> I often use it to construct
>> a chunk of encoded stuff in memory (by passing it to serialization
>> libraries expecting a Writer) then pull out the []byte at the end
>> (with (*bytes.Buffer).Bytes()), then stop using the bytes.Buffer and
>> pass the []byte I got to another layer for more work.
>
>
> Would I be correct in saying that this does not create another byte slice in
> memory and only keeps on copy of the bytes in memory the whole time? It's to
> my understanding that when I create a new Buffer using bytes.NewBuffer(b),
> the internal Buffer.buff is simply a pointer to b, not a copy. Is that the
> correct way to think about it?

Yes. It'll reuse the part of the underlying array for the slice you
gave it as long as it can (but it can allocate a new one if it needs
more space than the cap() of the original slice.)

Pedantically, the slice header is copied and the underlying array is
reused. Slices are pointer-ish types like channels and maps, being
implemented as a triplet of <pointer to first element>, <length>,
<capacity>, and those slice headers are copied by value when you pass
them as an argument. (This becomes important if you pass a []byte to a
function trying to modify its length, such as the built-in append();
you get a new slice header as a return value with a new length; the
argument you passed remains unchanged.)

-Chris K