Is anyone aware of a blocking ring buffer implementation?

[Marcin Romaszewicz]

Hi All,

Before I reinvent the wheel, and because this wheel is particularly tricky to get right, I was wondering if anyone was aware of a a library providing something like this

- conforms to io.Reader

- conforms to io.Writer

- Contains a buffer of fixed size, say, 64MB. If you try to write when the buffer is too full, write blocks. When you try to read from an empty one, read blocks.

This describes the behavior of make(chan byte, 64 * MB), however, it doesn't seem to be efficient to do this with a channel. Say I'm transferring a few hundred GB via this mechanism. A chan of byte would need a few hundred billion byte writes, and a few hundred billion reads. Doesn't sound like it could be efficient at all. You can't implement this with a channel of []byte, because you'd violate your buffering limit if you pass too large of a byte array, so you'd have to chop things up into blocks, but maybe that's simpler than a full fledged blocking ring buffer.

Anyhow, I've implemented such things in various OS level plumbing, so I know I can do it in Go much more easily, just hoping to avoid it :)

Thanks for any advice,

-- Marcin

[Robert Engels]

Just use a pipe. Part of the std lib. 

On Nov 21, 2019, at 9:47 PM, Marcin Romaszewicz <mar...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CA%2Bv29LsvbVU1oxBYz6wKO50-xNVPQ6-wRa7Dvhq4uY%3Du-FKigA%40mail.gmail.com.

[Kurtis Rader]

On Thu, Nov 21, 2019 at 7:57 PM Robert Engels <ren...@ix.netcom.com> wrote:

Just use a pipe. Part of the std lib.

I haven't written any production Go code but am a grey beard. Looking at the docs, https://golang.org/pkg/io/#Pipe, a io.Pipe provides somewhat different semantics compared to a traditional ring buffer. Specifically with respect to when "writes" block. Am I misunderstanding something? Substituting an io.Pipe may be fine for the O.P.s specific situation but in general would not be AFAICT.

--

Kurtis Rader

Caretaker of the exceptional canines Junior and Hank

[Dan Kortschak]

There is this: https://godoc.org/bitbucket.org/ausocean/utils/ring

It has been used in production fairly extensively.

[Robert Engels]

The OP specifically requested io.Reader/Writer interfaces.

A pipe is what he wants. Not a ring buffer. (A pipe essentially has a ring buffer in the implementation though).

> On Nov 21, 2019, at 10:20 PM, Dan Kortschak <d...@kortschak.io> wrote:
>
> There is this: https://godoc.org/bitbucket.org/ausocean/utils/ring

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/152a954eaa3eea02514e71bb142904480241ad6c.camel%40kortschak.io.

[Robert Engels]

The package Dan referenced will work too - but it doesn’t look different than using a pipe with a large write buffer.

> On Nov 21, 2019, at 10:20 PM, Dan Kortschak <d...@kortschak.io> wrote:
>

> There is this: https://godoc.org/bitbucket.org/ausocean/utils/ring

[Robert Engels]

Sorry in reading the implementation of a pipe in Go, it may not have the desired semantics, especially with a large write buffer since writes and reads are 1 to 1 (very weird impl - this is not a traditional pipe). Dan’s library is a better choice.

> On Nov 21, 2019, at 10:37 PM, Robert Engels <ren...@ix.netcom.com> wrote:
>
> The OP specifically requested io.Reader/Writer interfaces.

[Robert Engels]

Not sure why the go pipe doesn’t take a size for the inner channel to make the synchronous nature optional.

> On Nov 21, 2019, at 10:44 PM, Robert Engels <ren...@ix.netcom.com> wrote:
>
> Sorry in reading the implementation of a pipe in Go, it may not have the desired semantics, especially with a large write buffer since writes and reads are 1 to 1 (very weird impl - this is not a traditional pipe). Dan’s library is a better choice.

[Jamil Djadala]

On Thu, 21 Nov 2019 22:37:37 -0600
Robert Engels <ren...@ix.netcom.com> wrote:

> The OP specifically requested io.Reader/Writer interfaces.
>
> A pipe is what he wants. Not a ring buffer. (A pipe essentially has a
> ring buffer in the implementation though).

from https://golang.org/pkg/io/#Pipe :

The data is copied directly from the Write to the corresponding Read
(or Reads); there is no internal buffering

and looking in implementation, it seems that internally Pipe is using
channel.

--
Jamil Djadala

[Robert Engels]

Yes, I already addressed this but I know people often reply as they read.

This is not really a pipe in the traditional posix sense. A bad name, or bad implementation imo.

> On Nov 21, 2019, at 10:53 PM, Jamil Djadala <dja...@datamax.bg> wrote:
>
> On Thu, 21 Nov 2019 22:37:37 -0600

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/20191122065315.78316d02%40wolf.home.

[Marcin Romaszewicz]

I am in fact replacing an io.Pipe implementation, because I need to buffer some data. io.Pipe doesn't buffer, it just matches up read and writes with each other.

What I'm effectively doing is producing chunks of data at a certain rate from one server, and forwarding them to another. Due to the latency of issuing the read from the server, I need to do some pre-fetch into a buffer, but I don't want to prefetch to much.

With the pipe implementation, my reader had many stalls waiting for the server to produce the next  chunk. The server can produce data a lot quicker than the reader can read, however, setup time is high. It's a complicated mess :)

[Robert Engels]

I would just copy and paste the stdlib pipe.go and change the ctor to take a size for the channel. I’m pretty sure that is all that is needed to fix the problem. 

On Nov 21, 2019, at 11:18 PM, Marcin Romaszewicz <mar...@gmail.com> wrote:



[Marcin Romaszewicz]

To give a little bit more background on what I was doing - I have an API endpoint which is used to download large amounts of data that is backed by S3 in AWS. Using an io.Pipe to simply proxy from S3 tops out at around 30MB/sec due to single threaded S3 performance and setup time. I wrote an adapter around it which downloads increasingly more S3 chunks in parallel, until my producer rate starts to exceed the consumer rate - now I'm getting upwards of a 1GB/sec on really large files. The only tricky bit is sequencing chunks for feeding into the io.Pipe, but there's few of them, so an array with a mutex around it works just fine. I didn't have to write the ring buffer in the end.

Working in Go is a joy at times like this - having a multi threaded chunked fetcher hiding behind a single threaded HTTP GET all the while not buffering too much data is a fiendishly hard thing to do in other languages due to the nature of their threading libraries and the sparse library support for asynchronous primitives. I've done stuff like this in the past in the IRIX kernel (don't laugh), or in C/C++ and pthreads, and each of those would have been days of work to get right instead of half a day of writing simple code to glue together available library functions.

-- Marcin

[adam.a...@10xgenomics.com]

You could always use os.Pipe (which unlike io.Pipe is a standard posix pipe), and wrap one end or the other with bufio.Reader/Writer.  Or if all you want to do is buffer the response from http.Get you can just wrap the http response reader with bufio.Reader.  Or am I missing something here about this use case?

> To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.

[Jason E. Aten]

https://github.com/glycerine/rbuf

very hardened. Implements Reader and Writer. You would have to add your desired blocking strategy.