buffered io.Pipe implementation?

[Brad Fitzpatrick]

Before I reinvent a wheel, has anybody written something like a buffered io.Pipe? I need one for HTTP/2.

I need Writes to buffer and not block (I can guarantee it's bounded before I do the Write), and

I need Reads to do their normal blocking thing like an io.Pipe.

I can use an io.Pipe for now, but it's not ideal.

As background:

The HTTP/2 code has one goroutine reading frames from the network, and then processing those frames. The DATA frame processing involve looking up which stream (HTTP request) they're associated with and writing it to that pipe (the reading side of which is an http.Request.Body)

But if I use an io.Pipe, an http.Handler is doing some incremental processing of their Request.Body, they can stall other handlers from reading their request bodies.

HTTP/2 already has negotiation of buffer sizes and flow control, so I can be sure clients won't be sending more than we've asked for (else they get errors). I'm not concerned about buffers going crazy. i.e. there's already pushback. I just need a buffer that's the size we've advertised (or growing up to the advertised size), so just that stream being slow with their reads gets pushback all the way to the sender, without stalling the goroutine that's reading frames for other http.Handlers.

If it's possible to compose such a beast using primitives already in the standard library, it's either not obvious, or I'll blame sleepiness. It's also not much code, but effectively forking http://golang.org/src/pkg/io/pipe.go and making it even more complicated isn't my first choice if there's an easier or cleaner way.

[Gustavo Niemeyer]

This might be helpful:

https://codereview.appspot.com/4969056/

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Keith Rarick]

I ran into exactly the same thing with SPDY:

https://github.com/kr/spdy/blob/master/spdyframing/pipe.go
https://github.com/kr/spdy/blob/master/spdyframing/buffer.go

[Han-Wen Nienhuys]

sounds very similar to the channel dispatch for SSH. SSH uses

https://code.google.com/p/go/source/browse/ssh/buffer.go?repo=crypto#167

It's not exported, but you're welcome to copy & paste or figure out a
shared place for this code.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
Han-Wen Nienhuys
Google Munich
han...@google.com

[Dave Cheney]

Does Atwood's rule of threes come into effect at this point ?

[Brad Fitzpatrick]

That Tee doesn't seem to do what I need. I would expect to see a sync.Cond and for Read to use it, waiting for new data.

[Brad Fitzpatrick]

Keith, looks great.

Permission to use that? If so, either I can copy it, or you could send a PR to github.com/bradfitz/http2 with the tests as well, but without integrating it yet.

[Keith Rarick]

Certainly you can use it. PR coming up.