proposed changes to http server

[jesse.dailey]

I am working with the http server right now, and would like to make a
couple changes, one of them completely transparent, one of them not,
and I'd like to discuss the idea with people first.

The first, transparent one. It's very difficult / impossible to test
http.ListenAndServe right now because there is no way to use it
asynchronously. First, even if you launch "go http.ListenAndServe
(...)", you have no way of knowing when it is ready to accept
connections, and then once it is up, you have no way to signal it to
go down, and if you did, then no way to know that it is down, you just
have to sys.Exit() the whole process.

I would like to propose three new functions:
http.Stop()
http.WaitForReady()
http.WaitForStop()

These would be implemented with 3 hidden channels: (ready, done, exit
chan bool)
Stop() writes to the exit channel
WaitForReady() reads from the ready channel
WaitForDone() reads from the done channel

The http server's accept loop would signal ready before the loop
begins, and done when it breaks, and the loop body would be modified
to use "select" to read either newly Accept()ed connections, or an
exit message from the exit channel. The ready and done channels would
support a buffer of 1, so it wouldn't matter if you read them or not
(keeping things transparent if you don't want to use the async
behavior).

The second, non-transparent, change (but not impactful of existing
code). I'd like to expose http.NewConn().

The reason is this: I want to create a http.Handler that dispatches
requests to remote FastCGI responders (which share the same ServeMux
interface with the http server). What this allows you to do, is take
any existing http.Handler and set it to run behind a FastCGI server
instead of HTTP, then the HTTP server uses a special handler that just
marshals the requests back and forth using the FCGI protocol, and to
the user this is all nicely transparent. To do this, the FastCGI
responder needs to be able to create a fake http.Conn to pass to the
handler, one that when written to, actually engages with the FCGI
protocol to get data back to the http server.

This requires two changes: first, that http.newConn simply be exposed
as NewConn, and instead of taking a *net.Conn it needs to just take a
ReadWriteCloser. The rwc*net.Conn is already saved and used
everywhere as a ReadWriteCloser, it only uses the net.Conn data for
one thing: to get the RemoteAddr of the connection. This just needs
to be made a separate argument to NewConn.

func NewConn(rwc io.ReadWriteCloser, handler http.Handler, RemoteAddr
string)

Neither of these changes would impact anything outside of 'http',
including any existing code that uses it. I'd gladly contribute the
code for these changes, and am eager to hear people's thoughts.

-Jesse.

[Russ Cox]

On Sun, Jan 31, 2010 at 11:09, jesse.dailey <jesse....@gmail.com> wrote:

I am working with the http server right now, and would like to make a
couple changes, one of them completely transparent, one of them not,
and I'd like to discuss the idea with people first.

The first, transparent one.  It's very difficult / impossible to test
http.ListenAndServe right now because there is no way to use it
asynchronously.  First, even if you launch "go http.ListenAndServe
(...)", you have no way of knowing when it is ready to accept
connections, and then once it is up, you have no way to signal it to
go down, and if you did, then no way to know that it is down, you just
have to sys.Exit() the whole process.

I would like to propose three new functions:
http.Stop()
http.WaitForReady()
http.WaitForStop()

These are unnecessary.  Look at the tests in the src/pkg/gob directory

for an example of how to start an http server.  If you arrange to share

the net.Listener value, another goroutine can call l.Close() to knock it over,

and then you'll know that it's done when http.Serve returns.

The second, non-transparent, change (but not impactful of existing
code).  I'd like to expose http.NewConn().

Is that really all that's necessary?  If so, great, but the last time I looked

at the spec it seemed like you'd need to do a bit more.

Russ

[jesse.dailey]

> > http.Stop()
> > http.WaitForReady()
> > http.WaitForStop()
>
> These are unnecessary.  Look at the tests in the src/pkg/gob directory
> for an example of how to start an http server.  If you arrange to share
> the net.Listener value, another goroutine can call l.Close() to knock it
> over, and then you'll know that it's done when http.Serve returns.

Great, I think I see how that's intended to work now.

> The second, non-transparent, change (but not impactful of existing
>
> > code).  I'd like to expose http.NewConn().
>
> Is that really all that's necessary?  If so, great, but the last time I
> looked at the spec it seemed like you'd need to do a bit more.

As long as I can build up a ReadWriteCloser of my own to pass in as
the I/O for the http.Conn then I can make it work. I have a working
fcgi implementation already, and a branch of it mostly ported to this
structure... having some trouble with binary.Read/Write right now, but
I am confident that the basic idea will work once I iron out the
kinks.

There are a few parts of the spec that I think won't be utilized doing
it this way, for example, there is no correspondence between anything
in http.Request and FCGI_DATA, so no FCGI_DATA will be sent to a
responder, and there is no specific way to send error messages back
over a http.Conn that differentiates it from regular content, so it
only writes to FCGI_STDOUT and not FCGI_STDERR. Maybe there are
clever ways to fold all these things in later, but from what I can
tell it should be functional with just this change.

Jesse.

[jesse.dailey]

After reviewing the way the Listeners were working inside http, I was
able to accomplish all this without needing to make any of these
changes to http.

I have created a net.Listener whose Accept() returns a net.Conn that
hides the FastCGI protocol from the http internals... allowing the
http server to transparently engage with a FastCGI responder.

Similarly on the other side, if you want the http server to talk to a
FastCGI responder, I have also created an http.Handler that will do
round-robin dispatch to a group of FastCGI responders.

It is documented and has unit tests for the basic vertical
functionality, and I'd like to submit this for review for "pkg/http/
fcgi".

Before I do that, any comments on the design?

Here are the two most basic examples:

A webserver that sends all requests to a pool of FastCGI Responders.

http.Handle("/", fcgi.Handler([]string{
"127.0.0.1:7134",
"127.0.0.1:7135",
// ... repeat for each responder ...
}))
http.ListenAndServe(":80", nil)

A responder that answers requests from a webserver.

http.Handle("/hello", http.HandlerFunc(HelloServer))
if listen, err := fcgi.Listen("0.0.0.0:7134"); err == nil {
http.Serve(listen, nil)
}

Or, the shorter version of the same thing if you dont need to have
the listener around to call .Close() on (which kills the http server):

http.Handle("/hello", ...)
fcgi.ListenAndServe(":7134", nil)

[Russ Cox]

I have created a net.Listener whose Accept() returns a net.Conn that

hides the FastCGI protocol from the http internals... allowing the
http server to transparently engage with a FastCGI responder.

Very cool.  I love it when interfaces work out.

I'd suggest sending out two separate CLs, one

for the server, and then once that is in, one for

the client.

The interface in your examples looks good, except

that I'm a little confused: I thought that fcgi servers

listened on fd 0, not on a specific TCP port?

Russ

[Jesse Dailey]

On Wed, Feb 3, 2010 at 1:18 AM, Russ Cox <r...@golang.org> wrote:

I'd suggest sending out two separate CLs, one

for the server, and then once that is in, one for

the client.

So I split my code out to separate this stuff into three files (the listener, the handler, the common protocol), so I can submit the listener and the handler separately.

But, one problem I'm having here is that I can't do an automated test of just one side of it (for one-sided tests to pass we would need a real webserver, and a real fcgi application).

-Jesse.