DumpRequestOut from net/http/httputils fails to dump

[Cem Ezberci]

I have been trying to use DumpRequestOut function in net/http/httputil package for a small pet project. It seems like it fails to execute due to Scheme being an empty string in URL object that is part of the Request object.

I have put together a simple http server that I was able to test (I think) on my local with version 1.0.3 and latest tip version I pulled on April 4 (go version devel +e77430da3316 Thu Apr 04 16:36:10 2013 +1100 linux/amd64). It seems like Scheme field is not populated in either version. Is this a known bug or a bug at all?

http://play.golang.org/p/Rplem4q4Yy

When I run this and make a call to the server (http://127.0.0.1:8080/), I get the following output in stdout:

2013/04/08 13:39:04 Error while dumping request: unsupported protocol scheme ""
2013/04/08 13:39:04 Scheme:
2013/04/08 13:39:04 []

-Cem

[Brad Fitzpatrick]

This behavior is at least consistent with Transport: a Transport (the http Client) won't make a request without a scheme.

And the server handler's requests don't have a scheme (since it can be unknown via which listener a request arrived on).

-Cem

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Cem Ezberci]

Thanks, Brad. That makes sense.

I guess I should be more clear about what I am trying to do instead. The goal of this experiment is to see if I can build an http reverse proxy that accepts requests, compresses the entire request (to make sure everything is compressed including the headers -- applies to response too), streams out to the receiving end that decompresses the data, makes the actual http call to the origin that was intended for, retrieve the response, compress it and send it back to the initiator which then decompresses and passes it over to the original requester. I am looking for a sane way to pass these requests and responses between those two where http is not the protocol that I will be using. I do not see a reason to have request and response objects built just to dump them in byte slices, compress and send around. Is there a better way to do this? I am planning to use the lzma go package I found for compression and zmq push-pull (for passing requests) and pub-sub (for passing responses).

-Cem

[Brad Fitzpatrick]

What's the question?  How to serialize a Request/Response?  encoding/gob will do everything except the Body io.Reader field.  But you can nil that out and then gob it.

[Cem Ezberci]

Unfortunately encoding/gob doesn't work for Request because it has a URL field that has a User field, which is of type url.UserInfo and it does not have any exported fields. Please see below for the error when I try to encode a Request object. I set URL.User field to nil (following your advice for Body field) before encoding but that didn't seem to have helped.

2013/04/15 16:23:51 Gob'ing: Unable to prepare request for f0df68e386fb52fbc3a5c700555b37981d5e1a84: gob: type url.Userinfo has no exported fields

The workaround I found is to set URL.Scheme field in Request object to "http" (for now I'll just handle http), URL.Host to Host from Request object, RequestURI to "" (otherwise I wouldn't be able to make a Request using the Request object I'd reconstruct later using Request.Do) and then call the Write method of the Request with an io.Writer (lzma instance) to chain the write and dump the request as bytes and compress. This would need to be put in another struct with request id and forwarder id and gob encode and send over (so that the receiving end can return the response back to the original forwarder and requester but that's for routing with zmq etc. so it is not important at this point). On the receiving end, I'd gob decode the data received, decompress the compressed request in the struct, and then reconstruct the Request object using http.ReadRequest function. It looks like similar method exists for Response type (Response.Write) to do the reverse operation but I haven't found an equivalent of ReadRequest function for Response so that I could reconstruct the Response from wire format later on. Maybe I need to hijack the ResponseWriter and push out the data in wire format when serving back the response to the original requester.

[Tamás Gulácsi]

I think two viable solutions are:

a) send the original request, the wire format

b) save and send just what you need, and recreate a request on the other side from that minimal info.

If you don't really know all the possible request types beforehand, a) is better.