Reverse Proxies: It's been a while since someone mentioned them :)

[OmarShariffDontLikeIt]

Hi Gophers!

Messing about with a small project: it's basically a reverse proxy
that will convert json responses from the origin server to lovely html
in the client request. It decodes the json response from the origin
and passes this struct to the mustache templating engine which outputs
html.

The server has a config file allowing the user to specify url patterns
to match and destination urls to proxy to:
{
"pattern":"/hello",
"origin":"http://example.com/some/destination",
"template":"./hello.template"
} // etc, etc

I'm having difficulty understanding our friend
httputil.NewSingleHostReverseProxy(). It may have something to do with
the way I have approached this though.

Ideally I have a http server. This assigns a handler for each entry in
the config file, matched by pattern. The handler creates a new proxy
server using httputil.NewSingleHostReverseProxy() with the origin
config parameter as the target argument. I then have a custom
http.RequestWriter that reads the json body in response, parses it,
passes it to mustache and writes the output:

...
for _, route := range config.Routes {
url, _ := url.Parse(route.origin)
MustacheHandler := NewMustacheHandler(route.TemplateCache, url)
http.Handle(route.Pattern, MustacheHandler)
}
err = http.ListenAndServe(address, nil)
if err != nil {
log.Fatal("ListenAndServe: ", err)
}
...

The mustache handler looks a little bit like this:

func NewMustacheHandler(template []byte, url *url.URL)
http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
mw := NewMustacheWriter(w, template)
rw := mustacheResponseWriter{Writer: mw, ResponseWriter: w}

proxy := httputil.NewSingleHostReverseProxy(url)
proxy.ServeHTTP(rw, r)
}
}

And the mustache writer looky likey thisy:

type MustacheWriter struct {
Template []byte
w io.Writer
}

func (w MustacheWriter) Write(b []byte) (int, error) {
var f interface{}
err := json.Unmarshal(b, &f)
if err != nil {
log.Fatal(err, string(b))
}

m := f.(map[string]interface{})
d := mustache.Render(string(w.Template), m)

return w.Write([]byte(d))
}

The problem is that I keep getting "File not found" passed into the
Write(b []byte) argument, even though if I curl the URL being passed
as the origin I get valid JSON back.

So, questions!

1) Am I using NewSingleHostReverseProxy() correctly?
2) Is there a problem with my approach to this problem?
3) Is there a better way?

Cheers!
Ben

[Kyle Lemons]

The actual URL is created by combining r with url.  Just do an http.Get on URL unless you actually want a reverse proxy for a directory or something.

[OmarShariffDontLikeIt]

Ah, but a want a FULL HTTP proxy, passing though POSTs, GET, DELETE
etc;

Are you suggesting that NewSingleHostReverseProxy is not suitable for
this task? I may have misunderstood but I thought that this is exactly
what NewSingleHostReverseProxy is for.

[Kyle Lemons]

Yes, that's what it does.

You said that you can CURL the route URL, so I assumed that the route URL was the one that was returning the JSON.  The route URL is being modified by ReverseProxy, so that CURL is not going to reproduce the behavior.  I believe you can wrap the default Director with your own and log the new path if you want to see what it's doing.

[OmarShariffDontLikeIt]

> The route URL is being modified by ReverseProxy

Ah, interesting! Why does it do that!?

> believe you can wrap the default Director with your own and log the new
> path if you want to see what it's doing.

Ooo that sounds just what I need to do to work out the problem. Could
you give me an example of what you mean?

[Kyle Lemons]

On Thu, Apr 26, 2012 at 1:02 PM, OmarShariffDontLikeIt <omarsharif...@gmail.com> wrote:

> The route URL is being modified by ReverseProxy

Ah, interesting! Why does it do that!?

From the documentation: "NewSingleHostReverseProxy returns a new ReverseProxy that rewrites URLs to the scheme, host, and base path provided in target. If the target's path is "/base" and the incoming request was for "/dir", the target request will be for /base/dir."

So, thus, if your route is http://example.org/generate/gophers.js and the request is for http://localhost:4242/gophers.js the outgoing URL is http://example.org/genrate/gophers.js/gophers.js which is probably not what you intended :).

 

> believe you can wrap the default Director with your own and log the new
> path if you want to see what it's doing.

Ooo that sounds just what I need to do to work out the problem. Could
you give me an example of what you mean?

I think it looks something like

shrp := httputil.NewSingleHostReverseProxy(url)

director := shrp.Director

shrp.Director = func(req *http.Request) {

   director(req)

   log.Printf("Redirected URL: %s", req.URL)

}

[OmarShariffDontLikeIt]

Aha! That makes perfect sense! I will try that out tmrw.

Thanks for your help.

Cheers,
Ben

[da...@fit.ly]

You could have a look at falcore (github.com/ngmoco/falcore).  One of the core things we built it for was reverse proxying into other applications.

[OmarShariffDontLikeIt]

> You could have a look at falcore (github.com/ngmoco/falcore).  One of the
> core things we built it for was reverse proxying into other applications.

Having a quick read, I must say I really do like the pipelining of
filters. It reminds me a lot of Apaches filters (without the fiddly
mess!).

After a brief look, I can't find anything relating to actually
proxying a request to a back end. Is this built in, a filter, or would
I have to write this myself?

Thanks for the heads up on Falcore, I takes an approach I am familiar
with (filters/pipeline) so I may end up switching my project over to
this approach as it would result in a much cleaner architecture,
especially as I have several features I would like to add in future
that would be trivial to add as additional filters.

Cheers!
Ben

[OmarShariffDontLikeIt]

> After a brief look, I can't find anything relating to actually
> proxying a request to a back end. Is this built in, a filter, or would
> I have to write this myself?

Just found it; the upstream dir contains a upstream/proxy filter.

Cheers,
Ben