Can't get ReverseProxy to work

[grig...@gmail.com]

I don't know why this reverse proxy is not working. I've seen several examples and I can't find anything wrong with it. package main import ( "log" "net/url" "net/http" "net/http/httputil" ) func report(r *http.Request){ log.Print("URL: " + r.URL.Path) log.Print("Scheme: " + r.URL.Scheme) log.Print("Host: " + r.URL.Host) //r.URL.Scheme = "http" //r.URL.Host = "stackoverflow.com" //r.Header.Set("Host", "stackoverflow.com") //log.Print("Header Host: " + r.Header.Get("Host")) } func main() { proxy := httputil.NewSingleHostReverseProxy( &url.URL, Scheme:"http",Host:"myrealserver.com"}) proxy.Director = report // http.Handle("/", proxy) error := http.ListenAndServe("mylocalhost.com:8080", proxy) if(error != nil) { log.Fatal(error) } } It logs: 2014/04/18 21:32:50 URL: /arg/es 2014/04/18 21:32:50 Scheme: 2014/04/18 21:32:50 Host: 2014/04/18 21:32:50 http: proxy error: unsupported protocol scheme "" 2014/04/18 21:32:51 URL: /favicon.ico 2014/04/18 21:32:51 Scheme: 2014/04/18 21:32:51 Host: 2014/04/18 21:32:51 http: proxy error: unsupported protocol scheme "" If I uncomment the line that redefines the Schema the error message becomes: 2014/04/18 21:38:05 http: proxy error: http: no Host in request URL If I uncomment the line that redefines the host also, then the target server becomes stackoverflow.com (I mean, it never uses "myrealserver.com"). If I ask for mylocalhost.com:8080/somepath (or even /) then I get a 404 from Stackoverflow, no matter if stackoverflow.com/somepath exists or not. It says: Couldn't find mylocalhost.com:8080 The Q&A site mylocalhost.com:8080 doesn't seem to exist... yet It does not translate the Host header automatically. If then I uncomment the line that sets (and the other one that prints) the Header "Host". Then I can read "stackoverflow.com" in the log, but I still get the same 404 page reporting that I am trying to access "mylocalhost.com". How is it that I am supposed to make the program work as a proxy?

[Alex Zorin]

Just need to set http.Request.Host in the Director to get this working: http://play.golang.org/p/I17ZSM6LQb

If you read the source for SingleHostReverseProxy (http://golang.org/src/pkg/net/http/httputil/reverseproxy.go#L61), it sets its own Director which you are overriding.
So you need to reimplement what it already does plus the extra Host change.

[OpenNota]

2014/04/18 21:32:51 URL: /favicon.ico 2014/04/18 21:32:51 Scheme: 2014/04/18 21:32:51 Host: 2014/04/18 21:32:51 http: proxy error: unsupported protocol scheme ""

Are you trying to enter proxy address (mylocalhost.com:8080) into the browser address bar? You're doing it wrong.

[grig...@gmail.com]

I didn't really want a double host change.  I added a different host just to see wich one worked.

Ok, so the Director is not what I thought it was: a bridge function that allows you to tamper with the data aside from the main proxy functionality implemented internally.  The Director function HAS the functionality and as soon as you override it, you lost it and you have to rewrite it...  

As a design decision I can't argue against it, but It would have been easier to figure out if the documentation said something about that.

[grig...@gmail.com]

OpenNota, you mean I should set it as a proxy in my browser?

But it's not a traditional proxy.  It's a SingleHostReverseProxy, so as I understand it, it just sits in front of a web server and serves everything as an alias of the real server.  The Director can, then, cache the results or tamper with them in any way.

Otherwise, could you explain how is it supposed to be used?
And if it wheren't a reverse proxy but a traditional proxy, how should it be used?

It's my first attempt at a proxy.  I've only been working on this for several hours today and still have a lot to grok.

Thanks!

[OpenNota]

On Saturday, April 19, 2014 12:32:32 PM UTC+7, grig...@gmail.com wrote:

I didn't really want a double host change.  I added a different host just to see wich one worked.

The Director function HAS the functionality and as soon as you override it, you lost it and you have to rewrite it...  

Not necessarily.

package main

import (

"log"

"net/http"

"net/http/httputil"

"net/url"

)

func main() {

target, _ := url.Parse("http://google.com/")

proxy := httputil.NewSingleHostReverseProxy(target)

realDirector := proxy.Director

proxy.Director = func(req *http.Request) {

log.Println(req.URL)

realDirector(req)

}

err := http.ListenAndServe(":8080", proxy)

if err != nil {

log.Fatal(err)

}

}

[OpenNota]

On Saturday, April 19, 2014 12:43:17 PM UTC+7, grig...@gmail.com wrote:

But it's not a traditional proxy.  It's a SingleHostReverseProxy, so as I understand it, it just sits in front of a web server and serves everything as an alias of the real server.

You're right. I was wrong here.

[Alex Zorin]

> The Director can, then, cache the results or tamper with them in any way.

I don't think this is correct.

httputil.ReverseProxy simply copies a request that it received from a client and then passes it on to a target host.
The Director allows you to modify the request as it travels from the client to the target, but it has no way to interact with the response.
You should read the source to ReverseProxy.ServeHTTP, it is rather short and readable imo: http://golang.org/src/pkg/net/http/httputil/reverseproxy.go#L97

If you want to write a caching proxy, I don't think that httputil.ReverseProxy, on its own, can do it. Maybe you can write an outer http.HandlerFunc that wraps ReverseProxy?

    http.Handle("/", NewCachingHandler(httputil.NewSingleHostReverseProxy(...))
...
    func (h CachingHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
        if h.canServeFromCache(r) { h.serveFromCache(w, r) } else { h.innerHandler.ServeHTTP(w, r) }
    }

Forgive the inline pseudocode ...

[Sebastian Grignoli]

> If you want to write a caching proxy, I don't think that httputil.ReverseProxy, on its own, can do it.

Well, that's what I did.  Look at the answer to this same question in SO:

http://stackoverflow.com/a/23166390/290221

It's not yet changing anything or caching anything.
Also, I couldn't get the POST requests to work yet.
All kind of criticism is welcome.

[Sebastian Grignoli]

realDirector := proxy.Director

proxy.Director = func(req *http.Request) {

log.Println(req.URL)

realDirector(req)

}

 Cool.

Anyway, as Alex pointed out, I would have to step outside of the ReverseProxy package to modify or cache the response, so I'm glad I did that sooner.

But now I'm curious about the traditional proxy.  OpenNota, (or anyone reading, obviously) can you point me into the right direction on how to make my browser interface with it?

I did set it up as a Proxy for all navigation, but my server is not getting any request...  (not even just to let me look at them fail)
  
Looks like Chrome is still connecting to the Internet directly.

[OpenNota]

But now I'm curious about the traditional proxy.  OpenNota, (or anyone reading, obviously) can you point me into the right direction on how to make my browser interface with it?

I did set it up as a Proxy for all navigation, but my server is not getting any request...  (not even just to let me look at them fail)
  
Looks like Chrome is still connecting to the Internet directly.

 Virtual hosts? What if you change the Host HTTP header in the Director func, like

package main

import (

"log"

"net/http"

"net/http/httputil"

"net/url"

)

func main() {

target, _ := url.Parse("http://myrealserver.com/")

proxy := httputil.NewSingleHostReverseProxy(target)

realDirector := proxy.Director

proxy.Director = func(req *http.Request) {

req.Host = target.Host // myrealserver.com

realDirector(req)

}

err := http.ListenAndServe(":8080", proxy)

if err != nil {

log.Fatal(err)

}

}

Works for me in FF and luakit (I set reverse proxy address as an HTTP proxy in the browsers)

[Sebastian Grignoli]

Turns out that in Firefox, setting "localhost:8080" and running the server like this:
 ./proxy --target=""  
was enough to use it as a general purpose proxy, not just a single-host one.

Great!

(but Chrome keeps ignoring the Proxy settings)

[Sebastian Grignoli]

Oops, Chrome just needed a restart!