Proxy Protocol v1

[Siddharth Mathur]

Hello, 

I am investigating use of GoProxy based an AWS Elastic Load Balancer so as to achieve higher availability. The use of a forwarding intermediary unfortunately hides the true client IP/port information from my GoProxy powered server, which is important to have for X-Forwarded-For sanity. 

AWS supports turning on (HAProxy) Proxy Protocol v1 [Ref 1]  in the load balancer, which inserts a client info string such as "PROXY TCP4 198.51.100.22  203.0.113.7  35646  80\r\n" as the first few bytes in the request sent to the GoProxy instance. However, since Go's HTTP stack isn't aware of this, it bombs with the HTTP 400 Bad Request error and the GoProxy layer don't appear to kick in. 

Any ideas on how Proxy Protocol v1 could be added to goproxy/the application using it? 

IIUC, the http.hijack route can help one read the TCP socket for the first line, but it cuts out the Go HTTP stack for the rest of processing which isn't desirable. 

Suggestions? :)

Siddharth

Reference: [1] http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/TerminologyandKeyConcepts.html#proxy-protocol

[Elazar Leibovich]

Hi,

Thanks for considering goproxy.

I'm not sure I understand the goal. What do you expect the goproxy to do? Can you plot the architecture? Try https://www.draw.io and paste the URL.

--
You received this message because you are subscribed to the Google Groups "goproxy-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to goproxy-dev+unsubscribe@googlegroups.com.
To post to this group, send email to gopro...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/goproxy-dev/4cfe3316-7ea2-452b-bdb7-f40e9c38c31a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Siddharth Mathur]

Hi Elazar,

Thanks for your response. Here is the network topology:
http://ibin.co/1h4dXaRPWCdW

Communication endpoints A and B are the mobile phone and public
interface of the load balancer respectively.
The ELB then forwards at the TCP level, all incoming bytes in a
separate TCP connection established between endpoints C and D. Note
that C is the private IP of the ELB and D is the private IP of the
GoProxy instance.

As you can see above, GoProxy or any software for that matter at
endpoint D only sees the private IP/node C as the RemoteAddr().

To work around this common class of problem with load balancers +
proxies, the Proxy Protocol v1 is a HAproxy devised (?) standard which
AWS ELBs do implement (ref [1]). The ELB adds some bytes to the first
few lines of the HTTP connection, which a supporting HTTP server or
proxy ought to be able to detect and parse.
An example preamble is ""PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n"

Since Go's HTTP layer doesn't expect the above message and sends a
HTTP 400 Bad Request back, the GoProxy layer doesn't get to do any
work.

So my question is really if there are any straightforward ways to
"peek" into the TCP connection, parse this first line if present, and
hopefully also ensure that RemoteAddr() returns the real client's IP.

http.hijack() would be overkill for this need as it bypasses the
entire HTTP stack after a call to it.

Any pointers welcome.
Siddharth

[1] The Proxy Protocol v1 standard:
http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt
--
Blackbuck Computing | blackbuck.mobi | twitter.com/blkbck |
+91-888-483-4186 | +1-617-500-7576

>> email to goproxy-dev...@googlegroups.com.

[Elazar Leibovich]

This is a great question.

I thought about it a long time ago, when trying to support proxy'ing the ICY protocol, which requires HTTP phrase to be replaced with ICY.

I don't think there's a straightforward solution, but if there is, I think goproxy needs to remain as it is, but net/http package should be changed to support parsing flexible requests.

What I would do is, COPY net/http package, and try to change it so that it'll support the extra standard. It should still accept and serve `net/http.Handler` Then, have goproxy server use this handler.

I tried to do similar thing by copying `http.Transport`. https://github.com/elazarl/goproxy/tree/master/transport I'm not using it anymore, but I tried to have the transport give me more information.

Not the nicest solution, but I think it's the most practical one. That said, it's not an easy task. Go stdlib sometimes have internal dependencies which are not always possible to use outside.

Please update us on your progress, I might have more time to look at it tonight.

Thanks,

>> email to goproxy-dev+unsubscribe@googlegroups.com.

>> To post to this group, send email to gopro...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/goproxy-dev/4cfe3316-7ea2-452b-bdb7-f40e9c38c31a%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "goproxy-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to goproxy-dev+unsubscribe@googlegroups.com.

To post to this group, send email to gopro...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/goproxy-dev/CABMjiNtNMFdQFzMfOzUUp2Q%3DRzNVjCMUBRgJYaDfiUiiFQLrPA%40mail.gmail.com.

[Siddharth Mathur]

I agree 100% that net/http should have this feature, and all consumers
such as goproxy should get the benefits automatically, perhaps with a
boolean API flag to control Proxy Protocol v1 detection ON or OFF via
http.Server.

Having worked with the fast moving WebKit community in the past, I
will pass on the option to copy net/http into my own source tree and
massage it ;) . It gets very burdensome very fast.

I am looking at this go-proxyproto library
(https://github.com/armon/go-proxyproto/blob/master/protocol.go) which
allows me to wrap a net.Conn and help check for and parse the payload.
Now the questions are:
a) How does one grab a net.Conn handle for an incoming request,
without a full hijack() ?
b) How does one pass the (wrapped) net.Conn back into Go's HTTP stack.
go-proxyproto's wrapper makes RemoteAddr() return the right thing.

On (a), using the new http.Server.ConnState hook allows me to create a
proxyproto.Conn for inspection, but how does one do (b) above?


func changeHandler(conn net.Conn, state http.ConnState) {
if state == http.StateNew {
ppConn := proxyproto.NewConn(conn)
log.Println("ConnState is StateNew, with remote address ",
ppConn.RemoteAddr())
}
}

Siddharth

--
Blackbuck Computing | blackbuck.mobi | twitter.com/blkbck |
+91-888-483-4186 | +1-617-500-7576

>> >> email to goproxy-dev...@googlegroups.com.

>> >> To post to this group, send email to gopro...@googlegroups.com.
>> >> To view this discussion on the web visit
>> >>
>> >> https://groups.google.com/d/msgid/goproxy-dev/4cfe3316-7ea2-452b-bdb7-f40e9c38c31a%40googlegroups.com.
>> >> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "goproxy-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to goproxy-dev...@googlegroups.com.

[Elazar Leibovich]

I don't think you should hijack the connection. If you do that for every connection, than you might as well reimplement most of net/http yourself.

Another idea, is to implement my own net.Listener that wraps a net.Connect, parses the HTTP, and rewrites the request as a valid one for the net/http

>> >> email to goproxy-dev+unsubscribe@googlegroups.com.

>> >> To post to this group, send email to gopro...@googlegroups.com.
>> >> To view this discussion on the web visit
>> >>
>> >> https://groups.google.com/d/msgid/goproxy-dev/4cfe3316-7ea2-452b-bdb7-f40e9c38c31a%40googlegroups.com.
>> >> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "goproxy-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to goproxy-dev+unsubscribe@googlegroups.com.

[Siddharth Mathur]

On Thursday, November 13, 2014 2:06:28 PM UTC+5:30, Elazar Leibovich wrote:

Please update us on your progress, I might have more time to look at it tonight.

Hi Elazar,

Just to close the loop on this Proxy Protocol message topic, I have done a hack which works for me for now. Not very pretty though ;). 

I use the new http.Server.ConnState feature to intercept net.Conn in main(), and then use go-proxyproto to read the header. I create a "fix up" map of IP:port -> IP port mappings where the value is the corrected/as-seen-by-load balancer public IP of the client. Due to the read on the socket caused by go-proxyproto (and the co-operative standard which limits size of this header), the subsequent reads by Go's HTTP package proceed normally. 

I have added a new "User Data" field to the ProxyHttpServer struct so the above map is made visible to ProxyHttpServer.ServeHTTP() around the time a new ProxyCtx is created. As this is the entry point for goproxy, fixing RemoteAddr here and/or modifying headers here gets the job done. 

Thanks,

Siddharth

[Elazar Leibovich]

So if I understand that correctly, you patched the stream, and in fact made the equivalent of a reverse proxy to your proxy, which emits standard HTTP headers. Right?

Good idea, similar to what I've had in mind.

BTW, can't you do that with a dedicated reverse proxy? Can ngnix understand the proxy protocol, fix it, and simply reverse proxy the request to a normal golang net/http.Server?

--
You received this message because you are subscribed to the Google Groups "goproxy-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to goproxy-dev+unsubscribe@googlegroups.com.
To post to this group, send email to gopro...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/goproxy-dev/9e4fd47f-718f-434b-ba24-3291d746e392%40googlegroups.com.