How to make haproxy always set X-Forwarded-For header?

[bluebr...@gmail.com]

I deploy a java web application in a cf v2 environment. The application needs get the real request ip of the client from the any valid http request to the request page.

The configuration of the haproxy in this cf is as following, and the groute is also set the option for X-Forwarded-For:

global

log 127.0.0.1 syslog info

daemon

maxconn 64000

spread-checks 4

defaults

log global

timeout connect 30000ms

timeout client  300000ms

timeout server 300000ms

option httpclose

option forwardfor

frontend http_in

mode http

bind :80

option httplog

option httpclose

option forwardfor

reqadd X-Forwarded-Proto:\ http

default_backend http-nginxs

backend http-nginxs

mode http

balance roundrobin

server nginx1 127.0.0.1:8160 check inter 1000

backend tcp-nginxs

mode tcp

balance roundrobin

server nginx1 127.0.0.1:8160 check inter 1000

Theoretically, the servlet can get acquire the client ip from the X-Forwarded-For header. But the printing log of the servlet shows it can't always acquire the client ip from the X-Forwarded-For header.

Most of the time,  there is only ip of haproxy in X-Forwarded-For header, and sometimes ips of client and haproxy both exist.

After capturing the package via tcpdump, i guess that the problem just exists in the haproxy. haproxy only adds the X-Forwarded-For header for a request matching one of the following conditions:

1.The url of the request hasn't been dispatch to any gorouter, or has been into the gorouter selected by current balance algorithm.

     

2.The session of the request is new. That is to say current cookies of this request is newly generated and has not been processed by the HA.

These conditions are figured out by assumption based on tcp package capturing. The HA may have some cache mechanism to achieve such effects.

Does anyone know how to configure the HA to make it add X-Forwarded-For header for every valid http request without such cache mechanism?

Or how can the java application running in CF can get real client ip with or even without X-Forwarded-For header for every  http request?

With great appreciation to anyone, who can give me some advice or solution.

[Daniel Mikusa]

On Mon, Nov 3, 2014 at 12:45 AM, <bluebr...@gmail.com> wrote:

I deploy a java web application in a cf v2 environment. The application needs get the real request ip of the client from the any valid http request to the request page.

With a Java app, it should be as simple as using the Servlet api to grab the remote ip address.  This is because the Java build pack configures Tomcat with a RemoteIpValve, which process the X-Forwarded-For header for you.

  http://tomcat.apache.org/tomcat-7.0-doc/config/valve.html#Remote_IP_Valve

  https://github.com/cloudfoundry/java-buildpack/blob/master/resources/tomcat/conf/server.xml#L25

Is this not the behavior that you're seeing?  If not, what's your setup?  What version of CF are you deploying to?  What version of the Java build pack are you using?

Dan

 

--
You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/cc28ef69-a836-4eb7-b657-e99b86e60e77%40cloudfoundry.org.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[aa...@hubernet.net]

You probably need to add the http-server-close option, see https://groups.google.com/a/cloudfoundry.org/forum/?fromgroups#!searchin/vcap-dev/"http-server-close"/vcap-dev/HHG9IxJDBNE/lUmGU9jZCbwJ.

Aaron Huber

Intel Corporation

[史建伟]

Tks a lot bros. I have solve the problem by adding httpclose option in the frontend section of the config file of the haproxy in the cf.  The option http-server-close or forceclose is also alternative. Considering the configuration should make  little effects to the real http request of the client, i choose httpclose.

在 2014年11月4日星期二UTC+8上午8时00分23秒，aa...@hubernet.net写道：

[Jianwei Shi]

My java application is fine. The problem just exists in the haproxy, and i have just fixed it.

I quite appreciate your valuable comments about java application. Thanks a lot Dan!

在 2014年11月3日星期一UTC+8下午11时33分26秒，Daniel Mikusa写道：

[Wenhao Chen]

Hi Dan

I found a similar issue, when I tried to push an app to get real request ip,

what I found is that I can get the real client IP from "X-Forwarded-For" value the first time I send GET request , but the other requests were not working 

After checking the haproxy config, I found the default connection mode is tunnel ("option http-tunnel") for version 1.5-dev19 ,which is my haproxy version

And I have  found some more from http://cbonte.github.io/haproxy-dconv/configuration-1.5.html

  - KAL : keep alive ("option http-keep-alive") which is the default mode : all
    requests and responses are processed, and connections remain open but idle
    between responses and new requests.

  - TUN: tunnel ("option http-tunnel") : this was the default mode for versions
    1.0 to 1.5-dev21 : only the first request and response are processed, and
    everything else is forwarded with no analysis at all. This mode should not
    be used as it creates lots of trouble with logging and HTTP processing.

 - PCL: passive close ("option httpclose") : exactly the same as tunnel mode,
    but with "Connection: close" appended in both directions to try to make
    both ends close after the first request/response exchange.
As stated above, the cause of what happened on missing real client IP is the default tunnel mode.

And I doubt using "option httpclose" is not a good solution, since "keep alive" for connection is not working any more.

Eventually I used v1.5.8 instead of the outdated haproxy version, everthing is ok now.

I don't know if  haproxy in official  cf-release should be updated ,

Thank you 

Wenhao

在 2014年11月3日星期一UTC+8下午11时33分26秒，Daniel Mikusa写道：

[Dieu Cao]

Hi Wenhao,

I'll add a story to the Runtime backlog to investigate upgrading v1.5.8.

https://www.pivotaltracker.com/story/show/82271326

Thanks for bringing this up.

-Dieu

Cloud Foundry Runtime PM

[jbayer]

FYI, an upcoming cf-release, likely v196 should contain an updated HAProxy: https://www.pivotaltracker.com/story/show/82271326

[aa...@hubernet.net]

We're planning to move to v197 soon and I wanted to understand the implications of the new HAProxy version.  I've always been slightly unhappy that we were using "http-server-close" on the HTTPS backend but not on HTTP, which resulted in the issue reported above that HTTP connections only got the forwarded-for and proto headers on the first hit but not subsequent ones due to the keep-alive to the back-end servers.

Now that we've got the new version of HAProxy it sounds like this is fixed and the correct headers are inserted in every request even if keep-alive is configured to the back-end servers.  I'd like to have this work consistently across both HTTPS and HTTP back-ends.

Current state:

• Keep-alive supported from the client -> HAProxy connection
• Keep-alive supported from HAProxy -> Routers for HTTP
• Keep-alive disabled from HAProxy -> Routers for HTTPS (due to http-server-close, previously required to get headers to work)

Options:

• Remove http-server-close from the HTTPS back-end (as long as the headers still work)
• Add http-server-close to the HTTP back-end for consistency

Basically, if the only reason we added http-server-close was to fix the headers and the headers are now working without it, do we still want it?

Aaron Huber

Intel Corporation

[James Myers]

Hi Aaron,

We agree. Hopefully we will be able to remove the `http-server-close` property from HTTPS and still have `X-Forward-For` work correctly. We are going to have a pair take a look at this. You can follow the progress of the story here: https://www.pivotaltracker.com/story/show/87802174.

Thanks,

Jim and Serguei, CF Runtime Team

[aa...@hubernet.net]

Just FYI, I've confirmed that if I remove the http-server-close config in HAProxy we do not see the redirect loops that we used to get with the old HAProxy version.  I think we definitely can and should remove it from the configuration.  I'll be working on another pull request (discussed in another thread) for HAProxy to add some optional logging config so if this isn't completed before then, I will add the removal to my pull request for that.

Aaron Huber

Intel Corporation

[aa...@hubernet.net]

I've submitted a pull request that removes the http-server-close option along with some other changes.

https://github.com/cloudfoundry/cf-release/pull/624

Aaron Huber

Intel Corporation