Enforcing X-Forwarded-for http header

[yacov...@gmail.com]

Hi all.

I'm writing a distributed application that needs to know the IP address of the DEAs of its member nodes.
For now, I'm trying to avoid querying the cloud controller for that information, and in the integration environment what I'm doing is simply sending HTTP requests to the application's URL, and the nodes that handle the request extract the X-Forwarded-For http header from the request.
I also have a cloud foundry installation running on a single server installed with nise-bosh (https://github.com/yudai/cf_nise_installer) and I want to continue the development on this server.
The problem is that from some reason, the X-Forwarded-For header isn't set by the go-router on my CF installation.

I wrote a simple servlet that prints all the http headers:
host=srns.10.0.2.15.xip.io
user-agent=Java/1.7.0_45
connection=close
content-length=29
accept=text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
content-type=application/x-www-form-urlencoded
accept-encoding=gzip

and as you can see, the X-Forwarded-For header isn't set and therefor I have no other way of knowing the IP of the DEA the request came from.

I tried searching the gorouter's code and modified some lines I thought are relevant for setting the header, recompiled the gorouter but it still doesn't work.


Does anyone here know where exactly is the X-Forwarded-For http header set by gorouter or maybe knows an alternative method of knowing the ip of the DEA the request came from, aside from querying the Cloud Controller?



Thanks in advance,

Yacov Manevich.

[Guillaume Berche]

My understanding is that this is the load balancer in front of the gorouter which assigns the x-forwarded-for header, see https://github.com/cloudfoundry-community/sslproxy-boshrelease/blob/master/jobs/sslproxy/templates/location.conf.erb#L15

Do you have the sslproxy jobs (or alternatively haproxy) into your bosh-nise-based CF deployment ?

Guillaume.

[Aristoteles Neto]

The grouter job also sets the X-Forwarded-For header:

https://github.com/cloudfoundry/gorouter/blob/master/proxy/request_handler.go#L172-L183

(unless I’m reading that incorrectly)

However, from what I understood the original poster, I think he needs the backend IP addresses (he mentions DEA IPs), instead of the X-Forwarded-For (client) address. In which case I can’t imagine it being possible for an instance to tell the IPs of other instances without querying the API. But I’m new here, so feel free to completely ignore me :-)

-- Neto

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[Guillaume Berche]

Sorry my mistake, not sure why the router indeed does not affect initi XFF if missing.

As a workaround, would the caller apps looking up their local IP, and explictly passsing it into their request to the route URL work for your case ? I think it worked in my java app calling InetAddress.getAddress()

BTW, why do you need to get the container IPs ? Are you querying directly the CONSOLE or DEBUG port, or expecting non HTTP on the HTTP port (that gorouter would nor route) ?

Guillaume.

[James Bayer]

use the 'cf stats APPNAME -t' command or the new 'CF_TRACE=true gcf app APPNAME' command to see the IP:ports of the DEAs that host the apps in the json responses. X-forwarded-for is for the client IPs, not for the DEA IPs.

--

Thank you,

James Bayer

[James Bayer]

here’s an example:

GET /v2/apps/f2c13640-a2f3-4d22-b7e1-019eed09754b/stats

{
    "0": {
        "state": "RUNNING",
        "stats": {
            "name": "hello-go",
            "uris": ["hello-go-james.cfapps.io"],
            "host": "10.10.81.19",
            "port": 62847,
            "uptime": 35,
            "mem_quota": 268435456,
            "disk_quota": 1073741824,
            "fds_quota": 16384,
            "usage": {
                "time": "2014-01-17
07:40:04
+0000",
                "cpu": 0.0,
                "mem": 1720320,
                "disk": 6451200
            }
        }
    },
    "2": {
        "state": "RUNNING",
        "stats": {
            "name": "hello-go",
            "uris": ["hello-go-james.cfapps.io"],
            "host": "10.10.81.29",
            "port": 62287,
            "uptime": 10,
            "mem_quota": 268435456,
            "disk_quota": 1073741824,
            "fds_quota": 16384,
            "usage": {
                "time": "2014-01-17
07:40:04
+0000",
                "cpu": 0,
                "mem": 1720320,
                "disk": 6451200
            }
        }
    },
    "1": {
        "state": "RUNNING",
        "stats": {
            "name": "hello-go",
            "uris": ["hello-go-james.cfapps.io"],
            "host": "10.10.81.25",
            "port": 62597,
            "uptime": 10,
            "mem_quota": 268435456,
            "disk_quota": 1073741824,
            "fds_quota": 16384,
            "usage": {
                "time": "2014-01-17
07:40:04
+0000",
                "cpu": 0,
                "mem": 1724416,
                "disk": 6451200
            }
        }
    },
    "3": {
        "state": "RUNNING",
        "stats": {
            "name": "hello-go",
            "uris": ["hello-go-james.cfapps.io"],
            "host": "10.10.17.11",
            "port": 64259,
            "uptime": 9,
            "mem_quota": 268435456,
            "disk_quota": 1073741824,
            "fds_quota": 16384,
            "usage": {
                "time": "2014-01-17
07:40:04 +0000",
                "cpu": 0,
                "mem": 1708032,
                "disk": 6451200
            }
        }
    }
}

[yacov...@gmail.com]

Thanks, but I know how to communicate with the CC and get the data you showed me, but I need this to be done inside a running application and without the use of credentials.
That's why I thought of the X-Forwarded-For direction
The application's nodes communicate with each other, and in this way the X-Forwarded-For field contains the last ip address that went to the gorouter - meaning, the DEA ip (the client here is a peer DEA).

[yacov...@gmail.com]

the workaround won't work because the IP obtained from the indetaddress.getAddress() method is the virtual ip of the warden container and not the DEAs address. it has no meaning outside of the DEA and is not routable outside the container.

I need the container's IP so I could make the application's nodes communicate with each other in order for every node to know if its peers are ok or not, etc...

[Guillaume Berche]

Yacov, I understand you need nodes (app instances) to be able to target other nodes through HTTP.

What about simply using the normal HTTP route that load balances across all instances, and craft the __VCAP_ID__ cookie to hint to the router to send the request to the specific  instance (see James nice blog at http://www.iamjambay.com/2013/10/a-quick-tour-of-cloud-foundry-router.html )? The __VCAP_ID__ cookie seems to expect a value of the app instance GUID.

Guillaume.

[yaco...@gmail.com]

Thanks for the reply, but that's not what I need, I need to find the IP of the DEA the request came from, and only the IP of the DEA. An ability to forward HTTP request to specific DEAs doesn't help me at all.
What I'm actually trying to do is to create a small grid of nodes that have common agreement of group membership, and are able to provide some sort of basic name --> [list of ip:host] resolution to other client applications. For this, I need each node to be able to extract the ip address of the DEA from the HTTP request send by every peer node.

 

בתאריך יום חמישי, 23 בינואר 2014 22:57:31 UTC+2, מאת Guillaume Berche:

[Cornelia Davis]

One thing I want to clarify - application instances should not be communicating with one another peer to peer. In fact, I'd be surprised if you could pull this off at all.  This is by design.  DEAs are multi-tenant and the apps are isolated in warden containers.  As you've pointed out, Yacov, the ip address is of the warden container and is not reachable from other app instances - again by design.

There are two ways that app instances can communicate with one another. 1) Via services - i.e. a database, messaging server, etc. 2) via HTTP in which case the requests will go out and back in through the router.  Though it will probably prove difficult or impossible to address a specific app instance that way.

Cornelia

[yaco...@gmail.com]

They can communicate with each other directly, if and only if they have each other's DEA ip address. In the meantime what I do to get the ip addresses is query the Cloud Controller, but that requires for the applications to have credentials in them, and I don't like the idea (security-wise).
While the ip address of the warden container is not reachable, it is routable somehow, else the application won't be able to communicate.
According to my limited CF knowledge, when the request reaches to the gorouter it has port 80(http) as its destination port. The gorouter routes the request according to the URL to one of the DEAs, and changes the destination port to be the VCAP_APP_PORT of the warden container. In the DEA, the TCP packets are forwarded to the correct warden container by the destination port via the NAT tables of the DEA's firewall.

I "pulled this off" in 1 environment of CF that does puts the X-Forwarded-For by the gorouter, but in my single node installation it doesn't happen so I'm asking if there is any way to make this happen or is there any alternative way of discovering the ip addresses.

בתאריך יום שישי, 24 בינואר 2014 03:20:11 UTC+2, מאת Cornelia Davis:

[James Bayer]

cornelia is correct. even if you coerce the system into going from one warden container in a DEA to another DEA's NAT for another app, you are fighting against the design of the system and it is not intentional to enable this use case. we expect all app traffic to go through the router because the list of available endpoints (DEA host:port) change as app instances come and go, the dynamic routing table is updated almost instantly. using a hard-corded ip:port that isn't from the routing table in the router can easily be wrong.

[yaco...@gmail.com]

as app instances come and go

Can you elaborate? 

1) Let's say for example that an application has only 1 instance running on DEA-1, and you have DEA-1 up to DEA-10.
2) Same example but now with 5 instances on 5 different DEAs.
Can the applications in these 2 examples be migrated to different DEAs? if yes, than under what circumstances, and doesn't it harm the availability of the application in case number 1?

Thanks in advance.

בתאריך יום שבת, 25 בינואר 2014 06:40:36 UTC+2, מאת James Bayer:

[James Bayer]

yes, if availability is important to you, you should have more than 1 app instance because you could have a fault in any single DEA VM at any time for whatever reason.

CF will automatically balance app instances across different DEAs that meet the memory/disk requirements, currently that logic is in the cloud controller but is moving to a different part of the system in the diego architecture. additionally, the cloud controller will balance across zones, where a zone is defined in each DEA config file and may be aligned conceptually with an AWS availability zone or similar for your infrastructure.

cf will not stop your app instance on it's own unless DEAs are being upgraded, but that does happen according to the CF operator maintenance scheduled and your app should account for that.

[yaco...@gmail.com]

Got it, thanks.


בתאריך יום שבת, 25 בינואר 2014 18:27:36 UTC+2, מאת James Bayer: