vert.x cluster in cloud foundry

[Pavel F]

I had no problem deploying standalone vert.x application to cloud foundry.

The problem starts when i want to cluster my vert.x instances. In my cloud environment all communication outside of deployed container is blocked.  There is exception for external (off-platform) dependencies which can be configured with external host and port if application running in the cloud needs to access resources outside of the cloud.

I managed to setup cluster discrovery using JGroups TUNNEL approach and my vert.x app can form a cluster and discover other nodes. But still my app is unable to communicate to outside world.  I suppose this is because EventBus communication goes over separate (random) TCP port which is blocked by firewall.  Initially i assumed that event bus communication would go over same network layer as used by the clustermanager, i.e. JGroups TUNNEL in my case, but is it not the case?

There seems to be a confirmation of above in that post.

Is there way to route event bus communication via external host / port for which I can open firewall?  If i get it correctly, should i be setting up cluster-host and cluster-port to an off-platform host (which i am using as JGroups GossipRouter in TUNNEL)?

Some help / opinions will be appreciated.

Thanks

[Jez P]

Akka Cluster ran into the same problem http://developer.lightbend.com/guides/running-akka-cluster-on-cloudfoundry/?_ga=2.209939027.73095495.1494354674-2050881667.1487536158

This might give you some hints as to how to approach vert.x clustering on CF

[Pavel F]

I looked into source code and clustering doesn't work in cloud foundry because Vert.x nodes communicate via separate TCP socket.  Without container networking in cloud foundry all TCP communication outside containers is blocked.  There is not much one can do...

[Thomas SEGISMONT]

Hi,

Which cluster manager are you using? vertx-jgroups or vertx-infinispan? I'm asking because in both cases you need to configure JGroups for node discovery but vertx-jgroups is deprecated.

I'm not very knowledgeable with CloudFoundry but I'll have a look since it's important to make sure Vert.x apps can run there.

Regards,

Thomas

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/29f243b0-29ec-4e3e-86f5-e7a922a286e1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Pavel F]

I am using vertx-jgroups, but it doesn't really matter which cluster manager is used.  It can be off-platform and, as far i can see, Vert.x is using it mainly to store cluster map of discovered nodes.  The main obstacle here is that once discovered, nodes need to be able to communicate directly with each other.  This requires Cloud Foundry's container-networking.

JGroups is more flexible in this respect, as it allows to tunnel cluster communication via a configured proxy(ies) host/port, so nodes in JGroups cluster don't have to communicate directly. This may not be as performant as direct communication though.

Without container-networking, everything should be proxied via front-door load balancer / TCP router... which starts looking like service discovery of Vert.x nodes.

I am thinking that to make it work, a mapped route to container is required and cluster map should be storing route's (host/port) for nodes to reach each other.  Providing route's parameters in -cluster-host & -cluster-port doesn't work here as these are not local to container where vert.x is running (due to load balancing, NAT'ing and routing within Cloud Foundry) so Vert.x can't bind to them.  It seems not difficult as long as mapped route identifies nodes uniquely (which i am not sure about as CF does some load balancing for the route to support scaling)... thoughts?

[Hoobajoob]

I have access to a cloud foundry environment that supports container networking, but I'm still at a loss as to how to configure Hazelcast's TCP discovery in cluster.xml to work.

Has anyone successfully setup a working vert.x Hazelcast cluster in Cloud Foundry? If so, how do you setup working 'member' entries (http://docs.hazelcast.org/docs/3.0/manual/html/ch12s02.html) in cluster.xml?