Failed to start event bus: java.net.BindException: Cannot assign requested address

[Arik Bron]

This problem was reported in the past and none responded to it.

I would appreciate any suggestions on how to overcome the "Failed to start event bus java.net.BindException: Cannot assign requested address" problem.

---------------------------------------------------------------------------------

I am deploying a Vert.x cluster of 2 VMs into AWS EC2.

Hazelcast's AWS Auto-Detection works - both VMs find each other given a proper default-cluster.xml:

Members [2] {

Member [172.31.1.164]:50701

Member [172.31.6.14]:50701 this

}

 

[172.31.6.14]:50701 [prod-whildey] [3.6.2] Address[172.31.6.14]:50701 is STARTED 

The problem is that after finding the cluster-host node, the second node fails with the following error:

Failed to start event bus 

java.net.BindException: Cannot assign requested address

at sun.nio.ch.Net.bind0(Native Method)

at sun.nio.ch.Net.bind(Net.java:433)

at sun.nio.ch.Net.bind(Net.java:425)

at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)

at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)

at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)

at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485)

at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089)

at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:440)

at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:425)

at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903)

at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198)

at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348)

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)

at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)

at java.lang.Thread.run(Thread.java:745)

All the needed ports are opened via AWS Security Group:

TCP - 35678 - specified via -cluster-port Vert.x command line setting and supposedly used by clustered event bus

TCP - 50701 - 50801 - specified via default-cluster.xml and is used by Hazelcast.

Both VMs connect to each other on port 50701.

[Tim Fox]

Are you trying to start two nodes on the same machine with the same value of cluster-port?

I.e. by using

-cluster-port 35678

at the command line?

Sockets are defined uniquely by host:port so that's not going to work (it's not a Vert.x limitation, you'd have the same with any application).

You should either give each node a different port number of don't use cluster-port and let the server assign a random port.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/1a07c0fd-07bc-4a08-a51b-c67d5c585697%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Arik Bron]

Hi Tim,

Thanks a lot for getting back to me!

I am using 2 VMs, each having its own IP.

I used the same -cluster-host and -cluster-port command line param values on both VMs. Following your post above I tried to use different ports but it didn't help.

I have to use -cluster-port in order to open the port on the AWS firewall (via Security group setting).

I'm still lost here - getting the same error with different ports...

[Arik Bron]

It looks like Vert.x is NOT using the -cluster-port setting!

In the cluster-host log I see the following line:

Accepting socket connection from /172.31.6.14:41977

Port 41977 is clearly random - it is not specified anywhere and while relaunching vertx I see a different port instead of it.

My Vert.x is specifically configured to run with -cluster-port=35678

I try to overcome the port randomness problem by allowing "All TCP" incoming traffic on AWS firewall - this doesn't help and the second vertx instance (non-cluster-host) fails with the same problem: Failed to start event bus 

java.net.BindException: Cannot assign requested address

[Tim Fox]

On 17/04/16 17:12, Arik Bron wrote:

It looks like Vert.x is NOT using the -cluster-port setting!

In the cluster-host log I see the following line:

Accepting socket connection from /172.31.6.14:41977

What you're seeing above is the _client_ side of the connection. The cluster-port refers to the _server_ side, not the client side. It's normal for the client side of a TCP connection to be assigned an random port.

Port 41977 is clearly random - it is not specified anywhere and while relaunching vertx I see a different port instead of it.

Try running netstat -at to see all TCP connections on the box along with their local and remote host:port

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/4ef99489-988a-4675-b0af-ce3cd93e95f1%40googlegroups.com.

[Tim Fox]

On 17/04/16 16:29, Arik Bron wrote:

Hi Tim,

Thanks a lot for getting back to me!

I am using 2 VMs, each having its own IP.

I used the same -cluster-host and -cluster-port command line param values on both VMs.

You shouldn't be using the same cluster-host and cluster-port on both VMs. You mentioned above that each VM has it's own IP, so use the IP address of VM1 as cluster-host in VM1 and the IP address of VM2 as cluster-host in VM2.

You can never have two servers on the same network having the same host and port. That's just how sockets work, not a Vert.x limitation.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/ba4d87fb-ef20-4430-8381-2870e96fa024%40googlegroups.com.

[Arik Bron]

OK.

So if I use 2 different cluster-host IPs, how can I force the VM2 to join the same cluster on which the 1st VM runs?

The whole idea of having a cluster is sharing the bus between the 2 VMs..

I hope I'm not missing anything...

[Tim Fox]

Assuming you're using the Hazelcast cluster manager...

cluster-host/cluster-port define the host/port that each Vert.x event bus node _listen_ on to accept connections from other nodes.

The values of cluster-host and cluster-port are distributed to the other nodes using the cluster manager. In the case of Hazelcast the values are placed in a distributed map which is available from all the nodes. That's how they know about the other nodes so they can make connections to them.

The default config of the Hazelcast cluster manager will use multicast. You've mentioned you are running on AWS. Multicast is not available on AWS so I assumed you've configured Hazelcast to use TCP or something else in order to form the cluster. If you've configured it to use TCP then you can hard code the host/port of at least one other node in the HZ config so it can look it up to form the cluster. I believe HZ also provides an AWS discovery method to avoid hardcoding IPs and ports.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/da281c43-c13b-449f-9289-53bab2104fc3%40googlegroups.com.

[Arik Bron]

Thanks a lot Tim, I have a cluster running.

You were right - cluster-host values should be different on both VMs. Therefore I am running vertx without either host or port.

I am using Hazelcast with AWS Auto-detect - not TCP or multicast. The detection of nodes works well.

Thanks a lot for all the help!