Cluster elastic search failing with java.net.NoRouteToHostException: No route to host

[nickro...@gmail.com]

I'm having an issue with the cluster setup failing on 2.5.3 with a "java.net.NoRouteToHostException: No route to host" error in the log, see log snippet and node configurations below.

Does anyone have any suggestions for this, maybe it's a network issue? I also tried using multicast and that did get around the error but the elastic search still failed to initialize properly.


-------------------------------------------------------------------------------
LOG ON NODE 2
-------------------------------------------------------------------------------
 
[12/03/14 02:34:05:275 EDT]  INFO servlets.InitServlet:      Ping Sent
[12/03/14 02:34:05:275 EDT]  INFO servlets.InitServlet:
[12/03/14 02:34:05:320 EDT]  INFO elasticsearch.node: [dotCMS_2] {0.19.9}[1182]: initializing ...
[12/03/14 02:34:05:324 EDT]  INFO elasticsearch.plugins: [dotCMS_2] loaded [], sites []
[12/03/14 02:34:05:643 EDT]  INFO priv.a: ClusterThread start delayed for 5000 millis.
[12/03/14 02:34:07:130 EDT]  INFO elasticsearch.node: [dotCMS_2] {0.19.9}[1182]: initialized
[12/03/14 02:34:07:131 EDT]  INFO elasticsearch.node: [dotCMS_2] {0.19.9}[1182]: starting ...
[12/03/14 02:34:07:276 EDT]  INFO elasticsearch.transport: [dotCMS_2] bound_address {inet[/130.101.146.28:9304]}, publish_address {inet[/130.101.146.28:9304]}
[12/03/14 02:34:07:379 EDT]  WARN transport.netty: [dotCMS_2] Exception caught on netty layer [[id: 0x45a2f4b2]]
java.net.NoRouteToHostException: No route to host
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
        at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:404)
        at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:366)
        at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:282)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
 
---------------------------------------------------------------------------------
NODE 1 CLUSTER CONFIG
---------------------------------------------------------------------------------
 
DIST_INDEXATION_ENABLED=true
#In milliseconds
DIST_INDEXATION_INIT_DELAY=5000
DIST_INDEXATION_SLEEP=500
DIST_INDEXATION_SERVER_ID=1
#A comma separated list of server ids (integers), you have to pick the right number of ids to avoid
#the indexation table grows enormously, E.G. If you have only 2 servers you must set the variable like
DIST_INDEXATION_SERVERS_IDS=1,2
 
#Allows browsing to return to http after loading an https page.
FORCE_HTTP_CONNECTION=false
 
#Disable clickstream tracking in the database.
ENABLE_CLICKSTREAM_TRACKING=false
 
ENABLE_SCRIPTING=true
 
LDAP_FRONTEND_AUTH_IMPLEMENTATION=com.dotcms.enterprise.LDAPProxy
 
FORCE_CAPTCHA=false
 
CACHE_CLUSTER_THROUGH_DB=false
#CACHE_PROTOCOL=udp
#CACHE_MULTICAST_PORT=455
#CACHE_MULTICAST_ADDRESS=228.10.10.10
CACHE_FORCE_IPV4=true
CACHE_BINDPORT=7800
CACHE_BINDADDRESS=130.101.146.27
CACHE_TCP_INITIAL_HOSTS=130.101.146.27[7800],130.101.146.28[7800]
 
es.network.host=130.101.146.27
es.transport.tcp.port=9301
es.network.port=9302
es.http.port=9303
 
es.discovery.zen.ping.multicast.enabled=false
es.discovery.zen.ping.timeout=5s
es.discovery.zen.ping.unicast.hosts=130.101.146.27:9301,130.101.146.28:9304
 
es.index.number_of_replicas=1
 
 
---------------------------------------------------------------------------------
NODE 2 CLUSTER CONFIG
---------------------------------------------------------------------------------
 
DIST_INDEXATION_ENABLED=true
#In milliseconds
DIST_INDEXATION_INIT_DELAY=5000
DIST_INDEXATION_SLEEP=500
DIST_INDEXATION_SERVER_ID=2
#A comma separated list of server ids (integers), you have to pick the right number of ids to avoid
#the indexation table grows enormously, E.G. If you have only 2 servers you must set the variable like
DIST_INDEXATION_SERVERS_IDS=1,2
 
#Allows browsing to return to http after loading an https page.
FORCE_HTTP_CONNECTION=false
 
#Disable clickstream tracking in the database.
ENABLE_CLICKSTREAM_TRACKING=false
 
ENABLE_SCRIPTING=true
 
LDAP_FRONTEND_AUTH_IMPLEMENTATION=com.dotcms.enterprise.LDAPProxy
 
FORCE_CAPTCHA=false
 
CACHE_CLUSTER_THROUGH_DB=false
#CACHE_PROTOCOL=udp
#CACHE_MULTICAST_PORT=455
#CACHE_MULTICAST_ADDRESS=228.10.10.10
CACHE_FORCE_IPV4=true
CACHE_BINDPORT=7800
CACHE_BINDADDRESS=130.101.146.28
CACHE_TCP_INITIAL_HOSTS=130.101.146.27[7800],130.101.146.28[7800]
 
es.network.host=130.101.146.28
es.transport.tcp.port=9304
es.network.port=9302
es.http.port=9303
 
es.discovery.zen.ping.multicast.enabled=false
es.discovery.zen.ping.timeout=5s
es.discovery.zen.ping.unicast.hosts=130.101.146.27:9301,130.101.146.28:9304
 
es.index.number_of_replicas=1

[Jason Tesser]

make sure you can connect from each server on that port. use telnet or something. 

On Wed, Mar 12, 2014 at 3:30 AM, <nickro...@gmail.com> wrote:

es.transport.tcp.port

[nickro...@gmail.com]

Yeah it turns out the tcp ports were firewalled, so that did the trick. Thanks.