vertx ignite cluster failing

[Anil]

Hi,

I am using vertx-ignite-3.4.0.jar and i see following exception while deploying the jars. i dont see any issues with network.

2016-12-01 03:03:33 WARN  marshaller-cache-#83%vertx.ignite.node.eeec5127-2b45-41a7-bce7-3bd58ea6beeb% TcpCommunicationSpi:480 - Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/172.16.41.3:47100, failureDetectionTimeout=10000]

2016-12-01 03:03:56 ERROR marshaller-cache-#83%vertx.ignite.node.eeec5127-2b45-41a7-bce7-3bd58ea6beeb% CacheContinuousQueryHandler:495 - Failed to send event notification to node: 7c3bd833-ee3c-44b3-978c-240fafedad56

class org.apache.ignite.IgniteCheckedException: Failed to send message (node may have left the grid or TCP connection cannot be established due to firewall issues) [node=TcpDiscoveryNode [id=7c3bd833-ee3c-44b3-978c-240fafedad56, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.16.41.3], sockAddrs=[/0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0, /172.16.41.3:0], discPort=0, order=3, intOrder=3, lastExchangeTime=1480590055914, loc=false, ver=1.7.0#20160801-sha1:383273e3, isClient=true], topic=T4 [topic=TOPIC_CACHE, id1=1fd3a002-42a8-3e13-a1aa-bf164b7f2d64, id2=0e00d397-537c-4a7d-a890-d9d9123f2b62, id3=1], msg=GridContinuousMessage [type=MSG_EVT_NOTIFICATION, routineId=9ed9d5d9-9daf-49d6-a6c4-b961a831e85d, data=null, futId=null], policy=2]

        at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1309)

        at org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1540)

        at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1337)

        at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1308)

        at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1290)

        at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:945)

        at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:888)

        at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:787)

        at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$700(CacheContinuousQueryHandler.java:91)

        at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$1.onEntryUpdated(CacheContinuousQueryHandler.java:412)

        at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:347)

        at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2583)

        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2252)

        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1652)

        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1490)

        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:2950)

        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:130)

        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:268)

        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:266)

        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:748)

        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:353)

        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:277)

        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:88)

        at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:231)

        at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1238)

        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:866)

        at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:106)

        at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:829)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send message to remote node: TcpDiscoveryNode [id=7c3bd833-ee3c-44b3-978c-240fafedad56, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.16.41.3], sockAddrs=[/0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0, /172.16.41.3:0], discPort=0, order=3, intOrder=3, lastExchangeTime=1480590055914, loc=false, ver=1.7.0#20160801-sha1:383273e3, isClient=true]

        at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1996)

        at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1936)

        at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1304)

        ... 30 more

Caused by: class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=7c3bd833-ee3c-44b3-978c-240fafedad56, addrs=[/172.16.41.3:47100, /0:0:0:0:0:0:0:1%lo:47100, /127.0.0.1:47100]]

        at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2499)

        at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2140)

        at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2034)

        at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1970)

        ... 32 more

        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address: /172.16.41.3:47100

                at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2504)

                ... 35 more

        Caused by: java.net.SocketTimeoutException

                at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)

                at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2363)

                ... 35 more

        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address: /0:0:0:0:0:0:0:1%lo:47100

                at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2504)

                ... 35 more

Any help would be appreciated. 

Thanks

[Andrey Gura]

Anil,

from logs I see that your server node can't sent notification to client node. Ignite cluster manager doesn't start any clients node. So I can imagine that you have client node in docker container or in other subnet. If it is your case you should configure AddressResolver for your client node. Otherwise please provide more information about your set up.

[Anil]

Thanks Andrey !