IOException: No route to host

[Tomáš Heřman]

Hey guys, 

i am having this problem:

I have a chronicle server in USA and chronicle clients in Europe. Sometimes the connection between USA and Europe gets broken. This seems to cause following exception on the server side:

java.io.IOException: No route to host

at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_45]

at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_45]

at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_45]

at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_45]

at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_45]

at net.openhft.chronicle.tcp.TcpConnection.readAvailable(TcpConnection.java:233) ~[receiver-1.2.2-20150624103534-f7eac68.jar:na]

at net.openhft.chronicle.tcp.TcpConnection.readFullyOrEOF(TcpConnection.java:225) ~[receiver-1.2.2-20150624103534-f7eac68.jar:na]

at net.openhft.chronicle.tcp.SourceTcp$SessionHandler.readUpTo(SourceTcp.java:323) ~[receiver-1.2.2-20150624103534-f7eac68.jar:na]

at net.openhft.chronicle.tcp.SourceTcp$SessionHandler.onRead(SourceTcp.java:349) ~[receiver-1.2.2-20150624103534-f7eac68.jar:na]

at net.openhft.chronicle.tcp.SourceTcp$SessionHandler.onSelectionKey(SourceTcp.java:334) ~[receiver-1.2.2-20150624103534-f7eac68.jar:na]

at net.openhft.chronicle.tcp.SourceTcp$SessionHandler.vanillaNioLoop(SourceTcp.java:246) ~[receiver-1.2.2-20150624103534-f7eac68.jar:na]

at net.openhft.chronicle.tcp.SourceTcp$SessionHandler.run(SourceTcp.java:199) ~[receiver-1.2.2-20150624103534-f7eac68.jar:na]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]

at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]

The weird thing is that this doesn't seem to cause any exception or connection time out or whatever on the client side. I have set up my chronicle clients in such a way that they will try to automatically reconnect if something goes wrong but it seems that this use case never gets registered. I would expect the client to blow up with exception when connection is broken. 

Any idea why is this happening? Or how to force client to blow up if something like this happens? Thanks!

[Rob Austin]

could you write a unit test ( preferably using loopback, so that we can exercise the test from junit) that demonstrates this issue. ( preferably via a pull request ) - thanks

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Lawrey]

Hello Tomas,

This is almost certainly a issue with the network connection, not on either machine but the network between the client and the server.

We could only simulate this error as we are not hardware specialists. It is unlikely to be reproducible without a network which changing it's underlying routing.

While we cannot prevent this in software, we can try to make sure we handle it as best as possible. The question would be; how best to simulate this so that we have confidence we are handling this use case.

> The weird thing is that this doesn't seem to cause any exception or connection time out or whatever on the client side.

We treat this error as retryable unless the time out has been reached.  If you didn't see an error it is indeed retryable.

Regards,

   Peter.

[Tomáš Heřman]

Thanks for the answer, 

I checked with our sysadmin and he confirmed that there was a loss of connecting between our Prague datacenter and the US datacenter. So like you said, the error was in the underlying network. 

You said you are treating this error as retry-able. Does that mean that the client side of chronicle queue is trying to reestablish the connection and either it fails with exception or it gets reestablished and everything should work properly? Because from what I’ve seen in logs, it seems like the connection wasn’t reestablished, nor have i seen anything like ‘reconnecting to IP:PORT’ in the logs. There was no exception on the client side either. Logs look like there was never any error at all on the client. 

Maybe i need to set some timeout variable? Only one i have set on the client is 

   .reconnectTimeout(1, TimeUnit.SECONDS)

-- 

[Luca Burgazzoli]

What is the version you are using?

[Tomas Herman]

3.4.2

--
You received this message because you are subscribed to a topic in the Google Groups "Chronicle" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/java-chronicle/OgNhzHM0w8Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to java-chronicl...@googlegroups.com.