IOException: Connection reset by peer

[Rafael Merino García]

Hi guys,

We are using Vertx3 version 3.3.3 and we got randomly the following error:

java.io.IOException: Connection reset by peer

    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)

    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)

    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)

    at sun.nio.ch.IOUtil.read(IOUtil.java:192)

    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)

    at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)

    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1100)

    at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:367)

    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:118)

    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:610)

    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:551)

    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:465)

    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:437)

    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)

    at java.lang.Thread.run(Thread.java:745) 

We are using the Vertx asynchronous http client and after switching the flag keepalivee from true to false, the error never happened again. any idea?

thanks in advance

[Tim Fox]

This just means the connection was closed by the other end (the peer).

So, if your HTTP client is sending requests to a web server, then it means the web server has closed the connection. This is quite normal, web servers will typically close idle keep alive connections after a few seconds.

[Rafael Merino García]

I understand. The thing is that our tests fail, I mean, it's sort of the Vertx http client is trying to reuse a closed connection...

thanks for your quick response.

[Tim Fox]

I can't see anything here that points to Vert.x trying to reuse a closed connection. The stack trace above shows the exception was thrown from a read, not a write.

[Rafael Merino García]

Sorry, you are right, I copied and pasted another exception we are getting. Find the bellow the stack trace you were expecting:

2017-04-24 14:08:23,863 [vert.x-eventloop-thread-14] ERROR io.vertx.core.impl.ContextImpl Unhandled exception

io.vertx.core.VertxException: Connection was closed

at io.vertx.core.http.impl.ClientConnection.handleClosed(ClientConnection.java:396)

at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:314)

at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:190)

at io.vertx.core.net.impl.VertxHandler.channelInactive(VertxHandler.java:97)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:255)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:241)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:234)

at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:417)

at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:360)

at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:325)

at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:256)

at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:220)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:255)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:241)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:234)

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1329)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:255)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:241)

at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:908)

at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:744)

at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:440)

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)

at java.lang.Thread.run(Thread.java:745)

As a I said, it appears randomly and changing the flag keepalive from true (default value) to false made it disappear.

Regards

[Rafael Merino García]

Hi again,

I don't know if it is related to the issue we are talking about, but  just in case... We are finishing every http connection in our handlers by:

HttpServerResponse resp = ....

resp.end(...)

res.close()

If we use a client configured using  either keepalive=true or keepalive=false, does it have anything to do with the way we finish connections? I mean, after resp.end() when is resp.close() mandatory?

Regards

[Tim Fox]

Right, as I mentioned previously, the server will close keep alive connectioins - this is normal. This will cause the close handler on the client to be called:

https://github.com/eclipse/vert.x/blob/3.3.3/src/main/java/io/vertx/core/http/impl/ClientConnection.java#L396

^^ this is what is displaying in your logs. I don't see anywhere a closed connection attempting to be used.

Not really sure what the issue is here... do you have a reproducer?

[Tim Fox]

res.close() will close the underlying connection. There is no need to do this unless you really want to close it for some reason (?)

But, as I mentioned before, it shouldn't matter if the server closes the connection or not, the client should continue to function, as its normal behaviour for servers to close connections and clients need to deal with this gracefully...

[Rafael Merino García]

Hi again,

After you explanation I know I think what's going on. Those exceptions are normal stuff printed out by the server and we can see many more depending on how the server and client handle the connections, got it, so far so good. On the other hand, for my tests I'm using the Vertx http Client as I told you, and this object is being created by using the same vertx instance as the server:

//Server under test

Vertx vertx = Vertx.vertx(new VertxOptions());

//http client used for testing

HttpClient client = vertx.createHttpClient(options)

And for some reason those exceptions, when they are thrown by Vertx, are breaking up my tests. Maybe I should use another vertx intance to create the http client...

Does it make any sense to you?

Thank you very much for your help

[Tim Fox]

It's very difficult to make a diagnosis without seeing a reproducer (e.g. a test that demonstrates the issue)