Lots of "java.io.IOException: Connection reset by peer "

[Tim Lindgren]

Hello!

I have a Vertx HTTP server up and running and that in turn is using the WebClient to send requests to downstream services. The setup is working very nicely. However, I see quite a lot of "java.io.IOException: Connection reset by peer" from the io.vertx.core.net.impl.ConnectionBase class.

The documentation states: "It means that the client is resetting the HTTP connection instead of closing it. This message also indicates that you may have not consumed the complete payload (the connection was cut before you were able to)."

As far as I have understood from some forum posts, it might not be a problem and might even be expected. However, I would like to add some data to the log output to control it and perhaps remove it. I am still not sure if the error is for the Vertx HTTP server or WebClient here. So I tried to add an exceptionHandler for the HTTP server.

Here is some pseudo code I use when setting up the server:

  Router router = Router.router(vertx);

//init the router, skipping that part here.

vertx.createHttpServer(new HttpServerOptions().setIdleTimeout(75)).requestHandler(router::accept).exceptionHandler(new Handler<Throwable>() {       

    @Override

    public void handle(Throwable event) {

    logger.error("Caught with exceptionhandler, exception: " + event.getMessage(), event);

    }

}).listen(port, result -> { if (result.succeeded()) { logger.info("Verticle started at: " + port); future.complete(); } else { logger.error("Verticle failed", result.cause()); future.fail(result.cause()); } });

According to the documentation, the ExceptionHandler is used for the handshake of a new connection (?) so might not be useful for my case.

On the WebClient side I simply do this:

HttpRequest<Buffer> req = webClient.get(service.port, service.host, "/" + requestURI); req.timeout(requestParameters.readTimeout) .as(BodyCodec.buffer()) .rxSend();

               ....

Not sure how I would get some exceptionHandler or similar attached there.

For completeness sake I added an ExceptionHandler to Vertx itself:

vertx.exceptionHandler(new Handler<Throwable>() {       

    @Override

    public void handle(Throwable event) {

    logger.error("Caught with exceptionhandler, exception: " + event.getMessage(), event);

    }

});

Very vert helpful for any information since I have so many hours/days ob this!

Cheers,

Tim

[Thomas SEGISMONT]

On your client code, you invoke rxSend() but do you subscribe to the returned Single?

To better understand what's happening a simple reproducer on GitHub would be best.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/828f8349-bbb8-42b7-9df4-786103826311%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Lindgren]

Yes, I subscribe on the Single and it works as intended.

I should probably mention tI only see this error log on our production environment, I have not seen it on our test or QA environments. That is also the reason I haven't been able to reproduce it or know how to create a reproducer on GitHub. :/

I am wondering if this might be mobile clients misbehaving when communicating with our Vertx HTTP server. Or if this is our WebCient having some problems for internal requests (but then I believe I should see it on the test environments as well).

Cheers,

Tim

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

[Alex Z.]

Tim.... 

The following does not help you directly, but we also have (had now) lots of "java.io.IOException: Connection reset by peer" in production. What we ended up doing is creating a logback (https://logback.qos.ch/) filter to filter out these messages that we polluting our prod logs. Since we send our logs to Splunk, it was also a matter of reducing costs. We see these messages consistently in every environment. We do not use  WebCient since other services consume ours

Alex

[Manish Kumar]

Tim - It's okay to have this error if you are using connection pooling. You may see this error even though there is no traffic flowing over the n/w. A connection maybe reset literally by anyone on your n/w (firewall, remote server and list goes on).

As far as attaching the exception handler, you will have to add it to connection, e.g.

httpClientRequest.connectionHandler(conn -> {
    conn.exceptionHandler(ex -> {
    });
});

There isn't much Vertx can do here as Java/Oracle throws a very generic error(IOException) for RST status and Vertx won't know which one to log/throw and which one to ignore.

As far as reproducing is concerned, you can try using this tool to forcefully terminate your connection - http://www.nirsoft.net/utils/cports.html#DownloadLinks I think it may not generate RST response but you will get the idea what your are dealing with (thanks to Java).

[Tim Lindgren]

Thanks Alex and Manish!

I have thought about ignoring the error log by using a log filter (we are using Log4j2) but I really wanted to know why it happens and also prefarably add some context-dependent information to the log output. 

The WebClient we are using for internal traffic is set up to use HTTP 2 which means, as far as I understand, that a connection pool is not used. I am using the following configuration for the WebClient:

WebClientOptions webClientOptions = new WebClientOptions()

.setMaxPoolSize(200)

.setConnectTimeout(1000)

.setProtocolVersion(HttpVersion.HTTP_2)

.setSsl(false)

.setHttp2ClearTextUpgrade(false)

.setUseAlpn(false);

WebClient webClient = WebClient.create(vertx, webClientOptions);

Even though maxPoolSize is set, I thought that was only used if a HTTP 1.1 connection was established. Assuming HTTP 2 is used, can the WebClient cause the log outputs for "Connection reset by peer"?

Best regards,

Tim

[Tim Lindgren]

Byt the way, is it possible to set a connection handler on a request created by WebClient? HTTP Client is the underlying resource but haven't found out how to actually set the connection handler.

[Julien Viet]

you have a separate maxPoolSize for HTTP2 : http2MaxPoolSize

Julien

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/2fdb309d-c85c-4e0e-824e-d4e21b617b2c%40googlegroups.com.

[Julien Viet]

The WebClient fully manages that HttpClientRequest/HttpClientResponse

if you need some hooks, you can cast WebClient to WebClientInternal and set interceptors

Julien

On 19 Dec 2017, at 13:34, Tim Lindgren <tim.li...@snowprintstudios.com> wrote:

Byt the way, is it possible to set a connection handler on a request created by WebClient? HTTP Client is the underlying resource but haven't found out how to actually set the connection handler.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/58aa5c91-807a-40f3-8d4a-2e7455a96798%40googlegroups.com.

[Tim Lindgren]

Thanks Julien, I haven't seen the interceptor support!

I guess something like this might achieve what I am looking for then?

WebClientInternal webClientInternal = (WebClientInternal) webClient;

webClientInternal.addInterceptor(this::exceptionLoggingInterceptorHandler);

       ....

private void exceptionLoggingInterceptorHandler(HttpContext context) {

Handler<AsyncResult<io.vertx.ext.web.client.HttpResponse<Object>>> responseHandler = context.getResponseHandler();

context.setResponseHandler(ar -> {

if (ar.succeeded()) {

responseHandler.handle(ar);

} else {

logger.error("Caught error during request " + ar.cause());

responseHandler.handle(ar);

}

});

context.next();

}

However, I am using the rxified version of things and it seems I have to use io.vertx.ext.web.client.HttpRequest for that WebClientInternal and that doesn't support rxsend for example. Any advice for that as well?

Cheers, Tim

[Manish Kumar]

Tim regarding your question on why this error happens, you can refer to this. Although error mentioned in this post not exactly same but reasons are similar to what you see.

[Tim Lindgren]

Thanks all! It seems like filtering the error log is an acceptable solution that many chose to use. A similar forum post is the following https://groups.google.com/forum/#!topic/vertx/gS7Q-YcDTko which I think explains things nicely.