Irregular ReplicationTaskProcessor on aws

[em...@marcoaust.de]

Hi,

thanks for providing such great software as open source!

I am running eureka (1.4.10) on AWS in 2 availability zones (eu-central-1a & eu-central-1b) with DNS & replication for 3,5 months now.

For some reason I am seeing irregularly the following error in my logs (2-20 per day on both instances). It seems that it is not causing any problems and the retry is always succesful but I am a bit concerned what the reason might be as the log is on error level.

Any ideas what the cause might be?

Thanks
Marco

Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: ERROR c.n.e.c.ReplicationTaskProcessor: Network level connection to peer ec2-52-58-XX-XXX.eu-central-1.compute.amazonaws.com; retrying after delay
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:187)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at com.netflix.eureka.cluster.DynamicGZIPContentEncodingFilter.handle(DynamicGZIPContentEncodingFilter.java:48)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at com.netflix.discovery.EurekaIdentityHeaderFilter.handle(EurekaIdentityHeaderFilter.java:27)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at com.sun.jersey.api.client.Client.handle(Client.java:652)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:570)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at com.netflix.eureka.transport.JerseyReplicationClient.submitBatchUpdates(JerseyReplicationClient.java:116)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at com.netflix.eureka.cluster.ReplicationTaskProcessor.process(ReplicationTaskProcessor.java:71)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at com.netflix.eureka.util.batcher.TaskExecutors$BatchWorkerRunnable.run(TaskExecutors.java:187)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at java.lang.Thread.run(Thread.java:745)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: Caused by: java.net.SocketTimeoutException: Read timed out
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at java.net.SocketInputStream.socketRead0(Native Method)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at java.net.SocketInputStream.read(SocketInputStream.java:170)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at java.net.SocketInputStream.read(SocketInputStream.java:141)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:223)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:117)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:173)
Sep 13 04:41:23 ip-10-0-11-165 lsm-eureka-52.29.XXX.XX: ... 10 common frames omitted

[dl...@netflix.com]

Hi Marco,

The eureka server should be able to handle these kinds of errors in general, so isolated incidents should be ok. The exception itself is a standard Java SocketTimeoutException so it is hard to link it to any specific problem however. If you have insight into the system level metrics of your EC2 instances, you can try to see if there are abnormal tcp activity happening when you see these log errors?

Thanks