Jersey2ReplicationClient is not ready for error responses with body

104 views
Skip to first unread message

Panagiotis Partheniadis

unread,
Feb 28, 2018, 10:03:30 AM2/28/18
to eureka_netflix

Preconditions:
Cluster of custom Eureka servers using eureka-core 1.1.131 (dropwizard-based). And resource /peerreplication/batch is not registered on those servers, so fallback to one-by-one task is required for replication to work.
What I do:
Add updated Eureka server to the cluster. New server has version 1.8.6 of Eureka libraries and uses Jersey 2 client. And /peerreplication/batch resource is registered this time.
What I expect:
Replication from old Eureka to new Eureka works because there is replication resource
Replication from new Eureka to old Eureka works because though there is no replication resource on old Eureka, new Eureka performs fallback to one-by-one replication
What I get instead:
Replication from old Eureka to new Eureka works because there is replication resource
Replication from new Eureka to old Eureka doesn't work and I see an exception:

javax.ws.rs.ProcessingException: Error closing message content input stream.
at org.glassfish.jersey.message.internal.EntityInputStream.close(EntityInputStream.java:161)
at org.glassfish.jersey.message.internal.InboundMessageContext$EntityContent.close(InboundMessageContext.java:156)
at org.glassfish.jersey.message.internal.InboundMessageContext.close(InboundMessageContext.java:939)
at org.glassfish.jersey.client.InboundJaxrsResponse.close(InboundJaxrsResponse.java:167)
at com.netflix.eureka.transport.Jersey2ReplicationClient.submitBatchUpdates(Jersey2ReplicationClient.java:124)
at com.netflix.eureka.cluster.ReplicationTaskProcessor.process(ReplicationTaskProcessor.java:72)
at com.netflix.eureka.util.batcher.TaskExecutors$BatchWorkerRunnable.run(TaskExecutors.java:187)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 196; received: 0
at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:180)
at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:200)
at org.apache.http.impl.io.ContentLengthInputStream.close(ContentLengthInputStream.java:103)
at org.apache.http.impl.execchain.ResponseEntityProxy.streamClosed(ResponseEntityProxy.java:140)
at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)
at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:174)
at java.io.BufferedInputStream.close(BufferedInputStream.java:483)
at java.io.FilterInputStream.close(FilterInputStream.java:181)
at org.glassfish.jersey.apache.connector.ApacheConnector$3.close(ApacheConnector.java:644)
at java.io.FilterInputStream.close(FilterInputStream.java:181)
at org.glassfish.jersey.apache.connector.ApacheConnector$HttpClientResponseInputStream.close(ApacheConnector.java:621)
at org.glassfish.jersey.message.internal.EntityInputStream.close(EntityInputStream.java:158)
... 7 common frames omitted

First, a quick question: could you please tell, is there actually a fallback from batched to one-by-one replication in Eureka 1.8.6? I couldn't find it in code, though definitely saw it in older versions, like 1.1.126.

Second, looks like Jersey2ReplicationClient is not ready to 404 with body. In that case when we call response.close() in line 124 https://github.com/Netflix/eureka/blob/master/eureka-core-jersey2/src/main/java/com/netflix/eureka/transport/Jersey2ReplicationClient.java#L124 , Jersey issue jersey/jersey#3486 arises, and exception is being rethrown from finally block. Because if (!isSuccess(response)), we don't consume body. I understand that having peer Eureka which returns error with body is quite an unusual case, but maybe it is worth caring. But not sure yet how.
And even if there is a fallback to one-by-one replication, it will unlikely work, because when exception is thrown from finally, code is usually not ready to handle it.


Reply all
Reply to author
Forward
0 new messages