Connection timeout while reading from S3

[Mayank Ahuja]

In our project (Hadoop based) we are using 0.7.1 version of JetS3t for reading and writing to S3. To make use of the new features we have upgraded the version to 0.8.1. After the upgrade, there was an intermittent failure (see stack trace below) where the connection was reset while retrieving metadata of an object.

a) Have anybody experienced the same while using jet3t 0.8.1 ? We store/retrieve a lot of data to/from S3 but had never seen this error with earlier version.

b) We are using httpclient version 3.0.1. It does not look like the fix for connection leaks in version 3.1 is applicable in this case. Can it be linked to older httpclient version.

c) We changed the value of property s3service.admin-max-thread-count to 20 (default for 0.8.1 version) from 10 (default for older version). Same is true for httpclient.max-connections. Can this be the reason ?

Greatly appreciate any advice.

java.net.SocketException: Connection reset
	at java.net.SocketInputStream.read(SocketInputStream.java:185)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:312)
	at sun.security.ssl.InputRecord.read(InputRecord.java:350)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:850)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:807)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:94)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
	at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
	at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
	at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
	at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
	at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
	at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:366)
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:874)
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:1950)
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:1877)
	at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:2388)
	at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1682)
	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:150)

...

[Mayank Ahuja]

Correction to subject: 

Connection reset while reading from S3

[James Murty]

Hi,

Are you able to use the latest version of JetS3t, 0.9.0? It uses HttpClient version 4.x and updated configuration settings which might resolve this issue.

Aside from that suggestion:

a) I'm not aware of widespread "Connection reset" issues with JetS3t 0.8.1, but that mightn't mean much since things may have changed in S3 since that version was released

b) It might be a JetS3t bug or a HttpClient bug or just a configuration issue in one of these, but could be tricky to track down. This blog post [1] has one suggestion involving tweaking the connection configuration. However, since you're using old versions of both JetS3t and HttpClient if it is a bug you're unlikely to be able to get bug fixes unless you find and fix them yourselves, since the communities have moved on. If you need to stick with HttpClient 3.x versions, I'd definitely recommend you try with the very latest 3.x version to rule out any issues in the older version.

c) Increasing the size of the thread and connection pool should not cause any problems. As long as max-connections is as large as the largest *max-thread-count setting, this is fine.

[1]: http://subodh-gupta.blogspot.com.au/2011/03/javanetsocketexception-connection-reset.html

---
jamesmurty.com

--
You received this message because you are subscribed to the Google Groups "JetS3t Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/jets3t-users/-/M_MvrvrIjScJ.
To post to this group, send email to jets3t...@googlegroups.com.
To unsubscribe from this group, send email to jets3t-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/jets3t-users?hl=en.

[Mayank Ahuja]

Thanks James for your reply.

Since there are other components which have dependency on httpclient, we want to do the migration in steps. I can definitively try out the latest 3.x httpclient.

In this case what is the expected behavior ? Looks like a re-attempt in case of "connection reset" exception would be a safe option.

[James Murty]

In this case what is the expected behavior ? Looks like a re-attempt in case of "connection reset" exception would be a safe option.

Yes, I think an automatic retry would be sensible when you get "connection reset" errors, so if you can apply some extra logic in your application to do this you could at least work around the issue.

[Mayank Ahuja]

I had another question related to 'Connection reset' stack we observed while connecting to S3. 

Amazon S3 has a limit of accepting maximum 100 requests per connection (http://aws.amazon.com/articles/1904). How does jets3t handles this case ?

Thanks

Mayank

[James Murty]

JetS3t doesn't do anything special to handle the 100-requests-per-connection behaviour of S3.

This should be handled reasonably well (though not ideally) by checking for stale HTTP connections -- work which is done by the underlying HTTPClient library according to the "httpclient.stale-checking-enabled" property that JetS3t sets to true by default. See RestUtils#initHttpConnection for the relevant code.

James

---
jamesmurty.com

--

You received this message because you are subscribed to the Google Groups "JetS3t Users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/jets3t-users/-/Ll7qhcDeHGcJ.

[Mayank Ahuja]

Thanks for your reply.

Did some more investigations around http retry method in the release 0.8.1 and have made some changes to fix it. This might not be applicable for new releases as the httpclient library has changed. This might be useful for those trying to use 0.8.1 release. 

The http retry method in httpclient library version 3.x does not retry in case the http request was sent successfully but might fail for other reasons (e.g. server closing the connection - like more than 100 requests per connection to S3 server). Also in the case of socket read timeouts, the retry attempt is not made. 

We observed SocketException (connection reset) and SocketTimeoutException (read timeout) while sending requests to S3 servers. We have replaced the httpclient retry handler with custom retry handler (similar to one available in earlier releases).

Hope it is useful for someone using jets3t 0.8.1. 

Regards

Mayank

Qubole (www.qubole.com)