sporadic timeouts with S3 - best approach advice?

[aweber1nj]

OK, I'm using jets3t to store a lot of content, and it works great 99% of the time (really a lot better than that percentage).  Occassionally I get a stack trace starting with something like:

Request Error: org.apache.http.conn.ConnectTimeoutException: Connect to s3.amazonaws.com/s3.amazonaws.com/72.21.195.33 timed out at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:2486)

. . .

Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to s3.amazonaws.com/s3.amazonaws.com/72.21.195.33 timed out

at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:377)

at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)

at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149)

at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)

at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:561)

at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415)

at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)

at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:334)

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:281)

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:942)

at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2148)

at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2075)

at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:2483)

 

I notice a few properties in jets3t.properties that have to do with timeouts and retries, and I have not changed them from the defaults.  Has anyone else seen these sporadic errors?  Does anyone have any ideas how to set the retries/timeouts so that maybe jets3t can get around them?

I have a few other sporadic events, but will separate them into other threads.

Thanks in advance,

AJ

[James Murty]

Hi AJ,

I haven't run into these specific errors myself so I don't have specific advice. What follows is largely speculation, but hopefully it will help put you on the right track.

James

First of all, make sure that your "httpclient.max-connections" setting in JetS3t's properties file is as large as the greater of your *.max-thread-count and *.admin-max-thread-count settings. If your connection pool is smaller than the number of simultaneous connection threads you're using you're bound to run into trouble.

If the "httpclient.max-connections" setting isn't the problem, you could try bumping up the timeout values for "httpclient.connection-timeout-ms" and "httpclient.socket-timeout-ms" to see if that helps, though the default values of 60 seconds should be plenty for connections to be established so I worry that increasing the timeout would, at best, mask a deeper underlying issue.

JetS3t's REST service doesn't automatically retry low-level Exceptions like this, namely IOException errors without any kind of server-side response code. This might be something the service should handle better, but again that existing 60 second timeout mean there's likely a deeper issue.

Per the HttpClient documentation [1] this error might be caused by a real connection timeout, or by a timeout "waiting for an available connection from an HttpConnectionManager". I don't think the latter is happening though because JetS3t also sets a zero value for the HttpClient connection manager timeout, so I believe that should wait indefinitely, but I could be wrong.

If there's not an obvious fix then debugging this issue will involve examining and tweaking the HttpClient connection settings applied by JetS3t's RestUtils class, in particular the #createConnectionPool and #initHttpConnection methods which translate JetS3t's properties into HttpClient equivalents.

[1]: http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/ConnectTimeoutException.html

---
jamesmurty.com

--
You received this message because you are subscribed to the Google Groups "JetS3t Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/jets3t-users/-/D0gZCJRsg3kJ.
To post to this group, send email to jets3t...@googlegroups.com.
To unsubscribe from this group, send email to jets3t-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/jets3t-users?hl=en.

[aweber1nj]

James,

One thing that occurred to me is that I may be trying to use the S3Service (not S3Multi, etc.) from multiple threads -- threading is a little hard to track sometimes, as you know.  Could it be that when there is a putObject going on from one thread, another thread using the same S3Service to try and getObjectDetails or other call that has to go to S3 via httpclient is blocking and eventually timing-out?

I still can't see any calls going over 60sec unless something is really slowing down the internet or my servers at that point, but I'm just curious if I should be looking into using one of the threaded S3Service impl's instead of the "simple one" I'm using now?

-AJ

[James Murty]

Hi,

JetS3t's services are designed to be thread-safe and should definitely be shared between multiple threads since the underlying HTTP pooling and connection management is performed at the service level. In other words, you should definitely use a single service (or as few as possible) across multiple threads.

You could try using one of JetS3t's multi-threaded services, but it's unlikely this would help your symptoms since these just run multiple threads with a single service like you are already doing.

The next place to look might be the underlying HTTPClient connection settings. JetS3t sets some of these with defaults which should be sensible, but may not be in your case, and the HTTPClient library itself allows many different settings that could be worth looking in to. That's pretty vague advice, I know, but I'm afraid I don't have anything better to offer.

Good luck, let us know how you go with it.

James

---
jamesmurty.com

--

You received this message because you are subscribed to the Google Groups "JetS3t Users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/jets3t-users/-/bEB23c37g4QJ.