MultiThreadedHttpConnectionManager doesn't call releaseConnection on success

[jch...@gmail.com]

Using version 0.5.0.

I use JetS3t RestS3Service to upload files to S3 and am getting errors
like the following after a few uploads (the last line is the important
one):

2008-01-13 16:12:55,752 DEBUG
[org.jets3t.service.impl.rest.httpclient.RestS3Service] - Performing
PUT request, expecting response code 200
2008-01-13 16:12:55,752 DEBUG
[org.jets3t.service.impl.rest.httpclient.RestS3Service] - Adding
authorization for AWS Access Key 'XXXXXXXXXXXXXXXX'.
2008-01-13 16:12:55,753 DEBUG
[org.jets3t.service.impl.rest.httpclient.RestS3Service] - Canonical
string ('|' is a newline): PUT||image/jpeg|Sun, 13 Jan 2008 21:12:55
GMT|/download.sendalong.com/9eacff66-7308-4ad3-b3b2-e608afab9491
2008-01-13 16:12:55,753 DEBUG
[org.apache.commons.httpclient.HttpClient] - enter
HttpClient.executeMethod(HostConfiguration,HttpMethod)
2008-01-13 16:12:55,753 DEBUG
[org.apache.commons.httpclient.HttpClient] - enter
HttpClient.executeMethod(HostConfiguration,HttpMethod,HttpState)
2008-01-13 16:12:55,753 DEBUG
[org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] -
enter
HttpConnectionManager.getConnectionWithTimeout(HostConfiguration,
long)
2008-01-13 16:12:55,753 DEBUG
[org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] -
HttpConnectionManager.getConnection: config =
HostConfiguration[host=https://s3.amazonaws.com], timeout = 0
2008-01-13 16:12:55,753 DEBUG
[org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] -
enter
HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
2008-01-13 16:12:55,753 DEBUG
[org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] -
Unable to get a connection, waiting...,
hostConfig=HostConfiguration[host=https://s3.amazonaws.com]

It's saying that it's unable to get a connection. This is with
httpclient.max-connections=4 (the default).

If I set httpclient.max-connections=1, the error happens much more
quickly...like after one upload to S3.

I stepped through some of the code, and I can see several places where
MultiThreadedHttpConnectionManager calls releaseConnection() to return
the http connection back to the pool, however, I can't find anywhere
this happens on a successful PUT (i.e. I can only find it happening in
catch statements). I've also looked through the logs with TRACE level
enabled and I can see where MultiThreadedHttpConnectionManager calls
getConnection, but don't see any logs for releaseConnection.

I updated RestS3Service and added httpMethod.releaseConnection() at
line 975 and my problems seems to have disappeared. (I'm sure this
probably isn't the best place to put the call to releaseConnection(),
but I just took a shot and it seemed to work.)

Also, here's the code I use to actually perform the PUT:

S3Object o = new S3Object(bucket, file);
o.setContentType(contentType);
o.addMetadata("Content-Disposition", "attachment;
filename=" + friendlyName);
s3.putObject(bucket, o);
o.closeDataInputStream();


If this is a bug, I can file a bug report if needed. Any chance this
is already fixed in CVS?

Strangely enough, this started happening on Jan. 11th, the same day
that some other users reports some problems with S3 itself. I wonder
if they are somehow related, as I hadn't had this problem before the
11th. Here's the thread for reference:
http://developer.amazonwebservices.com/connect/thread.jspa?threadID=19024&start=0&tstart=0

[James Murty]

Thanks for your patch Jonathan,

you are right that the issue can be fixed by explicitly releasing the connection when a PUT request has completed. I have just committed an official fix to CVS, and have started a new thread in the S3 developer forums to publicise it and provide a number of ways to obtain it:

http://developer.amazonwebservices.com/connect/thread.jspa?threadID=19076&tstart=0

The fix is only slightly different to the one you mentioned, it just invokes the releaseConnection() method in a different place.

For completeness, I will include the full text of the thread's opening comment below so the information is easier to find in a Google search.

James


[Full text of leading comment in " Fix for JetS3t timeouts" thread in the S3 developers forum]

The issue

Recently, S3 slightly changed the way it sends response messages when a PUT request is performed. Unfortunately, this change has surprised some S3 client tools, including JetS3t, and has resulted in uploads failing due to connection starvation. See this thread to read about the symptoms of the problem:
http://developer.amazonwebservices.com/connect/thread.jspa?threadID=19024&tstart=0

The boring details

It now appears to some client programs that S3's responses to PUT requests include data in the response body, when in fact this "data" is zero bytes long. The presence of this phantom data prevents PUT connections from being cleaned up automatically.

The fix

Here are the steps I have taken to fix this issue for JetS3t users. Let me know if I can do more to help out.

- The online version of Cockpit has been updated with a fix and should now work OK.

- The latest version of the JetS3t codebase in CVS includes a patch for this issue. Check out this update if you can.

- If you are using the last public release of JetS3t (version 0.5.0) you can download a Jar file containing the patch from the following URL. Put this file in your "jets3t-0.5.0/jars" directory, replacing the original version.

http://s3.amazonaws.com/jets3t/applets-jets3t-0.5.0/jets3t-0.5.0.jar

- To fix the 0.5.0 code or your own derived version, add the following code before the "return" line of the performRestPut() method in RestS3Service.java (eg at line 571 in the original 0.5.0 release)

// Release connection after PUT (there ought to be no response content) httpMethod.releaseConnection();

Thanks to those people who alerted me to this issue and provided suggestions and patches. I hope this update helps.

James