Amazon’s S3, JetS3t, MultiThreadedHttpConnectionManager, and Unable to get a connection
Posted by jonchase
Apologies for the long title.
This entry focuses on a recent issue when using Amazon’s S3 in combination with the popular JetS3t library (version 0.5.0). If that means absolutely nothing to you, you can stop reading this entry now:D.
Update: Amazon has reverted S3 back to its prior state, completely fixing this issue.
Update: new link added at bottom - a JetS3t developer has contributed an official fix.
On January 11th, Amazon apparently made some changes in S3 so that their servers do not explicitly release HTTP connections. As long as your code is explicitly closing the HTTP connection to S3 (i.e. connection.release()), this shouldn’t affect you (indeed, most people using S3 weren’t affected).
However, if you’re using the JetS3t library, you may have noticed an exception similar to the following after, oh, I don’t know, exactly 4 uploads to S3:
2008-01-13 16:12:55,753 DEBUG [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] - HttpConnectionManager.getConnection: config = HostConfiguration[host=https://s3.amazonaws.com], timeout = 0
2008-01-13 16:12:55,753 DEBUG [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] - enter HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
2008-01-13 16:12:55,753 DEBUG [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] - Unable to get a connection, waiting..., hostConfig=HostConfiguration[host=https://s3.amazonaws.com]
Why is this happening? It turns out that the REST implementation of JetS3t doesn’t explicitly close the HTTP connection after a successful operation. It also turns out that the max number of connections JetS3t uses by default is 4 (see “httpclient.max-connections”). Thus, once you do 4 uploads to S3, the above error starts appearing (”error” is misleading - I had to turn logging to DEBUG level to get the above message to print out).
The fix is to update the JetS3t RestS3Service.java code to explicitly release the connection, which is exactly what I did (add “httpMethod.releaseConnection();” on line 571). I’ve attached the code here for those interested. Note that I haven’t extensively tested this, and I’m sure there’s a better place to call the release method, but this works for now, and is provided on an AS IS basis (NO WARRANTIES - don’t hold me or Juliesoft LLC liable for anything that this code breaks). Hopefully the JetS3t developers integrate the needed fix into the next version.
- - removed - see update below
Update: looks like James Murty, the JetS3t library author, has taken care of the issue and posted all of the details.
Because of the post from James, I’ve removed my versions of the fix and will instead direct you to his.
On a side note, I have to say how impressed I am with the Amazon AWS community members, the Amazon staffers moderating the forums, and James Murty - the amount of communication and the speed of the resolution for this problem make me glad that I’ve been lucky enough to fall into such an active community!
Leave a Reply
Subscribe to Jon's Blog