Problem downloading files from Amazon S3. java.net.SocketTimeoutException: Read timed out

[Gowtham Rupavatharam]

Hi James,
I have been facing the
java.net.SocketTimeoutException: Read timed out exception while
downloading files from Amazon S3. I have a good internet connection so
connectivity is not an issue definitely. The bucket i'm trying to
download is a US-based bucket and I have set s3service.disable-dns-
buckets property to true also so that it's accessible very fast.

So, basically to give a introduction of what we do.....we have an
application where we store the user files on Amazon S3 with private
access. So for downloading the files our users hit a Servlet on our
server and we use the following Jets3t method
org.jets3t.service.StorageService.getObject(String, String) to get the
S3Object and we copy the
org.jets3t.service.model.StorageObject.getDataInputStream() to
ServletOutputStream using
org.apache.cxf.helpers.IOUtils.copy(InputStream, OutputStream) method.
I have tried downloading a 30 MB file and file downloads starts and it
progresses until around 1 MB or so and then it suddenly stops and
checking the server logs here is the stack trace

java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at
com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:
293)
at
com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:
405)
at
com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:
798)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:
755)
at
com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:
75)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:
256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:
317)
at
org.apache.commons.httpclient.ContentLengthInputStream.read(ContentLengthInputStream.java:
170)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at
org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:
108)
at
org.jets3t.service.io.InterruptableInputStream.read(InterruptableInputStream.java:
78)
at
org.jets3t.service.impl.rest.httpclient.HttpMethodReleaseInputStream.read(HttpMethodReleaseInputStream.java:
136)
at java.io.InputStream.read(InputStream.java:85)
at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:112)
at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:75)


This happens consistently for a file size above 5 - 10 MB.

For couple of other reasons we cannot do a generate a signed URL for
that S3Object and re-direct the user to that signedURL.

Let me know if I'm doing something wrong or you have any suggestions
for me

Thanks
Gowtham

[James Murty]

Hi Gowtham,

My first thought with this issue is that tying the S3 download
directly to your servlet output stream, and thus your end-user's
download speed, could be a problem. S3 is fairly unforgiving about
slight read/write delays, so anything that causes the download to
pause for a couple of seconds will likely lead S3 to close down the
connection.

I'm not sure that's the cause of the timeouts in this case, but it is
likely to become a problem in general since any pause in your
end-user's file download will probably cause your server's S3 download
to also pause. That is, assuming the servlet's output stream has some
kind of write-ahead buffer that can be exceeded, which is likely.

Can you decouple the two downloads by downloading from S3 to a
temporary file on your server, then sending an input stream of that
temporary file to your end-user? The mechanics of doing this could get
complex if the user's download speed is faster than the speed from S3,
but I suspect you're going to have to do something like this.

Aside from that, I'd look into the buffer sizes of all the relevant
streams. One of these buffer sizes will likely correspond to roughly
the size where your downloads start failing.


James

> --
> You received this message because you are subscribed to the Google Groups "JetS3t Users" group.
> To post to this group, send email to jets3t...@googlegroups.com.
> To unsubscribe from this group, send email to jets3t-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/jets3t-users?hl=en.
>