java.net.SocketTimeoutException: Read timed out - jets3t-0.7.3.jar

[hluu]

Hi,

I have an application that constantly sends data up to S3 (whenever
there is new data) and after running for a little while (like a day or
two), then it gets SocketTimeoutException. Once this issue starts
happening for a particular S3Object, then the retry wouldn't help
because it keeps getting the same "SocketTimeoutException - Read timed
out" problem. Since my application do retry until a particular
S3Object is successfully sent to S3, it basically gets into an
infinite loop state.

Below is the stack trace. This application is using the default
HTTPClient time out, which I believed is 60 seconds.

My application is using jets3t-0.7.3.jar.

Any suggestions or ideas will be greatly appreciated.

Caused by: org.jets3t.service.S3ServiceException: Request Error. PUT '/
8161-1090238161' on Host '<bucket name I intentionally removed>'
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:
551)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestPut(RestS3Service.java:
940)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.createObjectImpl(RestS3Service.java:
1754)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.putObjectImpl(RestS3Service.java:
1685)
at org.jets3t.service.S3Service.putObject(S3Service.java:2145)
at com.<company name>.aws.S3.NFS3.putObjectNoResponse(NFS3.java:320)
... 7 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:155)
at com.ibm.jsse2.a.a(a.java:74)
at com.ibm.jsse2.a.a(a.java:8)
at com.ibm.jsse2.jc.a(jc.java:325)
at com.ibm.jsse2.jc.a(jc.java:554)
at com.ibm.jsse2.e.read(e.java:16)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:229)
at java.io.BufferedInputStream.read(BufferedInputStream.java:246)
at
org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:
78)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:
106)
at
org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:
1116)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager
$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:
1413)
at
org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:
1973)
at
org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:
1735)
at
org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:
1098)
at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:
398)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:
171)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:
397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:
323)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:
370)


Thanks,

Hien

[James Murty]

Hi Hien,

If you are seeing SocketTimeoutExceptions when uploading content to S3 it is possible that you are providing empty, or pre-consumed, input streams to the S3Objects you are sending. Someone had a similar issue recently where the input stream used in an S3Object had already been completely read, see http://groups.google.com/group/jets3t-users/browse_thread/thread/3d477d93ab80c585?hl=en

That would also explain why retrying doesn't help. It's trying to resend some amount of data, but all the data isn't available.

Can you try resetting your input streams before uploading S3Objects?

James

--
You received this message because you are subscribed to the Google Groups "JetS3t Users" group.
To post to this group, send email to jets3t...@googlegroups.com.
To unsubscribe from this group, send email to jets3t-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/jets3t-users?hl=en.

[hluu]

Hi James,

Thanks for the quick response.

You are right!! The root cause is reusing the same S3Object inside
the retry logic without resetting dataInputStream. I was not aware
that S3Object is kind of "stateful" (the dataInputStream)

Simply resetting the dataInputStream solves the issue, or creating a
brand new S3Object.

I noticed there are two different exceptions for the same underlying
cause and this is driven by the presence of a property setting
"httpclient.socket-timeout-ms".

With the "httpclient.socket-timeout-ms" property setting, I saw the
following:

Caused by: org.jets3t.service.S3ServiceException: S3 Error Message.
PUT '/my_small_object' on Host <bucketname>.s3.amazonaws.com' @ 'Tue,
27 Jul 2010 23:20:49 GMT' -- ResponseCode: 400, ResponseStatus: Bad
Request, XML Error Message: <?xml version="1.0" encoding="UTF-8"?
><Error><Code>RequestTimeout</Code><Message>Your socket connection to
the server was not read from or written to within the timeout period.
Idle connections will be closed.</Message><RequestId>5611CCACCDAD7E2E</
RequestId><HostId>sW4Mcwn0rxOsFlDkoN0HpzsHGry1hKL6RyvKGzinen0pYcbDb2SSEUWEZDjQQpLU</
HostId></Error>
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:
452)

at
org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestPut(RestS3Service.java:
940)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.createObjectImpl(RestS3Service.java:
1754)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.putObjectImpl(RestS3Service.java:
1685)
at
org.jets3t.service.S3Service.putObject(S3Service.java:2145)

Without the "httpclient.socket-timeout-ms" property setting, I saw the
following:

Caused by: org.jets3t.service.S3ServiceException: Request Error. PUT '/

my_small_object' on Host '<bucketname>.s3.amazonaws.com'

at
org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:
551)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestPut(RestS3Service.java:
940)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.createObjectImpl(RestS3Service.java:
1754)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.putObjectImpl(RestS3Service.java:
1685)
at
org.jets3t.service.S3Service.putObject(S3Service.java:2145)

... 25 more

Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native
Method)
at
java.net.SocketInputStream.read(SocketInputStream.java:155)

at com.ibm.jsse2.a.a(a.java:140)
at com.ibm.jsse2.a.a(a.java:96)
at com.ibm.jsse2.jc.a(jc.java:356)
at com.ibm.jsse2.jc.a(jc.java:538)
at com.ibm.jsse2.e.read(e.java:28)

Any ideas why that is the case?


Thanks a lot.

Hien

On Jul 26, 9:56 pm, James Murty <jamu...@gmail.com> wrote:
> Hi Hien,
>
> If you are seeing SocketTimeoutExceptions when uploading content to S3 it is
> possible that you are providing empty, or pre-consumed, input streams to the
> S3Objects you are sending. Someone had a similar issue recently where the

> input stream used in an S3Object had already been completely read, seehttp://groups.google.com/group/jets3t-users/browse_thread/thread/3d47...

> > jets3t-users...@googlegroups.com<jets3t-users%2Bunsubscribe@google groups.com>
> > .

[James Murty]

Hi,

These two errors differ because they are come from different sides of the client-to-S3 communication. In the first case, S3 realizes that you aren't sending the data it expects and cuts you off with a service-side error. In the second case the exception is thrown from the client side. The exact reasons for the differences will be somewhat buried in the HTTP library plumbing.

The S3Object's are stateful if you provide them an input stream or string, but if you create file-based S3Object's instead the input stream will be created as required behind the scenes. In general, though, it's best not to reuse S3Objects since you can accidentally end up duplicating metadata fields.

James

To unsubscribe from this group, send email to jets3t-users...@googlegroups.com.