Broken pipe

[david.loy]

I am currently using:
<dependency>
<groupId>net.java.dev.jets3t</groupId>
<artifactId>jets3t</artifactId>
<version>0.9.0</version>
</dependency>

When I attempt to upload a 13G file, after about 15 minutes I'm seeing:
org.jets3t.service.S3ServiceException: Request Error:
java.net.SocketException: Broken pipe
at org.jets3t.service.S3Service.putObject(S3Service.java:2267)
at org.cdlib.mrt.s3.sdsc.SDSCCloud.putObject(SDSCCloud.java:169)
at org.cdlib.mrt.s3.sdsc.SDSCCloud.putObject(SDSCCloud.java:229)

This is a standalone test so their should not be any problems of
overloading the network.

Does anyone have a suggestion for a next step for debugging this. I
don't even know if jets3t attempted to write anything.

Any suggestion would be terrific.

Thanks
David

[James Murty]

Hi David,

I suspect the root problem is that you cannot upload a 13GB file to S3 using a standard PUT (putObject) request since the maximum size of any uploaded object is 5GB. S3 supports storage of larger objects using "multipart" operations -- which allows sub-5GB objects to be uploaded then re-assembled service-side -- but it doesn't look like you're using this approach?

For debugging purposes you are probably best off turning on HTTP header logging for the HttpClient library JetS3t uses. JetS3t comes with a log4j.properties file that includes a line for tweaking this setting, or you can set the following property wherever your Log4J settings are drawn from:

log4j.logger.org.apache.http.headers=DEBUG

I suspect S3 is cutting off the pipe when it realises a 5GB+ object is incoming, though I would have expected it to return an error immediately (via the 100-continue mechanism) rather than waiting 15 minutes. Are you setting the Content-Length header prior to upload to inform the service of the incoming object's size?

Hope this helps,

James

---
jamesmurty.com

David

--
You received this message because you are subscribed to the Google Groups "JetS3t Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jets3t-users+unsubscribe@googlegroups.com.
To post to this group, send email to jets3t...@googlegroups.com.
Visit this group at http://groups.google.com/group/jets3t-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[david.loy]

Hi James
Thank you so much - that definitely is the problem. I do set the length but there may be other issues causing delays.

I'll try your example.

Thanks
David

---
jamesmurty.com

To unsubscribe from this group and stop receiving emails from it, send an email to jets3t-users...@googlegroups.com.

To post to this group, send email to jets3t...@googlegroups.com.
Visit this group at http://groups.google.com/group/jets3t-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "JetS3t Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jets3t-users...@googlegroups.com.