Multipart Uploads

[James Murty]

Amazon has announced another new feature, this time for S3: Multipart
Uploads. With multipart uploads you can send object data to S3 in
chunks and re-assemble the parts once they have all arrived.

This makes it possible to speed uploads of large objects by performing
multiple transmissions in parallel, or to reduce the risk and
inconvenience of transmission failures by sending large objects in
small parts at a time.

There are more details in the announcement:
http://aws.typepad.com/aws/2010/11/amazon-s3-multipart-upload.html

As of this evening JetS3t has basic toolkit support for Multipart
Uploads in the S3Service/RestS3Service classes. The API functions are
all in place and working, and you can see sample usage in the test
code: http://bitbucket.org/jmurty/jets3t/src/39dfc9fc633b/src/org/jets3t/tests/TestRestS3Service.java#cl-235

With this sophisticated new S3 feature there is plenty of scope for
better tool-level support. At present JetS3t will only help with the
fundamentals; to gain all the benefits of this feature you will need
to do much of the work to prepare and perform uploads yourself.

To help improve JetS3t's support I'd love to get feedback on how you
expect to use this feature, and as always patches and code
contributions are very welcome.

James

[David Kocher]

James,

Great to see jets3t is always part of the latest and greatest. Are there any plans to make this transparent to the API user by hiding it into a standard output stream that would take care of allocating the chunks?

-
David

> --
> You received this message because you are subscribed to the Google Groups "JetS3t Users" group.
> To post to this group, send email to jets3t...@googlegroups.com.
> To unsubscribe from this group, send email to jets3t-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/jets3t-users?hl=en.
>

[James Murty]

Hi David,

I do intend to offer some kind of convenience tools in JetS3t for
using the new multipart uploads feature, but I'm not yet sure what
form these should take. So I'm open to suggestions.

I also need to see what helpers Amazon have put in their own toolkit
for this. However, this isn't something I will have time to work on in
the next few weeks.

James

[Adrian Petrescu]

> I also need to see what helpers Amazon have put in their own toolkit
> for this. However, this isn't something I will have time to work on in
> the next few weeks.

In case you haven't already noticed it, the AWS SDK for Java has a
nice high-level "TransferManager" class:
https://github.com/amazonwebservices/aws-sdk-for-java/commit/f281199903631dfef8f5f98fe16fb9f0fe074c6b#diff-39