Rename of files larger than 5gb fails

[Stefan Hasse]

Hi,

i got some troubles when integration jetS3t, whereas I don't know if this is a known issue, not supported by amazon, just something new or maybe (hopefully) I'm just wrong...

I have to upload a 5GB+ file to a bucket and afterwards rename it (move within same bucket and temporary filename is replaced by another one).

The multi-part upload works fine, but when renaming I get the following exception:
org.jets3t.service.ServiceException: S3 Error Message. -- ResponseCode: 400, ResponseStatus: Bad Request, XML Error Message: <Error><Code>InvalidRequest</Code><Message>The specified copy source is larger than the maximum allowable size for a copy source: 5368709120</Message><RequestId>BA777F9***</RequestId><HostId>aVoMVP6C8uJ3BKaeoqSLS3ayy3n4tyucJ***</HostId></Error>
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:427)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:281)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestPut(RestStorageService.java:1043)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.copyObjectImpl(RestStorageService.java:2029)
at org.jets3t.service.StorageService.copyObject(StorageService.java:871)
at org.jets3t.service.StorageService.copyObject(StorageService.java:916)
at org.jets3t.service.StorageService.moveObject(StorageService.java:965)
at org.jets3t.service.StorageService.renameObject(StorageService.java:1011)


Like I know, it's already something else to copy files larger than 5 GB (http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html), but this is just renaming, see code snippet:
  public void rename(Ticket from, Ticket to) throws StorageServiceException {
    try {
      logger.debug("Begin renaming of {} into {}", from, to);
      service.renameObject(bucketName, from.getId(), to.toS3Object());
      logger.debug("{} renamed into {}", from, to);
    } catch (Exception e) {
      throw new StorageServiceException(String.format("Fail to rename ticket from '%s' to '%s'", from.toString(), to.toString()), e);
    }
  }


I'm grateful for any suggestion.

Thanks,
Steven

[James Murty]

Hi Steven,

I'm afraid JetS3t's renameObject() convenience method dates from the days before S3 supported 5GB+ object sizes and it is unable to copy large objects. Behind the scenes, this method just performs a standard PUT with the "x-amz-copy-source" header.

To copy large objects you will need to use the S3Service#multipartUploadPartCopy method, along with all the other fiddly overhead of initiating a multipart upload in advance and completing it when all the upload/copy operations are completed. JetS3t does not currently have a convenience method for handling all of this work, and even though you're notionally doing something very simple when just "renaming" a large object the API steps are actually fairly complex.

See JetS3t's code for the MultipartUtils#uploadObjects and S3Service#putObjectMaybeAsMultipart methods which are convenience methods for uploading multiple files as multipart objects, or normal ones, based on file sizes. You might need something similar in your app to handle object renames, using a simple approach when possible or the multipart-upload-copy approach when necessary.

Hope this helps,

James

---
jamesmurty.com

--
You received this message because you are subscribed to the Google Groups "JetS3t Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jets3t-users...@googlegroups.com.
To post to this group, send email to jets3t...@googlegroups.com.
Visit this group at http://groups.google.com/group/jets3t-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.