Large file upload to S3

[Andrew Stevens]

Hi,

   What's the current multipart upload limit to S3? Documentation says 100GB, however we have successfully uploaded 1TB files with Cyberduck. Unfortunately anything slightly larger seems to cause Cyberduck to crash whilst it's running overnight, and the resumption of multipart upload is failing, always returning to 0% complete.

(view multipart uploads permission is added to the account, and other options successfully resume with it)

[Forrest Aldrich]

Chiming in here,

You may find the open-source tool "rclone" to be more useful here:

https://github.com/rclone/rclone

You can expand it's uploading into sections, control bandwidth usage, etc.  I use something like this for large directory structures:

rclone copy --update --verbose --transfers 30 --checkers 8 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --stats 1s "/Volumes/some/Path" "googledrive:Path/directory/"

I have used this to transfer a huge amount of data, it rarely fails and will resume.   There are many options you can pass to it.

This project is actively developed, too.

Hope this helps.



Forrest

--
--
You received this message because you are subscribed to the Google Groups "Cyberduck" group. To post to this group, send email to cybe...@googlegroups.com
To unsubscribe from this group, send email to cyberduck+...@googlegroups.com
For more options, visit this group at https://groups.google.com/group/cyberduck
--
Post bug reports and feature requests
https://trac.cyberduck.io/newticket
--
Support development
https://cyberduck.io/donate/
---
You received this message because you are subscribed to the Google Groups "Cyberduck" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cyberduck+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cyberduck/d4aa4b38-06bb-4493-b5dc-97fb79f273dan%40googlegroups.com.

[Andrew Stevens]

Thanks Forrest, will give it a go, appreciate it!