Missing parts in multipart uploads

[Bryce Utting]

Hi,

Uploading reasonably-large files (1.7 GB, 2 GB) in 5 MB parts with
upload_part_from_file(), I'm noticing some parts quietly disappearing;
six from one file, nine from another.

What application logs I have aren't showing anything suspicious, so I
suspect that key.send_file() may be mishandling return codes (or S3
mightn't be returning the codes we'd hope for...), but can't prove it.
That aside, is there a way of confirming specific parts short of
retrieving the complete part list, or should I just pull the whole list
in anyway?

This -may- be related to using multiple threads to speed things up. (A
local test with a 1.4 GB file showed a speedup from 11 hours to 11
*minutes* with 20 threads, FWIW. Boggleage.) I haven't seen any
evidence of threads crashing, though.

2.0b4 on Python 2.5.2.

cheers,
Bryce

[Mitchell Garnaat]

Hi -

This commit:

https://github.com/boto/boto/commit/1b15436dc20fe68ed04820f18abd80f7f6fd648b

which occurred after the 2.0b4 release corrected an issue in boto related to the handling of error responses from S3 on multipart uploads.  For multipart, even though you receive a 200 response the response could still contain an error message in the body.  This was being ignored by boto and therefore failing silently.  I suspect that is the problem you are running into.

Can you try using the latest code in github master?

Mitch

--
You received this message because you are subscribed to the Google Groups "boto-users" group.
To post to this group, send email to boto-...@googlegroups.com.
To unsubscribe from this group, send email to boto-users+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/boto-users?hl=en.