httplib.IncompleteRead

[Eric Allen]

Hey guys,

I'm running into a hard-to-reproduce issue with boto and S3, where some downloads will throw an httplib.IncompleteRead, and subsequent retries continue to fail. I've only been able to catch it in production once every few days, and we're reading tens of thousands of files per day. Even after I call bucket.connection.close(), subsequent requests will immediately raise IncompleteRead errors. Has anybody else run into this?

Traceback (most recent call last):

  ...

  File "<file_wrapper.py",> line 22, in next

    line = self.readline()

  File "<file_wrapper.py",> line 37, in readline

    data = self.fh.read(self.buffer_size)

  File "<virtualenv/lib/python2.6/site-packages/boto/s3/key.py",> line 378, in read

    self.close()

  File "<virtualenv/lib/python2.6/site-packages/boto/s3/key.py",> line 349, in close

    self.resp.read()

  File "<virtualenv/lib/python2.6/site-packages/boto/connection.py",> line 411, in read

    self._cached_response = httplib.HTTPResponse.read(self)

  File "/usr/lib/python2.6/httplib.py", line 529, in read

    s = self._safe_read(self.length)

  File "/usr/lib/python2.6/httplib.py", line 621, in _safe_read

    raise IncompleteRead(''.join(s), amt)

I've tried toggling is_secure both ways with no impact.

my environment:

- Amazon EC2
- Ubuntu 11.10

- Python 2.6.7

- boto 2.12.0

-Eric

[Jonathan Lefman]

I've run into the same issue with v2.24.0. I would really like to find a viable workaround. None of my experiments have produced a solution. Were you able to resolve your issue?

Thanks,

J

[Christopher Rutter]

I am not sure that this is necessarily a "boto" issue.  I have been noticing similar issues w/ downloads from S3 using simple curl calls with publicly hosted tarballs.  Sometimes the download doesn't complete successfully, even after 5 retries.  These objects are all "standard" storage class (not reduced redundancy), without server-side encryption, and are hosted in the US Standard region.  I typically wait a few minutes and try again, and all is well.

Seems to me that Amazon S3 is just flaky at times.

-Chris

[Mitchell Garnaat]

Just out of curiosity, what is the location constraint for your bucket?  And what s3 service endpoint are you using to get to the bucket?

Mitch

--
You received this message because you are subscribed to the Google Groups "boto-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to boto-users+...@googlegroups.com.
To post to this group, send email to boto-...@googlegroups.com.
Visit this group at http://groups.google.com/group/boto-users.
For more options, visit https://groups.google.com/d/optout.

[Chris Rutter]

There is no location constraint on the bucket:

(Pdb) bucket = boto.s3.get_bucket('<bucket_name>')

(Pdb) bucket.get_location()

''

I am using wget (not curl as I previously mentioned) in the following form: 

wget -q -O /tmp/out.tgz https://s3.amazonaws.com/<bucket_name>/<directory>/<object>.tgz

-Chris

--
You received this message because you are subscribed to a topic in the Google Groups "boto-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/boto-users/YiPAOvxIrUY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to boto-users+...@googlegroups.com.