retry handlers for response.read()

[Robert Coup]

Hey all,

reviewing our S3 looping/retry code (again) and found that _mexe() does a whole bunch of sensible things, including dumping the HTTP connection and re-opening it, etc. That works great for many requests. We'd previously been wrapping all that in more retry logic, so i thought i'd look into removing it.

However, i discovered that for S3 Keys (and probably other requests) you get exceptions thrown from response.read() which happens outside the _mexe() retry logic... presumably something similar  occurs for put_file() too. When _mexe() was originally introduced [1] you said you had something else in mind for put_file? Any further thoughts down that path?

Off the top of my head a clever with() statement might deal with it, or maybe a decorator which essentially does the same thing as _mexe...

Cheers,

Rob :)

[1] http://groups.google.co.nz/group/boto-users/msg/c02005fc80cc1a26?hl=en

[mitch]

Hi -

Currently, for PUTing files to S3, all Key methods end up calling
Key.send_file() which does actually call make_request which in turns
calls _mexe(). So, the retry logic in _mexe should be working for
PUT's. For GET's, all Key methods end up calling Key.open_read()
which also calls make_request and ultimately _mexe(). So, my initial
look suggests that _mexe retry logic should be in place for both reads
and writes of Keys. Is that what you see?

Mitch

[Robert Coup]

On Wed, Jun 17, 2009 at 4:00 AM, mitch <Mitch....@gmail.com> wrote:

Currently, for PUTing files to S3, all Key methods end up calling
Key.send_file() which does actually call make_request which in turns
calls _mexe().  So, the retry logic in _mexe should be working for
PUT's.  For GET's, all Key methods end up calling Key.open_read()
which also calls make_request and ultimately _mexe().  So, my initial
look suggests that _mexe retry logic should be in place for both reads
and writes of Keys.  Is that what you see?

Yep, except that _mexe is long gone once Key.next() does a response.read(), which can throw socket/httplib errors as well..

Rob :)