Exceptions from Wasabi degraded service

[Anthony DeRobertis]

Wasabi is experiencing issues currently, and its greatly reduced
capacity results in overloaded messages. Adding a quick bit of debug
code to print one out, they look like this:

503 Service Unavailable
Connection: close
Content-Type: application/xml
Date: Mon, 07 Oct 2019 20:02:31 +0000
Content-Length: 218

<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/"><Error><Type>Sender</Type><Code>TemporarilyUnavailable</Code><Message>Maximum number of server active requests exceeded</Message></Error></ErrorResponse>

Unfortunately, S3QL (still on 3.0 here, but looks to be the same in HEAD
from the code) doesn't handle them. Most trivially, the line:

raise get_S3Error(tree.findtext('Code'), tree.findtext('Message'), resp.headers)

doesn't work; seems it needs
tree.findtext('.//{https://iam.amazonaws.com/doc/2010-05-08/}Code') and
the same with Message to actually find the elements. (Or of course pass
a namespace map to abbreviate). Not sure if actual Amazon S3 produces
the same XML or not...

Other than this, it'd be nice of S3QL would retry these requests,
instead of exiting. But at least the above gets the actual error reported.

[Anthony DeRobertis]

With this, adding:

    if code == 'TemporarilyUnavailable':
        code = 'ServiceUnavailable'

to get_S3Error (near the existing fix for Google Code issue #369) seems to at least get it retrying. Not sure yet if this fsck will ever finish. At least until Wasabi fixes the problem.

[drobertson adhis]

Thanks for mentioning this - I just a signed up for a wasabi demo account to test with S3QL and was having issues, but it looks like availability and performance problems with the service.

I'll wait for it to work better before continuing.

[Anthony DeRobertis]

This is happening again, the following (clearly a hack) patch seems to
keep it working... though I suspect it doesn't work with other S3
implementations. (Also, there are other searches for the Code tag...
maybe they could break too).


--- s3c.py.ORIG 2019-12-07 15:07:17.405625340 -0500
+++ s3c.py 2019-12-07 15:14:49.015102089 -0500
@@ -143,7 +143,8 @@

if isinstance(exc, (InternalError, BadDigestError, IncompleteBodyError,
RequestTimeoutError, OperationAbortedError,
- SlowDownError, ServiceUnavailableError)):
+ SlowDownError, ServiceUnavailableError,
+ TemporarilyUnavailableError)):
return True

elif is_temp_network_error(exc):
@@ -559,7 +560,7 @@
self._dump_response(resp, body))
raise

- raise get_S3Error(tree.findtext('Code'), tree.findtext('Message'), resp.headers)
+ raise get_S3Error(tree.findtext('.//{https://iam.amazonaws.com/doc/2010-05-08/}Code'), tree.findtext('.//{https://iam.amazonaws.com/doc/2010-05-08/}Message'), resp.headers)

def _parse_xml_response(self, resp, body=None):
'''Return element tree for XML response'''
@@ -1031,5 +1032,6 @@
class RequestTimeoutError(S3Error): pass
class SlowDownError(S3Error): pass
class ServiceUnavailableError(S3Error): pass
+class TemporarilyUnavailableError(S3Error): pass
class RequestTimeTooSkewedError(S3Error): pass
class NoSuchBucketError(S3Error, DanglingStorageURLError): pass