s3ql 3.8.1 still crashes on connection loss, routing changes

68 views

Skip to first unread message

Amos T

unread,

Feb 15, 2022, 6:49:16 AM2/15/22

to s3ql

i am sorry to say but s3ql keeps providing unhandled

error exceptions and still crashes when connections

are down or when the provider changes routing or s3

provider blocks (temporary) access.

log:

2022-02-14 19:03:59.995 2381527:Trio worker thread 929 s3ql.metadata.cycle_metadata: Backing up old metadata...
2022-02-14 19:07:20.606 2381527:Thread-9 s3ql.backends.common.wrapped: Encountered SSLError ([SSL: BAD_LENGTH] bad length (_ssl.c:2483)), retrying ObjectW.close (attempt 3)...
2022-02-14 19:29:58.464 2381527:Thread-2 s3ql.backends.common.wrapped: Encountered SSLError ([SSL: BAD_LENGTH] bad length (_ssl.c:2483)), retrying ObjectW.close (attempt 3)...
2022-02-14 21:23:04.989 2381527:Thread-7 s3ql.backends.common.wrapped: Encountered SSLError ([SSL: BAD_LENGTH] bad length (_ssl.c:2483)), retrying ObjectW.close (attempt 3)...
2022-02-14 23:16:22.325 2381527:Thread-10 s3ql.mount.exchook: Unhandled exception in thread, terminating
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/mount.py", line 58, in run_with_except_hook
run_old(*args, **kw)
File "/usr/lib/python3.9/threading.py", line 892, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/block_cache.py", line 318, in _upload_loop
self._do_upload(*tmp)
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/block_cache.py", line 383, in _do_upload
obj_size = backend.perform_write(do_write, 's3ql_data_%d'
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/common.py", line 108, in wrapped
return method(*a, **kw)
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/common.py", line 279, in perform_write
return fn(fh)
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/comprenc.py", line 389, in __exit__
self.close()
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/comprenc.py", line 383, in close
self.fh.close()
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/comprenc.py", line 548, in close
self.fh.close()
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/common.py", line 108, in wrapped
return method(*a, **kw)
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/s3c.py", line 913, in close
resp = self.backend._do_request('PUT', '/%s%s' % (self.backend.prefix, self.key),File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/s3c.py", line 532, in _do_request
self._parse_error_response(resp)
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/s3c.py", line 566, in _parse_error_response
raise get_S3Error(tree.findtext('Code'), tree.findtext('Message'), resp.headers)
s3ql.backends.s3c.S3Error: NoSuchEntity: The entity in your request cannot be found.
2022-02-14 23:16:22.385 2381527:Thread-10 root.excepthook: Uncaught top-level exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/mount.py", line 58, in run_with_except_hook
run_old(*args, **kw)
File "/usr/lib/python3.9/threading.py", line 892, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/block_cache.py", line 318, in _upload_loop
self._do_upload(*tmp)
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/block_cache.py", line 383, in _do_upload
obj_size = backend.perform_write(do_write, 's3ql_data_%d'
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/common.py", line 108, in wrapped
return method(*a, **kw)
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/common.py", line 279, in perform_write
return fn(fh)
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/comprenc.py", line 389, in __exit__
self.close()
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/comprenc.py", line 383, in close
self.fh.close()
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/comprenc.py", line 548, in close
self.fh.close()
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/common.py", line 108, in wrapped
return method(*a, **kw)
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/s3c.py", line 913, in close
resp = self.backend._do_request('PUT', '/%s%s' % (self.backend.prefix, self.key),
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/s3c.py", line 532, in _do_request
self._parse_error_response(resp)
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/s3c.py", line 566, in _parse_error_response
raise get_S3Error(tree.findtext('Code'), tree.findtext('Message'), resp.headers)
s3ql.backends.s3c.S3Error: NoSuchEntity: The entity in your request cannot be found.
2022-02-14 23:16:40.537 2381527:MainThread s3ql.mount.unmount: Unmounting file system...
2022-02-15 00:00:52.667 2381527:Thread-6 s3ql.mount.exchook: Unhandled top-level exception during shutdown (will not be re-raised)
2022-02-15 00:00:52.668 2381527:Thread-6 root.excepthook: Uncaught top-level exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/mount.py", line 58, in run_with_except_hook
run_old(*args, **kw)

File "/usr/lib/python3.9/threading.py", line 892, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/block_cache.py", line 318, in _upload_loop
self._do_upload(*tmp)
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/block_cache.py", line 383, in _do_upload
obj_size = backend.perform_write(do_write, 's3ql_data_%d'
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/common.py", line 108, in wrapped
return method(*a, **kw)
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/common.py", line 279, in perform_write
return fn(fh)
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/comprenc.py", line 389, in __exit__
self.close()
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/comprenc.py", line 383, in close
self.fh.close()
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/comprenc.py", line 548, in close
self.fh.close()
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/common.py", line 108, in wrapped
return method(*a, **kw)
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/s3c.py", line 913, in close
resp = self.backend._do_request('PUT', '/%s%s' % (self.backend.prefix, self.key),
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/s3c.py", line 532, in _do_request
self._parse_error_response(resp)
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/backends/s3c.py", line 566, in _parse_error_response
raise get_S3Error(tree.findtext('Code'), tree.findtext('Message'), resp.headers)
s3ql.backends.s3c.S3Error: NoSuchEntity: Cannot find user for the access key.
2022-02-15 01:17:59.768 2381527:Thread-7 s3ql.backends.common.wrapped: Encountered SSLError ([SSL: BAD_LENGTH] bad length (_ssl.c:2483)), retrying ObjectW.close (attempt 3)...
2022-02-15 01:18:02.346 2381527:Thread-4 s3ql.backends.common.wrapped: Encountered SSLError ([SSL: BAD_LENGTH] bad length (_ssl.c:2483)), retrying ObjectW.close (attempt 3)...
2022-02-15 01:18:02.448 2381527:Thread-11 s3ql.backends.common.wrapped: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying ObjectW.close (attempt 3)...
2022-02-15 02:58:14.676 2381527:Thread-1 s3ql.backends.common.wrapped: Encountered RequestTimeoutError (RequestTimeout: Your socket connection to the server was not read from or
written to within the timeout period, source: read tcp 130.117.252.13:443->178.119.8.200:12330: i/o timeout), retrying ObjectW.close (attempt 3)...
2022-02-15 02:58:14.792 2381527:Thread-1 s3ql.backends.common.wrapped: Encountered SSLError ([SSL: BAD_LENGTH] bad length (_ssl.c:2483)), retrying ObjectW.close (attempt 4)...
2022-02-15 03:01:14.242 2381527:Thread-8 s3ql.backends.common.wrapped: Encountered RequestTimeoutError (RequestTimeout: Your socket connection to the server was not read from or
written to within the timeout period, source: read tcp 130.117.252.11:443->178.119.8.200:7724: i/o timeout), retrying ObjectW.close (attempt 3)...
2022-02-15 03:01:14.361 2381527:Thread-8 s3ql.backends.common.wrapped: Encountered SSLError ([SSL: BAD_LENGTH] bad length (_ssl.c:2483)), retrying ObjectW.close (attempt 4)...

After a fsck:

Starting fsck of s3c://XXXX:443/XXX/

_do_request(): redirected to s3.eu-central-1.wasabisys.com
Using cached metadata.
ERROR: Uncaught top-level exception:
Traceback (most recent call last):
File "/usr/local/bin/fsck.s3ql", line 33, in <module>
sys.exit(load_entry_point('s3ql==3.8.1', 'console_scripts', 'fsck.s3ql')())
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/fsck.py", line 1175, in main
db = Connection(cachepath + '.db')
File "/usr/local/lib/python3.9/dist-packages/s3ql-3.8.1-py3.9-linux-x86_64.egg/s3ql/database.py", line 71, in __init__
cur.execute(s)
apsw.BusyError: BusyError: database is locked