Failed to upgrade a swift FS

[Nicolas Gif]

Hi,

I am upgrading some S3QL Swift FS as I upgraded from v2.21 to 3.3.2.

It work well for 2 FS but crashed for the 3rd.

I fskd'ed it with v2.21 without problem before attempting to upgrade it.

Then here is what I get:

s3qladm upgrade swiftks://auth.cloud.ovh.net/GRA:XXXXXX

Getting file system parameters..
Detected Swift features for storage.gra.cloud.ovh.net:443: copy via COPY, Bulk delete 1000 keys at a time, maximum meta value length is 255 bytes


I am about to update the file system to the newest revision.
You will not be able to access the file system with any older version
of S3QL after this operation.


You should make very sure that this command is not interrupted and
that no one else tries to mount, fsck or upgrade the file system at
the same time. You may interrupt the update with Ctrl+C and resume
at a later time, but the filesystem will not be in a usable state
until the upgrade is complete.


Please enter "yes" to continue.
> yes
Downloading and decompressing metadata...
Reading metadata...
..objects..
..blocks..
..inodes..
..inode_blocks..
..symlink_targets..
..names..
..contents..
..ext_attributes..
Upgrading from revision 23 to 24...
ERROR: Thread Thread-2 terminated with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/s3ql-3.3.2-py3.7-linux-x86_64.egg/s3ql/common.py", line 373, in run
    self.run_protected()
  File "/usr/local/lib/python3.7/site-packages/s3ql-3.3.2-py3.7-linux-x86_64.egg/s3ql/common.py", line 425, in run_protected
    self.target(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/s3ql-3.3.2-py3.7-linux-x86_64.egg/s3ql/adm.py", line 488, in upgrade_loop
    meta = backend.lookup(key)
  File "/usr/local/lib/python3.7/site-packages/s3ql-3.3.2-py3.7-linux-x86_64.egg/s3ql/backends/comprenc.py", line 92, in lookup
    return self._verify_meta(key, meta_raw)[1]
  File "/usr/local/lib/python3.7/site-packages/s3ql-3.3.2-py3.7-linux-x86_64.egg/s3ql/backends/comprenc.py", line 175, in _verify_meta
    % (stored_key, key))
s3ql.backends.common.CorruptedObjectError: Object content does not match its key (s3ql_metadata vs s3ql_metadata_bak_0)
..processed 133/14171 objects (0%)..ERROR: Uncaught top-level exception:
Traceback (most recent call last):
  File "/usr/local/bin/s3qladm", line 11, in <module>
    load_entry_point('s3ql==3.3.2', 'console_scripts', 's3qladm')()
  File "/usr/local/lib/python3.7/site-packages/s3ql-3.3.2-py3.7-linux-x86_64.egg/s3ql/adm.py", line 98, in main
    return upgrade(options)
  File "/usr/local/lib/python3.7/site-packages/s3ql-3.3.2-py3.7-linux-x86_64.egg/s3ql/common.py", line 441, in wrapper
    return fn(*a, **kw)
  File "/usr/local/lib/python3.7/site-packages/s3ql-3.3.2-py3.7-linux-x86_64.egg/s3ql/adm.py", line 399, in upgrade
    update_obj_metadata(backend, backend_factory, db, options)
  File "/usr/local/lib/python3.7/site-packages/s3ql-3.3.2-py3.7-linux-x86_64.egg/s3ql/adm.py", line 450, in update_obj_metadata
    t.join_and_raise()
  File "/usr/local/lib/python3.7/site-packages/s3ql-3.3.2-py3.7-linux-x86_64.egg/s3ql/common.py", line 395, in join_and_raise
    raise EmbeddedException(exc_info, self.name)
s3ql.common.EmbeddedException: caused by an exception in thread Thread-2.
Original/inner traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/s3ql-3.3.2-py3.7-linux-x86_64.egg/s3ql/common.py", line 373, in run
    self.run_protected()
  File "/usr/local/lib/python3.7/site-packages/s3ql-3.3.2-py3.7-linux-x86_64.egg/s3ql/common.py", line 425, in run_protected
    self.target(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/s3ql-3.3.2-py3.7-linux-x86_64.egg/s3ql/adm.py", line 488, in upgrade_loop
    meta = backend.lookup(key)
  File "/usr/local/lib/python3.7/site-packages/s3ql-3.3.2-py3.7-linux-x86_64.egg/s3ql/backends/comprenc.py", line 92, in lookup
    return self._verify_meta(key, meta_raw)[1]
  File "/usr/local/lib/python3.7/site-packages/s3ql-3.3.2-py3.7-linux-x86_64.egg/s3ql/backends/comprenc.py", line 175, in _verify_meta
    % (stored_key, key))
s3ql.backends.common.CorruptedObjectError: Object content does not match its key (s3ql_metadata vs s3ql_metadata_bak_0)

Now I don't know what to do ...

Can you help me to fix that mess ?

[Nicolas Gif]

Well, I retried with --debug. It's long but it worked. Don't know what happened.

. It's long but it worked ...