Crashed filesystem after upgrade from s3ql 2.21 to 3.7.0

[Ben Hymers]

Hi, I recently upgraded Ubuntu from 18.04 to 20.04, and built and installed s3ql 3.7.0 (replacing an install of 2.21). All dependencies are at the correct versions according to the docs, and the tests ran successfully. I upgraded my Amazon S3 filesystem successfully. Unfortunately though it looks like there was a problem mounting it, and now it seems to be stuck in a crashed state. What can I do to recover this situation?

~# fsck.s3ql s3://XXX

ERROR: Can not check mounted file system.

~# umount.s3ql /tmp/XXX

ERROR: File system appears to have crashed.

~# tail ~/.s3ql/mount.log -n 55

2018-11-11 04:01:15.389 28329:MainThread s3ql.mount.main: All done.

2021-02-05 04:00:12.729 151015:MainThread s3ql.mount.determine_threads: Using 4 upload threads.

2021-02-05 04:00:12.782 151015:MainThread s3ql.mount.main: Autodetected 1048532 file descriptors available for cache entries

2021-02-05 04:00:13.402 151015:MainThread s3ql.metadata.download_metadata: Downloading and decompressing metadata...

2021-02-05 04:00:14.636 151015:MainThread s3ql.metadata.download_metadata: Reading metadata...

2021-02-05 04:00:14.673 151015:MainThread s3ql.metadata.restore_metadata: ..objects..

2021-02-05 04:00:14.810 151015:MainThread s3ql.metadata.restore_metadata: ..blocks..

2021-02-05 04:00:15.051 151015:MainThread s3ql.metadata.restore_metadata: ..inodes..

2021-02-05 04:00:16.348 151015:MainThread s3ql.metadata.restore_metadata: ..inode_blocks..

2021-02-05 04:00:17.118 151015:MainThread s3ql.metadata.restore_metadata: ..symlink_targets..

2021-02-05 04:00:17.118 151015:MainThread s3ql.metadata.restore_metadata: ..names..

2021-02-05 04:00:17.250 151015:MainThread s3ql.metadata.restore_metadata: ..contents..

2021-02-05 04:00:18.444 151015:MainThread s3ql.metadata.restore_metadata: ..ext_attributes..

2021-02-05 04:00:19.727 151015:MainThread s3ql.mount.main_async: Setting cache size to 147472 MB

2021-02-05 04:00:19.753 151015:MainThread s3ql.mount.main_async: Mounting s3://eu-west-1/benhymers-jack-backup at /tmp/XXX...

2021-02-05 04:00:19.854 151364:MainThread s3ql.daemonize.detach_process_context: Daemonizing, new PID is 151365

2021-02-05 04:00:57.074 151365:MainThread s3ql.mount.unmount: Unmounting file system...

2021-02-05 04:00:57.664 151365:MainThread root.excepthook: Uncaught top-level exception:

Traceback (most recent call last):

  File "/usr/local/lib/python3.8/dist-packages/s3ql-3.7.0-py3.8-linux-x86_64.egg/s3ql/database.py", line 143, in get_row

    row = next(res)

StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/usr/local/lib/python3.8/dist-packages/s3ql-3.7.0-py3.8-linux-x86_64.egg/s3ql/block_cache.py", line 472, in upload_if_dirty

    block_id = self.db.get_val('SELECT id FROM blocks WHERE hash=?', (hash_,))

  File "/usr/local/lib/python3.8/dist-packages/s3ql-3.7.0-py3.8-linux-x86_64.egg/s3ql/database.py", line 127, in get_val

    return self.get_row(*a, **kw)[0]

  File "/usr/local/lib/python3.8/dist-packages/s3ql-3.7.0-py3.8-linux-x86_64.egg/s3ql/database.py", line 145, in get_row

    raise NoSuchRowError()

s3ql.database.NoSuchRowError: Query produced 0 result rows

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/usr/local/bin/mount.s3ql", line 11, in <module>

    load_entry_point('s3ql==3.7.0', 'console_scripts', 'mount.s3ql')()

  File "/usr/local/lib/python3.8/dist-packages/s3ql-3.7.0-py3.8-linux-x86_64.egg/s3ql/mount.py", line 131, in main

    trio.run(main_async, options, stdout_log_handler)

  File "/usr/local/lib/python3.8/dist-packages/trio/_core/_run.py", line 1932, in run

    raise runner.main_task_outcome.error

  File "/usr/local/lib/python3.8/dist-packages/s3ql-3.7.0-py3.8-linux-x86_64.egg/s3ql/mount.py", line 289, in main_async

    unmount_clean = True

  File "/usr/local/lib/python3.8/dist-packages/trio/_core/_run.py", line 815, in __aexit__

    raise combined_error_from_nursery

  File "/usr/local/lib/python3.8/dist-packages/s3ql-3.7.0-py3.8-linux-x86_64.egg/s3ql/mount.py", line 749, in run

    await self.block_cache.upload_if_dirty(el)

  File "/usr/local/lib/python3.8/dist-packages/s3ql-3.7.0-py3.8-linux-x86_64.egg/s3ql/block_cache.py", line 491, in upload_if_dirty

    await self._queue_upload((el, obj_id))

  File "/usr/local/lib/python3.8/dist-packages/s3ql-3.7.0-py3.8-linux-x86_64.egg/s3ql/block_cache.py", line 536, in _queue_upload

    raise NoWorkerThreads('no upload threads')

s3ql.block_cache.NoWorkerThreads: no upload threads

[Nikolaus Rath]

On Feb 11 2021, Ben Hymers <b.hy...@gmail.com> wrote:
> "/usr/local/lib/python3.8/dist-packages/s3ql-3.7.0-py3.8-linux-x86_64.egg/s3ql/block_cache.py",
> line 536, in _queue_upload
> raise NoWorkerThreads('no upload threads')
> s3ql.block_cache.NoWorkerThreads: no upload threads

Does it help if you run with `--fg`? If so, you justs need to wait for the
next release for a proper fix.

Best,
-Nikolaus

--
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[Ben Hymers]

I'm sorry, which part should I try running with `--fg`? I tried it with fsck and umount and the options aren't recognised - and I saw '--force' is an option to fsck so I tried that too (output is below)

Thanks,

Ben

~# fsck.s3ql --fg s3://XXX

usage: fsck.s3ql [-h] [--log <target>] [--cachedir <path>] [--debug-modules <modules>] [--debug] [--quiet]

                 [--backend-options <options>] [--version] [--authfile <path>] [--compress <algorithm-lvl>]

                 [--keep-cache] [--batch] [--force] [--force-remote]

                 <storage-url>

fsck.s3ql: error: unrecognized arguments: --fg

~# umount.s3ql --fg /tmp/XXX

usage: umount.s3ql [-h] [--log <target>] [--debug-modules <modules>] [--debug] [--quiet] [--version] [--lazy]

                   <mountpoint>

umount.s3ql: error: unrecognized arguments: --fg

~# fsck.s3ql --force s3://XXX

ERROR: Can not check mounted file system.

[Ben Hymers]

Hi Nikolaus,

I'd still really love some help with this if you have time, I'm a bit stuck!

Thanks,

Ben

--
You received this message because you are subscribed to a topic in the Google Groups "s3ql" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/s3ql/CJjXsOo9Ago/unsubscribe.
To unsubscribe from this group and all its topics, send an email to s3ql+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/337c77f8-7b9b-4e73-99f5-22cdaf645b29n%40googlegroups.com.

[Nikolaus Rath]

On Mar 12 2021, Ben Hymers <b.hy...@gmail.com> wrote:
>> I'm sorry, which part should I try running with `--fg`?

mount.s3ql, but this is meant to help with the...

>>> > line 536, in _queue_upload
>>> > raise NoWorkerThreads('no upload threads')
>>> > s3ql.block_cache.NoWorkerThreads: no upload threads
>>>
>>> Does it help if you run with `--fg`? If so, you justs need to wait for
>>> the
>>> next release for a proper fix.

...problem, which has since been fixed in a release. So you might just
as well upgrade to the newest release instead.

To clean up the current state, use `umount` + `fsck.s3ql`.

[Ben Hymers]

I'm up and running! Thank you Nikolaus! Upgrading to 3.7.1 fixed everything and I have now run fsck and have a successful mount.

Thanks again,

Ben

--
You received this message because you are subscribed to a topic in the Google Groups "s3ql" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/s3ql/CJjXsOo9Ago/unsubscribe.
To unsubscribe from this group and all its topics, send an email to s3ql+uns...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/87wnuc5p43.fsf%40vostro.rath.org.