Bad md5 download-metadata

[Sergey E]

Hi all!
and sorry for my language.


Everything worked well, suddenly (mount.log)

and now I see

# fsck.s3ql swift://path

Backend does not have anything stored under key 's3ql_metadata'

# s3qladm download-metadata swift://path

The following backups are available:
 No  Name                    Date          
  0  s3ql_metadata_new       2019-05-30 19:35:33
Enter no to download: 0
Downloading and decompressing metadata...
WARNING: Object closed prematurely, can't check MD5, and have to reset connection

No other metadata_back.
What do you tell me to do?

---------------

mount.log

2019-05-30 19:51:11.525 2503:Metadata-Upload-Thread s3ql.mount.run: Dumping metadata...
2019-05-30 19:51:11.526 2503:Metadata-Upload-Thread s3ql.metadata.dump_metadata: ..objects..
2019-05-30 19:51:12.214 2503:Metadata-Upload-Thread s3ql.metadata.dump_metadata: ..blocks..
2019-05-30 19:51:23.891 2503:Metadata-Upload-Thread s3ql.metadata.dump_metadata: ..inodes..
2019-05-30 19:51:26.982 2503:Metadata-Upload-Thread s3ql.metadata.dump_metadata: ..inode_blocks..
2019-05-30 19:51:29.450 2503:Metadata-Upload-Thread s3ql.metadata.dump_metadata: ..symlink_targets..
2019-05-30 19:51:29.450 2503:Metadata-Upload-Thread s3ql.metadata.dump_metadata: ..names..
2019-05-30 19:51:39.147 2503:Metadata-Upload-Thread s3ql.metadata.dump_metadata: ..contents..
2019-05-30 19:51:41.103 2503:Metadata-Upload-Thread s3ql.metadata.dump_metadata: ..ext_attributes..
2019-05-30 19:51:41.452 2503:Metadata-Upload-Thread s3ql.metadata.upload_metadata: Compressing and uploading metadata...
2019-05-30 19:52:30.432 2503:Metadata-Upload-Thread root.excepthook: Uncaught top-level exception:
Traceback (most recent call last):
  File "/usr/lib/s3ql/s3ql/mount.py", line 64, in run_with_except_hook
    run_old(*args, **kw)
  File "/usr/lib/s3ql/s3ql/mount.py", line 645, in run
    upload_metadata(backend, fh, self.param)
  File "/usr/lib/s3ql/s3ql/metadata.py", line 319, in upload_metadata
    metadata=param, is_compressed=True)
  File "/usr/lib/s3ql/s3ql/backends/common.py", line 108, in wrapped
    return method(*a, **kw)
  File "/usr/lib/s3ql/s3ql/backends/common.py", line 340, in perform_write
    return fn(fh)
  File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 716, in __exit__
    self.close()
  File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 710, in close
    self.fh.close()
  File "/usr/lib/s3ql/s3ql/backends/common.py", line 108, in wrapped
    return method(*a, **kw)
  File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 940, in close
    headers=self.headers, body=self.fh)
  File "/usr/lib/s3ql/s3ql/backends/swift.py", line 258, in _do_request
    raise HTTPError(resp.status, resp.reason, resp.headers)
s3ql.backends.s3c.HTTPError: 409 Conflict
2019-05-30 19:53:24.079 2503:MainThread s3ql.mount.unmount: Unmounting file system...
2019-05-31 03:19:38.236 3591:MainThread s3ql.mount.determine_threads: Using 4 upload threads.
2019-05-31 03:19:38.238 3591:MainThread s3ql.mount.main: Autodetected 1048532 file descriptors available for cache entries
2019-05-31 03:19:38.625 3591:MainThread root.excepthook: Uncaught top-level exception:
Traceback (most recent call last):
  File "/usr/lib/s3ql/s3ql/backends/swift.py", line 313, in lookup
    resp = self._do_request('HEAD', '/%s%s' % (self.prefix, key))
  File "/usr/lib/s3ql/s3ql/backends/swift.py", line 256, in _do_request
    raise HTTPError(resp.status, resp.reason, resp.headers)
s3ql.backends.s3c.HTTPError: 404 Not Found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/mount.s3ql", line 11, in <module>
    load_entry_point('s3ql==2.21', 'console_scripts', 'mount.s3ql')()
  File "/usr/lib/s3ql/s3ql/mount.py", line 129, in main
    (param, db) = get_metadata(backend, cachepath)
  File "/usr/lib/s3ql/s3ql/mount.py", line 374, in get_metadata
    param = backend.lookup('s3ql_metadata')
  File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 74, in lookup
    meta_raw = self.backend.lookup(key)
  File "/usr/lib/s3ql/s3ql/backends/common.py", line 108, in wrapped
    return method(*a, **kw)
  File "/usr/lib/s3ql/s3ql/backends/swift.py", line 317, in lookup
    raise NoSuchObject(key)
s3ql.backends.common.NoSuchObject: Backend does not have anything stored under key 's3ql_metadata'

full console

fsck.s3ql swift://auth.selcdn.ru/sea4
Starting fsck of swift://auth.selcdn.ru/sea4/
Using cached metadata.
ERROR: Uncaught top-level exception:
Traceback (most recent call last):
  File "/usr/lib/s3ql/s3ql/backends/swift.py", line 313, in lookup
    resp = self._do_request('HEAD', '/%s%s' % (self.prefix, key))
  File "/usr/lib/s3ql/s3ql/backends/swift.py", line 256, in _do_request
    raise HTTPError(resp.status, resp.reason, resp.headers)
s3ql.backends.s3c.HTTPError: 404 Not Found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/fsck.s3ql", line 11, in <module>
    load_entry_point('s3ql==2.21', 'console_scripts', 'fsck.s3ql')()
  File "/usr/lib/s3ql/s3ql/fsck.py", line 1179, in main
    elif backend.lookup('s3ql_metadata')['seq_no'] != param['seq_no']:
  File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 74, in lookup
    meta_raw = self.backend.lookup(key)
  File "/usr/lib/s3ql/s3ql/backends/common.py", line 108, in wrapped
    return method(*a, **kw)
  File "/usr/lib/s3ql/s3ql/backends/swift.py", line 317, in lookup
    raise NoSuchObject(key)
s3ql.backends.common.NoSuchObject: Backend does not have anything stored under key 's3ql_metadata'


s3qladm download-metadata swift://auth.selcdn.ru/sea4
The following backups are available:
 No  Name                    Date          
  0  s3ql_metadata_new       2019-05-30 19:35:33
Enter no to download: 0
Downloading and decompressing metadata...
WARNING: Object closed prematurely, can't check MD5, and have to reset connection
ERROR: Uncaught top-level exception:
Traceback (most recent call last):
  File "/usr/bin/s3qladm", line 11, in <module>
    load_entry_point('s3ql==2.21', 'console_scripts', 's3qladm')()
  File "/usr/lib/s3ql/s3ql/adm.py", line 101, in main
    return download_metadata_cmd(backend, options.storage_url)
  File "/usr/lib/s3ql/s3ql/adm.py", line 142, in download_metadata_cmd
    download_metadata(backend, cachepath + ".db", name)
  File "/usr/lib/s3ql/s3ql/metadata.py", line 300, in download_metadata
    backend.perform_read(do_read, name)
  File "/usr/lib/s3ql/s3ql/backends/common.py", line 108, in wrapped
    return method(*a, **kw)
  File "/usr/lib/s3ql/s3ql/backends/common.py", line 319, in perform_read
    res = fn(fh)
  File "/usr/lib/s3ql/s3ql/metadata.py", line 297, in do_read
    stream_read_bz2(fh, tmpfh)
  File "/usr/lib/s3ql/s3ql/metadata.py", line 285, in stream_read_bz2
    buf = decompressor.decompress(buf)
OSError: Invalid data stream

[Nikolaus Rath]

On May 30 2019, Sergey E <willia...@gmail.com> wrote:
> Hi all!
> and sorry for my language.
>
>
> Everything worked well, suddenly (mount.log)
>
> and now I see
>
> # fsck.s3ql swift://path
>
> Backend does not have anything stored under key 's3ql_metadata'
>>
>
> # s3qladm download-metadata swift://path
>
> The following backups are available:
>> No Name Date
>> 0 s3ql_metadata_new 2019-05-30 19:35:33
>> Enter no to download: 0
>> Downloading and decompressing metadata...
>> WARNING: Object closed prematurely, can't check MD5, and have to reset
>> connection

It seems that something went wrong when you unmounted the filesystem,
and you're now trying to recover without having access to the local
cache holding the most recent metadata (normally ~/.s3ql). Without this
directory, your data is gone.

That said, for how long did "everything go well"? You have no metadata
backup, and normally these are created daily.



Best,
-Nikolaus

--
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[Sergey E]

Hello!

There is a local cache

 ~/.s3ql
swift:=2F=2Fauth.selcdn.ru=2Fsea4=2F.db (actual date)
swift:=2F=2Fauth.selcdn.ru=2Fsea4=2F.params (2 month ago)

and when it starts up, it sees it, but it swears anyway.

# fsck.s3ql swift://auth.selcdn.ru/sea4

Starting fsck of swift://auth.selcdn.ru/sea4/
Using cached metadata.

...
s3ql.backends.common.NoSuchObject: Backend does not have anything stored under key 's3ql_metadata'


The file itself opens SQLite - without errors
How can I use it to check the data and mount it?
-
Mount the storage in ro - there is not a single backup_0 file
--

>>That said, for how long did "everything go well"? You have no metadata backup, and normally these are created daily.

It is not known how long ago, I got in this form.

[Nikolaus Rath]

On Jun 01 2019, Sergey E <willia...@gmail.com> wrote:
> Hello!
>
> There is a local cache
>
> ~/.s3ql
> swift:=2F=2Fauth.selcdn.ru=2Fsea4=2F.db (actual date)
> swift:=2F=2Fauth.selcdn.ru=2Fsea4=2F.params (2 month ago)
>
> and when it starts up, it sees it, but it swears anyway.
>
> # fsck.s3ql swift://auth.selcdn.ru/sea4
> Starting fsck of swift://auth.selcdn.ru/sea4/

> *Using cached metadata.*

> ...
> s3ql.backends.common.NoSuchObject: Backend does not have anything stored
> under key 's3ql_metadata'
>
>
> The file itself opens SQLite - without errors
> How can I use it to check the data and mount it?

Ah, ok. In that case you should be able to recover. However, before
attempting to do anything, I'd strongly recommend to examine how you got
into that situation. According to mount.log, uploading the metadata
failed with an "409 Conflict" HTTP error. You should examine why the
your storage provider returned this error, and make sure it doesn't
happen again.

*After* that, I would try to recover. The following (temporary) patch
may help fsck.s3ql to run even if there is no metadata object at all:

diff --git a/src/s3ql/fsck.py b/src/s3ql/fsck.py
index cd4bb7f..8316669 100644
--- a/src/s3ql/fsck.py
+++ b/src/s3ql/fsck.py
@@ -1161,9 +1161,9 @@ def main(args=None):
log.warning('File system has not been unmounted cleanly.')
param['needs_fsck'] = True

- elif backend.lookup(meta_obj_name)['seq_no'] != param['seq_no']:
- log.warning('Remote metadata is outdated.')
- param['needs_fsck'] = True
+ #elif backend.lookup(meta_obj_name)['seq_no'] != param['seq_no']:
+ # log.warning('Remote metadata is outdated.')
+ # param['needs_fsck'] = True

else:
param = backend.lookup(meta_obj_name)

[Sergey E]

Thanks! It's worked for me!

The provider said that there were short-term difficulties within the HTTP.409 network was a few seconds. Bad luck.

Why there is no backup_matedata - we will understand. To begin, let's update from 2.21 to the current version one.
The file system is currently being checked.
It seems to be without errors.

воскресенье, 2 июня 2019 г., 16:06:24 UTC+4 пользователь Nikolaus Rath написал:

[Esteban Fonseca]

Hi Nikolaus,

How can I prevent loosing this folder, or how could I recover the backup data if for some reason my system crashes and I cannot recover it ? should I run 

# s3qlctrl upload-meta 

after my backups ? I'm running daily backups with jetbackup on cpanel, on my AWS instances to S3 buckets.

Thanks,

Esteban.

[Nikolaus Rath]

Hi Esteban,

A: Because it confuses the reader.
Q: Why?
A: No.
Q: Should I write my response above the quoted reply?

..so please quote properly, as I'm doing in the rest of this mail:

On Jun 03 2019, Esteban Fonseca <este...@gmail.com> wrote:
>> It seems that something went wrong when you unmounted the filesystem,
>> and you're now trying to recover without having access to the local
>> cache holding the most recent metadata (normally ~/.s3ql). Without this
>> directory, your data is gone.
>

> How can I prevent loosing this folder, or how could I recover the backup
> data if for some reason my system crashes and I cannot recover it ? should
> I run
>
> # s3qlctrl upload-meta
>
> after my backups ? I'm running daily backups with jetbackup on cpanel, on
> my AWS instances to S3 buckets.

Yes, if you keep the filesystem permanently mounted then running
s3qlctrl after a backup would be a good idea. If you unmount, this
happens automatically. Note also that mount.s3ql automatically does this
periodically, by default every 24h.

Best,