Mounted S3-Compatible Drive crashing.

[Sonia Taturell]

s3ql is crashing every few hours after average-intensive work. I am using a s3c with Object Storage from Scaleway.com. 

Link to Scaleway S3 API Compatibility

This is the latest mount log crash record:

2019-03-12 13:47:59.546 1417:Thread-12 root.excepthook: Uncaught top-level exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/s3ql-3.0-py3.6-linux-x86_64.egg/s3ql/mount.py", line 58, in run_with_except_hook
    run_old(*args, **kw)
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/dist-packages/s3ql-3.0-py3.6-linux-x86_64.egg/s3ql/block_cache.py", line 768, in _removal_loop_simple
    backend.delete('s3ql_data_%d' % id_)
  File "/usr/local/lib/python3.6/dist-packages/s3ql-3.0-py3.6-linux-x86_64.egg/s3ql/backends/comprenc.py", line 289, in delete
    return self.backend.delete(key, force)
  File "/usr/local/lib/python3.6/dist-packages/s3ql-3.0-py3.6-linux-x86_64.egg/s3ql/backends/common.py", line 108, in wrapped
    return method(*a, **kw)
  File "/usr/local/lib/python3.6/dist-packages/s3ql-3.0-py3.6-linux-x86_64.egg/s3ql/backends/s3c.py", line 218, in delete
    resp = self._do_request('DELETE', '/%s%s' % (self.prefix, key))
  File "/usr/local/lib/python3.6/dist-packages/s3ql-3.0-py3.6-linux-x86_64.egg/s3ql/backends/s3c.py", line 528, in _do_request
    self._parse_error_response(resp)
  File "/usr/local/lib/python3.6/dist-packages/s3ql-3.0-py3.6-linux-x86_64.egg/s3ql/backends/s3c.py", line 562, in _parse_error_response
    raise get_S3Error(tree.findtext('Code'), tree.findtext('Message'), resp.headers)
s3ql.backends.s3c.SignatureDoesNotMatchError: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
2019-03-12 13:48:01.015 1417:MainThread s3ql.mount.unmount: Unmounting file system...
2019-03-12 13:48:01.064 1417:MainThread root.excepthook: Uncaught top-level exception:
Traceback (most recent call last):
  File "/usr/local/bin/mount.s3ql", line 11, in <module>
    load_entry_point('s3ql==3.0', 'console_scripts', 'mount.s3ql')()
  File "/usr/local/lib/python3.6/dist-packages/s3ql-3.0-py3.6-linux-x86_64.egg/s3ql/mount.py", line 214, in main
    raise RuntimeError('Received signal %d, terminating' % (ret,))
RuntimeError: Received signal 15, terminating

My set-up is:

Ubuntu Bionic

s3ql installed with python 3.6 according to official guide on rath.org

s3ql started with systemd, here is the service:

[Unit]
Description=S3QL Scaleway mount -------
After=network.target nss-lookup.target systemd-resolved.service sys-fs-fuse-connections.mount boot-efi.mount local-fs.target ssh.service network-online.target
Conflicts=shutdown.target


[Service]
Type=simple
ExecStartPre=/bin/sleep 10
ExecStartPre=/usr/local/bin/fsck.s3ql --batch --force --keep-cache s3c://s3.nl-ams.scw.cloud:443/*****
ExecStart=/usr/local/bin/mount.s3ql --fg --systemd --keep-cache --cachesize 30000000 --max-cache-entries 10000 --compress zlib-2 --threads 4 --allow-other s3c://s3.nl-ams.scw.cloud:443/****/mnt/s3
ExecStop=/usr/local/bin/umount.s3ql /mnt/s3
ExecStopPost=/bin/sleep 10
LimitNOFILE=1000000
TasksMax=1000000
TimeoutStartSec=120min
TimeoutStopSec=120min


[Install]
WantedBy=multi-user.target

Any ideas ? :(

Thank you in advance!

[Sonia Taturell]

Forgot to mention that s3ql is version 3.0, and I have done the t5_full.py test with test credential for the scaleway object storage and the s3c test result was OK / PASSED.

[Nikolaus Rath]

On Mar 12 2019, Sonia Taturell <timja...@gmail.com> wrote:
> s3ql is crashing every few hours after *average-intensive* work. I am using

> a s3c with Object Storage from Scaleway.com.

> *Link to Scaleway S3 API Compatibility
> <https://www.scaleway.com/docs/s3-object-storage-api/>*
>
>
> *This is the latest mount log crash record:*
>
[...]

> s3ql.backends.s3c.SignatureDoesNotMatchError: SignatureDoesNotMatch: The
> request signature we calculated does not match the signature you provided.
> Check your key and signing method.

I suspect Scaleway has a slighly different way to calculate the request
signature that is related to timestamps (which is why you don't see it
happening every time). I'm afraid there isn't much more I can
say. Amazon S3 sends a number of debugging information together with the
error message, if Scaleway does the same then you could try to look at
that.


Best,
-Nikolaus

--
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[Sonia Taturell]

Ok. NICE... Is that it??

Thanks to s3ql, 5 months later my data is still unacceptable - No way to recover access due to the way s3ql encrypts and protects data from "unauthorised decryption." 

I spent weeks reading manuals and trying everything I can. 

It seems that s3ql can't work out simple storage errors.. (if there are any??). Which trashes the whole data.

I want to say that I switched to rclone and since then never had any problems with it or any of Scaleway S3 buckets. Even when the usage was extremely harsh and intensive. And all errors handled in a production manner, and whats more over if your server crashes, you can still access your encrypted data mounting the same bucked on another VPS because there is no stupid "protection" from unauthorised access. simply provide the same pw and salt pw, and you have the access...

A lot of wasted time and disappointment in one of the fastest mount drivers out there. But useless for production environment...

DONT TRUST S3QL YOUR DATA. or make double backups with other tools.

going further with the story...

[Nikolaus Rath]

Hi Sonia,

A: Because it confuses the reader.
Q: Why?
A: No.
Q: Should I write my response above the quoted reply?

..so please quote properly, as I'm doing in the rest of this mail:

On Jul 24 2019, Sonia Taturell <timja...@gmail.com> wrote:
> On Tuesday, March 12, 2019 at 7:58:55 PM UTC+1, Nikolaus Rath wrote:
>>

>> On Mar 12 2019, Sonia Taturell <timja...@gmail.com <javascript:>> wrote:
>> > s3ql is crashing every few hours after *average-intensive* work. I am using
>> > a s3c with Object Storage from Scaleway.com.
>> > *Link to Scaleway S3 API Compatibility
>> > <https://www.scaleway.com/docs/s3-object-storage-api/>*
>> >
>> > *This is the latest mount log crash record:*
>> >
>> [...]
>> > s3ql.backends.s3c.SignatureDoesNotMatchError: SignatureDoesNotMatch: The
>> > request signature we calculated does not match the signature you
>> provided.
>> > Check your key and signing method.
>>
>> I suspect Scaleway has a slighly different way to calculate the request
>> signature that is related to timestamps (which is why you don't see it
>> happening every time). I'm afraid there isn't much more I can
>> say. Amazon S3 sends a number of debugging information together with the
>> error message, if Scaleway does the same then you could try to look at
>> that.
>>

> Ok. NICE... Is that it??
>
> Thanks to s3ql, 5 months later my data is still unacceptable - No way to
> recover access due to the way s3ql encrypts and protects data from
> "unauthorised decryption."
> I spent weeks reading manuals and trying everything I can.

I am sorry that you are having trouble with S3QL. Your original message
only mentioned "crashing every few hours" - with no mention of your data
being inaccessible. Has something changed?

I also spend time looking into this and gave you a suggestion how to
further debug the problem. This was time that I could have spent earning
money, doing something with my family, or a myriad of other things
instead of trying to help a stranger on the internet. The fact that you
didn't even acknowledge the response, and instead followup with an
accusatory complaint months later is disappointing.