does s3ql work with wasabi?

[Marcin Ciesielski]

Hi

I am using version 2.28 with wasabisys as s3 backend.
few times a day s3ql will report such error (see below)
does it mean that there is a problem with wasabisys compatibility?

---

ERROR: Uncaught top-level exception:
Traceback (most recent call last):
File "/home/kisiel/s3ql-2.28/src/s3ql/mount.py", line 64, in run_with_except_hook
run_old(*args, **kw)
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/home/kisiel/s3ql-2.28/src/s3ql/block_cache.py", line 409, in _upload_loop
self._do_upload(*tmp)
File "/home/kisiel/s3ql-2.28/src/s3ql/block_cache.py", line 436, in _do_upload
% obj_id).get_obj_size()
File "/home/kisiel/s3ql-2.28/src/s3ql/backends/common.py", line 108, in wrapped
return method(*a, **kw)
File "/home/kisiel/s3ql-2.28/src/s3ql/backends/common.py", line 340, in perform_write
return fn(fh)
File "/home/kisiel/s3ql-2.28/src/s3ql/backends/comprenc.py", line 371, in exit
self.close()
File "/home/kisiel/s3ql-2.28/src/s3ql/backends/comprenc.py", line 365, in close
self.fh.close()
File "/home/kisiel/s3ql-2.28/src/s3ql/backends/comprenc.py", line 530, in close
self.fh.close()
File "/home/kisiel/s3ql-2.28/src/s3ql/backends/common.py", line 108, in wrapped
return method(*a, **kw)
File "/home/kisiel/s3ql-2.28/src/s3ql/backends/s3c.py", line 948, in close
headers=self.headers, body=self.fh)
File "/home/kisiel/s3ql-2.28/src/s3ql/backends/s3c.py", line 550, in _do_request
self._parse_error_response(resp)
File "/home/kisiel/s3ql-2.28/src/s3ql/backends/s3c.py", line 572, in _parse_error_response
raise HTTPError(resp.status, resp.reason, resp.headers)
s3ql.backends.s3c.HTTPError: 400 Bad Request: Network error.

[Nikolaus Rath]

On Jun 21 2018, Marcin Ciesielski <ciesiels...@gmail.com> wrote:
> Hi
>
> I am using version 2.28 with wasabisys as s3 backend.
> few times a day s3ql will report such error (see below)
> does it mean that there is a problem with wasabisys compatibility?

> ------------------------------
[...]

>
> File "/home/kisiel/s3ql-2.28/src/s3ql/backends/s3c.py", line 572, in
> _parse_error_response
> raise HTTPError(resp.status, resp.reason, resp.headers)
> s3ql.backends.s3c.HTTPError: 400 Bad Request: Network error.

A 400 error means that the client (in this case S3QL) sent a malformed
request. However, if this happens only once in a while then this is very
unlikely. Also, the message "Network Error" doesn't seem to indicate a
client problem.

More likely, there is some problem with Wasabisys or a proxy between you
and them. In either case, it's a bug in the program that sends the error
code. If there is a a temporary problem the server should report
something like 408 (cf https://en.wikipedia.org/wiki/List_of_HTTP_status_codes).


Best,
-Nikolaus

--
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[Marcin Ciesielski]

It happens every few hours a day. I guess maybe every 3-6 hours. I am transferring a lot of data so it might be the factor.

I was not able to connect this to any specific activity.

After that, I get the transport disconnected message and have to reset the system.

I don't use any proxy so this cannot be the reason, hence my question if s3ql does not work with wasabi.

I assume this is not tested. Please correct me if I am wrong.

Would there be a way for me to provide you more detailed logs to see if that's not error handling on the s3ql side?

[Nikolaus Rath]

Hi Marcin,

A: Because it confuses the reader.
Q: Why?
A: No.
Q: Should I write my response above the quoted reply?

..so please quote properly, as I'm doing in the rest of this mail:

On Jun 22 2018, Marcin Ciesielski <ciesiels...@gmail.com> wrote:
> On Friday, 22 June 2018 09:02:29 UTC+1, Nikolaus Rath wrote:
>> On Jun 21 2018, Marcin Ciesielski <ciesiels...@gmail.com> wrote:
>> > Hi
>> >
>> > I am using version 2.28 with wasabisys as s3 backend.
>> > few times a day s3ql will report such error (see below)
>> > does it mean that there is a problem with wasabisys compatibility?
>> > ------------------------------
>> [...]
>> >
>> > File "/home/kisiel/s3ql-2.28/src/s3ql/backends/s3c.py", line 572, in
>> > _parse_error_response
>> > raise HTTPError(resp.status, resp.reason, resp.headers)
>> > s3ql.backends.s3c.HTTPError: 400 Bad Request: Network error.
>>
>> A 400 error means that the client (in this case S3QL) sent a malformed
>> request. However, if this happens only once in a while then this is very
>> unlikely. Also, the message "Network Error" doesn't seem to indicate a
>> client problem.
>>
>> More likely, there is some problem with Wasabisys or a proxy between you
>> and them. In either case, it's a bug in the program that sends the error
>> code. If there is a a temporary problem the server should report
>> something like 408 (cf
>> https://en.wikipedia.org/wiki/List_of_HTTP_status_codes).
>

> It happens every few hours a day. I guess maybe every 3-6 hours. I am
> transferring a lot of data so it might be the factor.
> I was not able to connect this to any specific activity.
>
> After that, I get the transport disconnected message and have to reset the
> system.
>
> I don't use any proxy so this cannot be the reason, hence my question if
> s3ql does not work with wasabi.

Just because you're not aware of it doesn't mean you don't use one. Your
ISP may silently pass your requests through its proxy, or Wasabi itself
may be using one or more reverse proxies without you being able to do
anything about it :-).

> I assume this is not tested. Please correct me if I am wrong.

I have never heard of Wasabi nor did I test S3QL with it. I don't know
if anyone else has. That's all I can say.

> Would there be a way for me to provide you more detailed logs to see if
> that's not error handling on the s3ql side?

You could capture the full server response with a Wireshark to see if
there's more useful information in the body. But most likely you'll have
to contact Wasabi support.