backblaze b2 backend

[Paul Harris]

Hi all,

I'm confused why more people don't use backblaze's b2 ?

It seems cheaper than AWS"s S3 ...

I see the b2 backend isn't being tested.

How are the others being tested?

You can get a b2 account with 10GB for free, that can surely be used for testing?

I see there are automated tests for PRs run for s3ql.

There was mention of a "server for mock tests needed", I'm wondering what exactly that would entail.

This project looks really sweet.

Definitely runs better than s3fs in my situation.

(I gave that a try but I had problems with bind-mount and writing from an LXC).

Anyway, I made a PR that fixes a problem I had with b2.

https://github.com/s3ql/s3ql/pull/337

Question: how can I set up s3c backend for b2?
I thought I'd run a test for that.

Thanks

Paul

[Georg Pfuetzenreuter]

On 11/28/23 15:24, Paul Harris wrote:
> Hi all,
>
> I'm confused why more people don't use backblaze's b2 ?
> It seems cheaper than AWS"s S3 ...

I can also recommend checking Wasabi's S3 offering.

>
> I see the b2 backend isn't being tested.
> How are the others being tested?
> You can get a b2 account with 10GB for free, that can surely be used for
> testing?
>
> I see there are automated tests for PRs run for s3ql.
> There was mention of a "server for mock tests needed", I'm wondering
> what exactly that would entail.
>
> This project looks really sweet.
> Definitely runs better than s3fs in my situation.
> (I gave that a try but I had problems with bind-mount and writing from
> an LXC).
>
> Anyway, I made a PR that fixes a problem I had with b2.
> https://github.com/s3ql/s3ql/pull/337
>
> Question: how can I set up s3c backend for b2?
> I thought I'd run a test for that.
>
> Thanks
> Paul
>

> --
> You received this message because you are subscribed to the Google
> Groups "s3ql" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to s3ql+uns...@googlegroups.com
> <mailto:s3ql+uns...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/s3ql/104d4bc3-ba88-4f52-b0da-7f44196284c8n%40googlegroups.com <https://groups.google.com/d/msgid/s3ql/104d4bc3-ba88-4f52-b0da-7f44196284c8n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Paul Harris]

On Tue, 28 Nov 2023 at 23:05, 'Georg Pfuetzenreuter' via s3ql <s3...@googlegroups.com> wrote:

On 11/28/23 15:24, Paul Harris wrote:
> Hi all,
>
> I'm confused why more people don't use backblaze's b2 ?
> It seems cheaper than AWS"s S3 ...

I can also recommend checking Wasabi's S3 offering.

I looked at that, it looks really good too.

I just didn't like the uncertain egress/download caps.

Would prefer to pay a known potential amount rather than annoy them and have them disable my account, however unlikely that could be.

[Henry Wertz]

I mean, not automated but I make HEAVY use of the local backend so I can guarantee you that one is pretty solid.  (Including forgetting to unmount it a few times when I shut the computer off, a couple power outages (or letting battery run dead in the case of a laptop), I have one on a portable USB drive where I had the USB cable start popping out now and then for a while (although I got a "tighter" cable and it's fine now), and even one disk where I started getting bad sectors.  I can assure from both looking at the database design, and personal experience with flakey hardware, the design is VERY failsafe and the fsck.s3ql is VERY robust about recovering from issues.

I found it to be effectively just as safe as straight ext4 -- I mean, the drive that spontaneously got bad sectors, I lost one or two files that had s3ql blocks stored on bad sectors (just as I would with ext4); if I was writing to a s3ql when the power went out, maybe I'd lose the last couple seconds of stuff I'd written (just as would happen with ext4.)   A few times I had things drop at the wrong moment and had to use the "sqlite3 broken.db ".recover" | sqlite3 new.db" method to recover the database (in local cache, it didn't back up a bad database to storage) before I could run fsck.s3ql, but still lost nothing but the last couple seconds of data.

In addition, it regularly backs up the database so even if the ".recover" process had failed I would have had a good copy of the database from earlier in the day rather than a total loss.

Have a good day!

--Henry

[Daniel Jagszent]

Hello Paul,

[...] see there are automated tests for PRs run for s3ql.

There was mention of a "server for mock tests needed", I'm wondering what exactly that would entail.

[...]

We have mock servers for the s3c and swift backend defined here:
https://github.com/s3ql/s3ql/blob/master/tests/mock_server.py

Only these get executed in the normal test suite. The Backblaze B2 backend would very much benefit from such a mock server but nobody came around to write one.

https://www.rath.org/s3ql-docs/installation.html#running-tests-requiring-remote-servers explains how you could run the test suite against the real Backblaze servers. The GitHub Actions do not run any such real-live tests.

[Daniel Jagszent]

Hello Paul,

[...] Question: how can I set up s3c backend for b2?

I thought I'd run a test for that.

https://www.rath.org/s3ql-docs/backends.html#s3-compatible explains the connection string syntax.

The hostname to use (called endpoint) can be found here:
https://www.backblaze.com/docs/cloud-storage-call-the-s3-compatible-api

[Paul Harris]

That's really great to hear!

re EXT4 losing data, you shouldn't lose any data if you use ordered-data log mode.

I have had a couple of failures on ext3/4 where the metadata was logged but the data wasn't, so the file was "completed" but data inside was zeros after recovery.

Just thought I'd mention that.

--
You received this message because you are subscribed to a topic in the Google Groups "s3ql" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/s3ql/403-rpd9TV4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to s3ql+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/30044eb5-1952-4800-9f66-e8ab60a96193n%40googlegroups.com.

[Nikolaus Rath]

Hi,

What this means is that Backblaze isn't as fully S3 compatible as it perhaps advertises. In seems what's missing in particular is support for "V2 Signature auth".

Best,

-Nikolaus

On Wed, 29 Nov 2023, at 02:56, Paul Harris wrote:

Doesn't work for me.

In authinfo2, I have

[s3c-test]

backend-login: 00etcetc

backend-password: Ketcetc

test-fs: s3c://s3.us-east-005.backblazeb2.com/bucketnameEtcEtc

I run

python3 -m pytest tests/t1_backends.py::test_readinto_write_fh

and it says HTTPError: 400 (The V2 signature auth is not supported)

--

You received this message because you are subscribed to the Google Groups "s3ql" group.

To unsubscribe from this group and stop receiving emails from it, send an email to s3ql+uns...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/f4fd43f3-195a-44e0-b8a4-56620aaa6d71n%40googlegroups.com.

[Nikolaus Rath]

Hi,

The s3 backend uses v4 auth.

The s3c backend uses the original S3 specification back from whenever it was first published.

Best,

-Nikolaus

On Fri, 1 Dec 2023, at 14:28, Paul Harris wrote:

Sorry i think i hit the wrong button to reply...

I thought s3 only supports V4 auth now?  I thought I read that somewhere.

Or was it b2 only supports V4.

Why don't we use V4 auth?

To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/e5d6f55c-9d27-4de4-92c1-fef3ec8fe7c7n%40googlegroups.com.