Use of NBD?

[Nikolaus Rath]

Hi,

I am currently looking at options to make backups to the cloud. I have so far been a happy S3QL user (as a matter of fact, I'm the S3QL author) but I am getting increasingly annoyed by the fundamental limitation of having to upload an entire SQLite database on every umount.

One option would of course be to finally fix this deficiency, but I doubt that I have enough time available - and it may even be fundamentally impossible given the design constraints of S3QL. Therefore my renewed interest in other solutions :-).

What I really like about S3Backer is that I can use it with an arbitrary filesystem. What I don't like so much is that it uses FUSE and loopback mounts - this feels like a rather weird contortion.

I was wondering - have you ever considered to replace FUSE + losetup with NBD (https://nbd.sourceforge.io/)? It seems to me that having S3Backer provide an NBD server would be a much more natural solution with lower overhead.

Best,

-Nikolaus

[Archie Cobbs]

Hi Nikolaus,

On Monday, June 7, 2021 at 3:49:15 AM UTC-5 Nikolaus Rath wrote:

I was wondering - have you ever considered to replace FUSE + losetup with NBD (https://nbd.sourceforge.io/)? It seems to me that having S3Backer provide an NBD server would be a much more natural solution with lower overhead.

Great question. When s3backer was originally written I was unaware of NBD, so didn't consider it.

Some random thoughts...

It would likely be more efficient that FUSE, but the difference is probably lost in the noise when compared to the slowness of the network I/O involved.

FUSE is a bit more portable, e.g., it runs (or used to run) on Mac OS X.

NBD would give clearer semantics about when data must be committed (flushed) and ordering constraints in general.

-Archie

 

[Nikolaus Rath]

Do you think there would be fundamental obstacles to make s3backer support NBD (in addition to FUSE)? It seems to me that it should be relatively straightforward to map incoming NBD requests to the equivalent FUSE read/write requests...

Best,

-Nikolaus

--

GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

[Archie Cobbs]

Hi Nikolaus,

On Tuesday, June 8, 2021 at 4:35:46 AM UTC-5 Nikolaus Rath wrote:

Do you think there would be fundamental obstacles to make s3backer support NBD (in addition to FUSE)? It seems to me that it should be relatively straightforward to map incoming NBD requests to the equivalent FUSE read/write requests...

No it should be "straightforward". Looks like there are some libraries out there that could provide some of the NBD server-side functionality.

FYI I also ran across this project: s3bkldev .. don't know anything about it.

-Archie