Question: Adapting s3backer to BackBlaze B2 cloud storage?

[Paul Jackson]

I am considering adapting s3backer to BackBlaze's B2 cloud storage: https://www.backblaze.com/b2/cloud-storage.html

So far I have no code, no commitments, and no schedules, and so make no promises.  But the combination of s3backer's "virtual block device" in the cloud, with BackBlaze's B2, perhaps also using F2FS (that I asked about in an earlier question) would seem to be an excellent fit for what I'm trying to do, which is essentially running a web server that supports a discussion forum (at least Apache/MySQL/PHP/vBulletin based, and perhaps also more recent technology based) and that serves up a fair number of 1 Mbyte to 1 Gbyte document and media files (perhaps using a Calibre based frontend), using a smallish local disk, made to look like a larger local store, using s3backer technology.  I look to do this while (1) optimizing my opportunity to gain more cloud computing experience, (2) contributing my coding experience and abilities to worthwhile cloud projects, (3) minimizing critical dependencies on any one dominant cloud vendor's specific technology (i.e. - no critical Google, Amazon or Microsoft technology dependencies), and (4) minimizing ongoing monthly fees.  I am, and have been for the last six years, administering a website that provides such forum and file serving to a few hundred thousand users per month (unique IP's per month), so I have a ready source of users (aka victims) for my work.

If I had to guess, based on my recent past work on other such projects, it will take me two to five months to deliver a stable result (obviously making extensive use of the excellent and mature s3backer code.)  Or, there is considerable risk that nothing comes of this at all.

So -- my questions for the day:

1) Are there any similar or overlapping efforts underway that I should know about?

2) How should I package this?  The trivial option would be to fork the s3backer project, into say "b2backer".  But that would divorce b2backer from the ongoing, low rate, but quite useful, maintenance of s3backer.  Another option would end up with me sending Archie patches to s3backer that added another product (at least a bin/b2backer executable and man/b2backer.1 man page) to the existing s3backer build.  Perhaps Archie, or others, have other options worth considering (or perhaps reasons why I might not want to do this at all <grin>.)

My current back-of-the-napkin plans have me performance testing f2fs, versus a couple of more common file systems, against a B2 backed "virtual block device", once I have this "b2backer" variant working.  Testing these file system options against an S3 backed "virtual block device" is not on the path that leads to where I'm trying to go, so (at present guess) is less likely to ever be done by me.

[Archie Cobbs]

On Sunday, March 5, 2017 at 2:08:20 PM UTC-6, Paul Jackson wrote:

1) Are there any similar or overlapping efforts underway that I should know about?

FWIW I have not heard of an.
 

2) How should I package this?  The trivial option would be to fork the s3backer project, into say "b2backer".  But that would divorce b2backer from the ongoing, low rate, but quite useful, maintenance of s3backer.  Another option would end up with me sending Archie patches to s3backer that added another product (at least a bin/b2backer executable and man/b2backer.1 man page) to the existing s3backer build.  Perhaps Archie, or others, have other options worth considering (or perhaps reasons why I might not want to do this at all <grin>.)

Forking on github seems like the right option... this facilitates future merge needs. Presumably you'd just implement a swap-in replacement for the http_io.c layer.

-Archie

[Paul Jackson]

On Sunday, March 5, 2017 at 3:04:24 PM UTC-6, Archie Cobbs wrote:

Forking on github seems like the right option...

Ok - that makes sense - thanks!