possible MogileFS 3.x, somewhat different direction?

[Eric Wong]

Some thinking aloud...

The past few months I've been thinking about a
spray-files-anywhere replicating FS inspired by MogileFS,
but oriented towards individual (desktop/workstation) use.

There'd be a FUSE (or kernel module) component, I'm sure. What
I'm not sure of is whether adapting MogileFS for the rest is
worth it, or if I'd be better off starting from a new codebase
(still Perl5, but likely w/ Inline::C).

I'd like to remove the TCP requirement. MogileFS+(c)mogstored
HTTP can be adapted to have all work done over Unix sockets.
Having TCP listeners (even on localhost) could leak information
on shared systems.

SQLite would be the main supported metadata DB. I've often
found writing optimal SQL for SQLite makes it suboptimal for
other DBs. I guess this applies for any DB engine, really, but
I've barely touched MySQL/Maria/Pg in over a decade.

For MogileFS, I would also like to remove unmaintained
dependencies such as Danga::Socket and Sys::Syscall. Either
merge stripped-down (and faster :P) versions into the codebase
(as I've done so far in another project), or use Inline::C.

Anyways, thoughts?


Disclaimer: it's been a difficult pandemic for me as I'm sure it
has for others, so my brain can be a bit foggy and maybe I'm not
thinking straight.

[okwap]

Great to see any enhancements of MogileFS these days.

I guess it really depends on the workload.

It should work on some workloads, e.g., image post-processing by photographers. We do CRUD on each image file one by one in this scenario.

It might not a good idea on huge small file workloads. SQLite or any centric database will be the bottleneck. 

 

ZFS might be a suitable solution for your requirements? (spray-files-anywhere replicating FS,  towards individual (desktop/workstation) use.)

[Eric Wong]

okwap <peter...@gmail.com> wrote:
> Great to see any enhancements of MogileFS these days.

Fwiw, I'm less likely to want to contribute to this project
due to space overhead in HTML and top-posted mails :P

> I guess it really depends on the workload.
> It should work on some workloads, e.g., image post-processing by
> photographers. We do CRUD on each image file one by one in this scenario.
> It might not a good idea on huge small file workloads. SQLite or any
> centric database will be the bottleneck.

Maybe the SQL for MogileFS won't need to change at all...

There's a bunch of socket optimizations available, I'm using
AF_UNIX SOCK_SEQPACKET more and more in other projects to
simplify processing (FreeBSD has it for many years, now).

The FUSE component would most likely be separate and
independent. I do know Danga::Socket is way slower than it
needs to be :>

> ZFS might be a suitable solution for your requirements?
> (spray-files-anywhere replicating FS, towards individual
> (desktop/workstation) use.)

Nope. AFAIK ZFS doesn't work for mismatched drives, the RAM
requirement is too high, and GPL-incompatibility means it
can never be in the kernel.

Btrfs raid1c3 (what I currently use) isn't bad, but 3 copies is
too much for some cases. There's no class-based replication
policies in btrfs like there are for MogileFS, so everything
on the FS has the same replication count.

It's also difficult to control metadata performance with btrfs.
With most metadata in SQLite (or any other DB), I can keep that
on an SSD.