Cluster Filesystem on FreeBSD
Erik Scholtz, ArgonSoft GmbH
I did some research the last two weeks on how to build a cluster
filesystem on FreeBSD.
Since neither GFS ( http://sources.redhat.com/cluster/gfs/ ) nor OCFS2 (
http://oss.oracle.com/projects/ocfs2/ ) are ported to FreeBSD, I took a
look at the GlusterFS-project ( http://www.gluster.org/ ), which also
does not compile on FreeBSD 7.x, as well as PVFS ( http://www.pvfs.org/ ).
My solution at the moment is, to rsync all filesystems once a minute,
which is rather to rare. So I tried to get a hook with KQueue to rsync
the filesystems on data-change. Unfortunatly I could not find a working
solution (had a try with IO::KQueue using perl).
NFS isn't really a solution, since it is slow (even on 1GBit) and is a
single point of failure: Clustering the NFS server with machines in
stand-by, that do a fail-over on a crash of the master-NFS is unstable
and makes a lot more costs to the project.
I'm with FreeBSD since the 2.1 release and I really love this system for
it's performance and stability. But in the last year I came very often
to a point, where I had no solution on FreeBSD and several solutions for
Linux (like SAS-support, ...).
How do you guys solve this problem (of a shared filesystem with
rw-option)? Any hints are welcome, since I'm getting very frustrated at
the moment.
Greetings,
Erik
--
My blog: http://blog.elitecoderz.net
Luke Marsden
The best way I've found to monitor for filesystem activity on FreeBSD is
with the audit_control kernel option. Recompile your kernel if necessary
and pop this into /etc/security/audit_control (owned by root:wheel and
0600 perms):
dir:/var/audit
flags:fc,fd,fw
minfree:20
naflags:lo
policy:cnt
filesz:0
Then this Python script (easily ported to Perl if you wish) will do the
job of printing a path to stdout whenever a file is modified (and also
pruning the praudit logfiles in /var/audit which aren't desirable for
this application - we couldn't find a nicer way to do this):
import popen2, sys, os
pipe = popen2.Popen4("/usr/sbin/praudit /dev/auditpipe")
count = 0
while True:
line = pipe.fromchild.readline()
if line=='':
break # so that we die when praudit does
if 'header' in line:
header=line
if '/path-to-monitor/' not in line: # useful to avoid a flood
continue
if 'path' in line:
count += 1
if count % 1000 == 0: # clear out audit files to save disk space
os.system('for X in /var/audit/*; do echo "" > "$X"; done')
if 'fcntl' not in header:
if line.split(',')[0].strip() == 'path':
print "MODIFIED: "+line.split(',')[1].strip()
sys.stdout.flush()
Hope you find this useful, it's one small part of a FreeBSD-based web
cluster product I'm working on, select parts of which we plan to release
under the BSD license in the next few months in order to hopefully
attract a developer community of FreeBSD cluster hackers.
If you make any improvements to the script I'd appreciate getting them,
just send them in to lu...@hybrid-logic.co.uk
Regarding filesystem replication (and without giving too much away) I
recommend the ZFS filesystem with its fast, atomic snapshots and zfs
send / zfs recv to do the job of rsync much more efficiently and at the
block level.
You won't find multi-master read/write with this solution on its own
though. It is more useful to implement "streaming backups". At some
point I'm planning to combine this approach with NFS failover in order
to make every filesystem writable on every machine at all times,
although this is not a requirement for our application right now.
Stay tuned to www.hybrid-logic.co.uk for a timeline for open source
availability of more of our cluster technology. (That part of the site
isn't ready just yet, but should be in a few weeks.)
Best Regards,
Luke Marsden
Hybrid Logic Ltd.
Leinier Cruz Salfran
<esch...@argonsoft.de> wrote:
> Hi,
>
hi
> I did some research the last two weeks on how to build a cluster filesystem
> on FreeBSD.
>
> My solution at the moment is, to rsync all filesystems once a minute, which
> is rather to rare. So I tried to get a hook with KQueue to rsync the
> filesystems on data-change. Unfortunatly I could not find a working solution
> (had a try with IO::KQueue using perl).
>
i use rsync to make partial data backup .. ie: /etc, /usr/local/etc,
/usr/home, /var/logs ...
> How do you guys solve this problem (of a shared filesystem with rw-option)?
> Any hints are welcome, since I'm getting very frustrated at the moment.
>
there is a project named 'hast'[1] for a clustered filesystem .. it's
being developed by pawel .. the project has some completed milestones,
so you can get it from fbsd src svn tree .. hast can do clustered
filesystem right now but it's not complete, so there is no stable yet
other way is gmirror[2] + ggated .. with that you can get a raid1 over
net solution .. but i think it's not prepared to be used as
master-master soluction
> Greetings,
> Erik
>
okey, same
[1] http://www.freebsdfoundation.org/project%20announcements.shtml#Pawel
[2] http://www.freebsddiary.org/gmirror.php
Doug Poland
On Mon, March 1, 2010 12:11, Leinier Cruz Salfran wrote:
> On Sun, Feb 28, 2010 at 6:09 AM, Erik Scholtz, ArgonSoft GmbH
> <esch...@argonsoft.de> wrote:
>
>> I did some research the last two weeks on how to build a cluster
>> filesystem on FreeBSD.
>>
>> My solution at the moment is, to rsync all filesystems once a
>> minute, which is rather to rare. So I tried to get a hook with
>> KQueue to rsync the filesystems on data-change. Unfortunatly I could
>> not find a working solution (had a try with IO::KQueue using perl).
>>
>
> i use rsync to make partial data backup .. ie: /etc, /usr/local/etc,
> /usr/home, /var/logs ...
>
>> How do you guys solve this problem (of a shared filesystem with
>> rw-option)?
>> Any hints are welcome, since I'm getting very frustrated at the
>> moment.
>>
>
> there is a project named 'hast'[1] for a clustered filesystem .. it's
> being developed by pawel .. the project has some completed milestones,
> so you can get it from fbsd src svn tree .. hast can do clustered
> filesystem right now but it's not complete, so there is no stable yet
>
> other way is gmirror[2] + ggated .. with that you can get a raid1 over
> net solution .. but i think it's not prepared to be used as
> master-master soluction
>
one "side" of the storage is available for writes at a point in time.
Filesystems like OCFS2 and GFS allow multiple, simultaneous read-write
access to block devices.
Given there is not true cluster filesystem available for FreeBSD at
this time, I wonder aloud why so many people are so quick to dismiss
NFS? NFS provides "most" of features of a cluster filesystem today.
If one were to choose NFS for shared storage, one could use tools
available today to make NFS highly available (hast, gmirror+ggated).
--
Regards,
Doug
Erik Scholtz, ArgonSoft GmbH
have you ever tried to put a NFS-volume into the working-path of an
apache-server? I did, and I could messure a significant performance-loss
of the webserver by 1sec per page-load.
The NFS was mounted on a 1GBit dedicated connection between the
webserver and the nfs-server; I tried several TCP-options and
MTU-Settings without any important change on the performance-loss.
Compared to a dedicated mounted ISCSI-Volume this 1sec loss is a lot! I
think NFS is great for changing big amounts of data. But for short
read/write-access NFS does not seem to be the first choice.
Greetings,
Erik
--
My blog: http://blog.elitecoderz.net