Best way to use fuse based filesystem in multiple containers

[Eric de Ruiter]

Hi,

For our new setup we are looking into using CoreOS for our web-cluster. In our current setup we are using a fuse based clustering file system, which we would like to use in our new setup to. The most efficient setup would be to have one mount per host and expose that mount to the containers which need them (the webserver and php containers).
For now we have this working by copying the required libraries and the fuse file system client into CoreOS. However this seems to go against the very nature of CoreOS (everything should be in a container), what would be the recommended way of doing this?
We already figured out that running a fuse based filesystem inside a container is possible, however it seems is not possible to export the fuse-mount to the host or another container.

Kind regards,

  Eric

[anton....@coreos.com]

Hi Eric,

It seems you are talking about GlusterFS. I'm not sure if it will work with fuse but CoreOS has built-in fuse kernel module. The only thing you have to do is to build fusermount tool.

You can download latest fuse source code and compile with in docker container (you don't have to compile kernel module so you don't have to install kernel package). By default binary will be compiled with static libraries.

Download fuse:

curl -O http://netcologne.dl.sourceforge.net/project/fuse/fuse-2.X/2.9.4/fuse-2.9.4.tar.gz

Extract:

tar -xf fuse-2.9.4.tar.gz

Run configure and make:

./configure --prefix=/opt

make

Copy your ./util/fusermount binary into coreos:/opt/bin/

But I would recommend you mount your FS using NFS protocol.

Regards,

Anton

[Eric de Ruiter]

Hi Anton,

Actually we have developed our own specialized clustered fuse filesystem (it replicates all data to all nodes, does all reads locally and replicates all writes to all nodes using ZMQ). So using NFS is not an option (as the whole idea of our filesystem was to eliminate spof's and avoid the need for a san).
So would building a static version / bundling all required libraries and storing it in /opt be the recommended way of doing this? Or is there a way of doing this inside a container?

Kind regards,

  Eric


Op donderdag 16 juli 2015 11:16:49 UTC+2 schreef anton....@coreos.com:

[Seán C. McCord]

Yes, you can use FUSE from within a container.  When i used it last, it required `--privileged`, but at that time, there were rumblings of getting rid of that requirement.  I moved on to mounting CephFS to the host, so I don't use FUSE anymore.

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Seán C McCord

CyCore Systems, Inc

888-240-0308

[anton....@coreos.com]

Hi Eric,

I would also recommend you to copy mount.fuse tool from ./util directory to /opt/bin/.

/opt is recommended path for storing third-party binaries which require running on host machine.

You can also use docker if you would like to mount fuse fs in each container, but I assume this way will require running container with --privileged option.

Regards,

Anton

[Eric de Ruiter]

Hi Seán,

Yes inside a single container works with --privileged, however as this filesystem would be needed by more containers, I would need to export the file system to other containers or to the host, and as I understand it, this is not possible (however I would like to be proven wrong on this :P )

Kind regards,

  Eric

Op donderdag 16 juli 2015 14:39:19 UTC+2 schreef Seán McCord:

[Eric de Ruiter]

Hi Anton,

Thanks for your advice, I will go forward with placing the binaries in /opt.

Kind regards,
 
  Eric

Op donderdag 16 juli 2015 14:43:50 UTC+2 schreef anton....@coreos.com:

[Vish]

Hi,

(on CoreOS) how do mount the gluster volume?

This does not work; mount -t glusterfs <my gluster volume location> test-mount-point

mount does nor recognize the glusterfs as known file system in coreos.

Is  there a way I can mount (on host) gluster volumes?

Thanks.

[anton....@coreos.com]

Hi Vish,

You can try to build glusterfs module using this repo: https://github.com/ianblenke/docker-coreos-build-kernel-module , then load it running "sudo insmod /home/core/module.ko".

I didn't try it myself, but I suppose it should help. Otherwise you have to build your own kernel with modules following this doc: https://coreos.com/os/docs/latest/sdk-modifying-coreos.html

Let me know if you have any further questions.

Regards,

Anton

[Vish]

Hi Anton,

[repo: https://github.com/ianblenke/docker-coreos-build-kernel-module ]

I could not figure out what should be done after building this docker image. I mean the steps to create a glusterfs module for coreos.

I am not going for coreos sdk due to other reasons.

Thanks.