Best way to mount a distributed filesystem into a container

[Anand Patil]

Hi everyone,

I need to make a distributed filesystem, such as Gluster or Ceph, available to a container. As far as I can tell, I have at least three options:

1. Run the container in privileged mode and mount the DFS in the container itself. I'd like to avoid running privileged containers when possible.

2. Export the DFS using NFS, mount it to the host, and bind-mount it into the container.

3. Install the Ceph or Gluster client to the host in nonstandard locations, and mount the DFS directly to the host.

4. I don't know if this is possible, but: install the Ceph or Gluster client to a container, and somehow use that to mount the DFS directly to the host.

Is 4 possible, and does anyone have any thoughts about the best practice on CoreOS?

Thanks, 

Anand

[Anand Patil]

This blog post: http://s3hh.wordpress.com/2011/09/22/sharing-mounts-with-a-container/ shows how to make 4 possible, and it does work on my Ubuntu build box, but does not work on CoreOS.

Here are the steps:

$ mkdir /media/shared

$ sudo mount --bind /media/shared /media/shared                                                          

$ sudo mount --make-unbindable /media/shared

$ sudo mount --make-shared /media/shared                                                                 

$ sudo mkdir /media/shared/test

$ sudo docker run -i -t --privileged -v /media/shared/test:/shared ubuntu:12.04 /bin/bash

Then, in the container, test whether mounts are shared to the host:

# mkdir /shared/usr

# mount --bind /usr /shared/usr

In the host, /media/shared/usr now contains local, lib, and so forth on Ubuntu but is empty on CoreOS.

Thanks,

Anand

[Jonathan Boulle]

There's an example here of installing GlusterFS inside a Docker container, but it looks like for now it might still require privileged=True.

[Anand Patil]

On Wednesday, April 2, 2014 2:05:07 PM UTC-7, Jonathan Boulle wrote:

There's an example here of installing GlusterFS inside a Docker container, but it looks like for now it might still require privileged=True.

Hi Jonathan,

Thanks for getting back to me.

So, my goal is to somehow get the GlusterFS filesystem mounted into unprivileged container A, which runs my actual app. That means I can't run the glusterfs client in A; but it's probably fine to run it in a separate privileged container B, mount the Gluster folder from B to the host, and then mount it from the host to A using docker run -v.

The hard step there is mounting the Gluster folder from B to the host, Docker doesn't have any support for mounting in that direction AFAIK. The mount --make-shared technique seems like a good solution, if it could be made to work on CoreOS.

Anand

[Humble Devassy Chirammal]

Hi Anand,

You may look at this project https://github.com/humblec/dockit/

using it you can export bricks from host system and it will be mounted inside containers.. It can create a glusterFS  volume inside containers from these exported bricks.

Please let me know if you have any questions.

--Humble

[Рыжий Лис]

Hi Anand, do you solve your problem?

четверг, 3 апреля 2014 г., 0:20:15 UTC+3 пользователь Anand Patil написал:

[Anand Patil]

Hi,

No, unfortunately I stopped trying after reading this:
https://groups.google.com/d/msg/coreos-dev/AhEQiObNo1U/puHjqlQTUZ0J

Please post if you find a way!

Anand

[Рыжий Лис]

I use latest beta CoreOS and i can share folder from container to host. I do like in 2nd post in this thread. But i mount /dev/sda5, not existed folder(2nd step).

Now i have some error with glusterfs, but i think this is cuz i didnt know docker and gluster well. When i try to gluster probe host1(localhost) from docker, i got an error:
"Error through RPC layer, retry again later". Probing host2(from docker on host1) success

среда, 29 октября 2014 г., 22:21:46 UTC+3 пользователь Anand Patil написал:

[Рыжий Лис]

Finally, i got it working. Do you neet Dockerfile or some how-to?

среда, 29 октября 2014 г., 22:21:46 UTC+3 пользователь Anand Patil написал:

[Alexandre Teles]

​​

Please, post here your Dockerfile and a simple tutorial/how to.

This will help other people to achieve and improve your solution. If I can give you a little advice, please use GitHub our BitBucket to post your solution, this way we all can help improve this.

Thank you!

Alexandre Teles

Bacharelando Interdisciplinar em Ciência e Tecnologia - Universidade Federal da Bahia

Gerente de Infraestrutura Auxiliar - Projeto Ekaaty Linux

Tel: (71)9214-8614 (TIM) - (71)8534-4536 (OI)

Skype: kabbalista

Facebook: http://fb.com/alex.kabbal
Website: http://www.operari.info

[Рыжий Лис]

Can you provide link for thread for posting solution? Or do you mean use my own repo for that?

пятница, 31 октября 2014 г., 16:44:40 UTC+3 пользователь Alexandre Teles написал:

[Alexandre Teles]

You can use gist for post the files.

[Рыжий Лис]

Here is How To instruction with using GlusterFS and systemd-nspawn on CoreOS.

1. mkdir /root/debian && cd /root/debian && wget -qO- https://github.com/tianon/docker-brew-debian/blob/dist/wheezy/rootfs.tar.xz?raw=true | tar xpJvf - && wget http://download.gluster.org/pub/gluster/glusterfs/3.5/LATEST/Debian/pubkey.gpg
   
2. mkdir -p /shared
3. mount /dev/sda5 /shared/glusterfs
   

4. systemd-nspawn --bind /shared
   

5. apt-key add /pubkey.gpg && echo deb http://download.gluster.org/pub/gluster/glusterfs/3.5/LATEST/Debian/wheezy/apt wheezy main > /etc/apt/sources.list.d/gluster.list && apt-get update && apt-get install glusterfs-server -y
6. /etc/init.d/glusterfs-server start && gluster peer probe SERVER1 && gluster peer probe SERVER2 && gluster volume create sda5 replica 2 etokki.net:/mnt meshroom.ru:/mnt force && gluster volume start sda5
7. mkdir -p /etc/systemd/system/multi-user.target.wants/
8. touch /etc/systemd/system/multi-user.target.wants/systemd-nspawn\@glusterfs.service && ln -sf /etc/systemd/system/multi-user.target.wants/systemd-nspawn\@glusterfs.service /etc/systemd/system/systemd-nspawn\@glusterfs.service
9. touch /etc/systemd/system/multi-user.target.wants/systemd-nspawn\@glusterfs_mount.service && ln -sf /etc/systemd/system/multi-user.target.wants/systemd-nspawn\@glusterfs_mount.service /etc/systemd/system/systemd-nspawn\@glusterfs_mount.service
10. https://pastee.org/x5rc9 > /etc/systemd/system/systemd-nspawn\@glusterfs.service
    

11. https://pastee.org/t3bwf > /etc/systemd/system/systemd-nspawn\@glusterfs_mount.service
    

12. systemctl daemon-reload
13. systemctl enable systemd-nspawn\@glusterfs.service
14. systemctl enable systemd-nspawn\@glusterfs_mount.service

Sry, but i didnt have github, so i cant post gist.

среда, 29 октября 2014 г., 22:21:46 UTC+3 пользователь Anand Patil написал:

[Eelco]

Hi,

sounds great, I'm going to try this on my new CoreOS cluster. (I'm new to CoreOS and Docker, but have lots of Linux experience. I tryed following the given steps )

I cannot get this to work. These are the steps I have taken:

correcting first line missing "wheezy" at the end:

1. mkdir /root/debian && cd /root/debian && wget -qO- https://github.com/tianon/docker-brew-debian/blob/dist/wheezy/rootfs.tar.xz?raw=true | tar xpJvf - && wget http://download.gluster.org/pub/gluster/glusterfs/3.5/LATEST/Debian/wheezy/pubkey.gpg

changed:

2. mkdir -p /shared/glusterfs

After steps 13. and 14 I tried starting those scripts with . I tried to start will fail. After removing the  symbolic links from 8 and 9 it works. The symbolic links are created automatically.

However, on my first server I could create files in /shared directory. They would NOT show up on the second server. (hence they get written to the filesystem, not gluster)

I tried rebooting. one server after another.

Now I got a mess. gluster did not want to start. I removed the files from steps 10 and 11 and after reboot I could start gluster daemons on both servers (step 6 first part)

If I do the line after ExecStart= from the file (glusterfs.service) in step 10 as root

# /usr/bin/systemd-nspawn --keep-unit --share-system --directory=/root/debian /usr/sbin/glusterd -p /var/run/glusterd.pid

it seems to start allright.

So on to the file from step 11 (glusterfs_mount.service):

# /usr/bin/mount /dev/sdb1 /shared 

# /usr/bin/systemd-nspawn --keep-unit --directory=/root/debian --bind=/shared --share-system /bin/bash -c 'mount -t devtmpfs devtmpfs /dev && mount.glusterfs localhost:coreosdata /shared'

Now I get an error-message:

/sbin/mount.glusterfs: according to mtab, GlusterFS is already mounted on /shared

Any help appreciated!

Eelco

[Eelco van Tilborg]

Hi,

part of the problem is solved now. I reformatted my drive for the gluster filesystem with ext4 on both machines en started afresh. It seems I forgot to reformat one of the drives.

After _lots_ of errorsolving, I managed to get GlusterFS started by following the given steps. 

now right after start is stops with soms DNS resolution problems when mounting in Gluster FS Client. 

trying to figure it out for 2 days now, but I'm a systemd-nspawn noob :-(

[Рыжий Лис]

Yes, i make some mistakes when posting solution, sry (=

If you use systemd-nspawn, you can try to --machine=HOSTNAME. And you can edit /root/debian/etc/hostname /root/debian/etc/hosts

Also, if u use chroot instead of systemd-nspawn, you can check RootDirectory directive for systemd unit file, f.e.:

[Service]

RootDirectory=/root/debian

ExecStart=/binary/inside/chroot

Anyway, i can't use glusterfs as backend for docker, so i forget about my idea. And here is some bug with glsuterfs when i mount glusterfs in directory, mounted with --bind, so i decide to use Ceph RBD, but that not clearly for me. Yet.

четверг, 27 ноября 2014 г., 0:42:04 UTC+3 пользователь Eelco написал:

[Eelco van Tilborg]

too bad...

I had the DNS problem solved. 

after that, I can see the cluser being started. however writing to it, does not result in syncing to the 2nd node in REPLICATED mode.

I Now have the following errors in my log (in the debian container):

[2014-11-27 08:00:52.259196] E [rpcsvc.c:1314:rpcsvc_program_register_portmap] 0-rpc-service: Could not register with portmap

[2014-11-27 08:00:52.259207] E [nfs.c:332:nfs_init_versions] 0-nfs: Program  MOUNT3 registration failed

[2014-11-27 08:00:52.259214] E [nfs.c:1312:init] 0-nfs: Failed to initialize protocols

[2014-11-27 08:00:52.259220] E [xlator.c:403:xlator_init] 0-nfs-server: Initialization of volume 'nfs-server' failed, review your volfile again

[2014-11-27 08:00:52.259227] E [graph.c:307:glusterfs_graph_init] 0-nfs-server: initializing translator failed

[2014-11-27 08:00:52.259233] E [graph.c:502:glusterfs_graph_activate] 0-graph: init failed

I did nothing with nfs myself, possibly this is used for sharing between the coreos system and de systemd-spawn container.

My idea was like:

- have several servers with same architecture, applications in containers (docker would be best, if not, I'll stick with OpenVZ) and shared replicated storage. Separate storage system would be overkill for my applications.

so:

- mount a disk to coreos system (call is /shared)

- have some kind of container (docker, but your way with systemd-spawn is just as good) to act as a Gluster node using this /share as storagebrick

- publish a share (/mnt/gluster) on the container to the coreos system (via a second mount) as the actual interface to the 

- share /mnt/gluster to the docker containers using it as a "normal" file system

The docker containers would not need to know about GlusterFS then? Or would they?

[Chris Mavin]

Sorry to resurrect an old thread but its an important topic.

How did you get on with this?

[Chris Mavin]

Eelco, replied to me directly. 

Good guy. :-)

[ksac...@gmail.com]

I got my glusterfs cluster up but I am hitting a road block while trying to build "docker-volume-glusterfs". Can someone here provide some inputs/help on what is happening

http://stackoverflow.com/questions/42426894/unbale-to-install-glusterfs-driver-on-mac-os-10-11-6-docker-version-1-13-0

I posted the above Q on stackoverflow as well.

[gerritbi...@gmail.com]

Hello,

Maybe a bit lit. But https://registry.hub.docker.com/u/jsafrane/glusterfs-mounter/) works well. Used it on CoreOS.

Greetings Gerrit