CoreOS + CephRBD

[Mateusz Gelmuda]

I am working on a platform which my company can use to host containerized application for out own purposes.

We use the following solution: PXE server -> PXE boot CoreOS -> Docker -> Ceph cluster in Docker containers.

Everything works great, we have built our own provisioning-service which uses Ignition-files to configure the host. The last step (Mounting Ceph Block Device) is the biggest issue for me.

When I mount it in CentOS7 it's pretty simple, I only need to install ceph-common and everything works like charm, but now I need to be able to mount it inside a Docker container on CoreOS.

What is really the best practice to achieve this? I would really appreciate an example or link to article about it as every guide I come across is simply 3 or 4 years old and the solutions don't work anymore.

[Christian Hüning]

Hi Mateusz,

We have exactly the same setup. We use rook to orchestrate ceph and simply run our kubelet from the hyperkube image. With that you need to configure the volume-dir option for the cephalopods VolumePlugin:

In your systemctl service description for the kubelet have something like this: 

ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volume-plugins

…

ExecStart=/opt/bin/kubelet \

...

--volume-plugin-dir=/var/lib/kubelet/volume-plugins

Then make sure to configure the Rook Operator accordingly. The docs here have a section for ContainerLinux as well: https://rook.io/docs/rook/v1.1/flexvolume.html 

You also might want to use the RestartOperator:

https://rook.io/docs/rook/v1.1/container-linux.html

If you’re not using Rook, I am sorry ;-) 

Best,

Christian

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/coreos-user/0353dddc-f006-4215-8cd1-95b7fec521a5%40googlegroups.com.

[Mateusz Gelmuda]

Hi Christian,

Thank you for your tip and the time you spent on trying to help me.

We are not using Rook + Kubernetes... yet (is it something we are planning to use at the second phase of this project).

I finally solved my issue by mounting the RBD through a container with ceph-common installed on it. I faced a lot of problems while trying to achieve this, because I used our own macvlan adapter instead of --net=host.

To all the people facing connection problems with rbd map in ceph container on CoreOS check if you use --net=host.

If you really need to use macvlan on your Ceph containers you can use the following workaround:

- Create a macvlan interface on your host to be able to communicate with it's containers which are using your macvlan = https://blog.oddbit.com/post/2018-03-12-using-docker-macvlan-networks/ (Check Host access part)

- On this website someone explained why the rbd map command hangs while using macvlan https://github.com/rook/rook/issues/2021

This part should be better documented in Ceph or CoreOS.