Currently recommended way of having databases in Kubernetes?

[Arve Knudsen]

Hey guys

So I've finally got a Container Linux/Kubernetes cluster up and running on DigitalOcean, via my port of the Tectonic Installer to this platform. Now it's time to do something practical with it.

Can someone please let me know of the currently recommended way of running a database cluster under Kubernetes, in this case RethinkDB? I recall having stateful applications, that need to write to disk, was always a thorny issue with Kubernetes, I actually received the firm recommendation to have my DB cluster outside of it. So I originally set up a 2-node RethinkDB cluster in CoreOS, with fleet as the orchestration tool, which worked quite well. But, AFAIK, Kubernetes has now replaced fleet as CoreOS/Container Linux' orchestration system.

Therefore, I need to be brought up to speed as to how it's currently done with Kube :) Any advice in this regard would be greatly appreciated.

Thanks!

Arve

[Rob Szumski]

The latest versions of Kubernetes have a few concepts that help you run stateful apps on Kubernetes:

PersistentVolumes/Volume Claims: provide a method for apps to claim storage provided by the cluster

StatefulSets: an object designed to run stateful apps, which really comes down to being more controlled about how they are started and scaled

Cloud Provider: integration with the cloud provider to auto-create LBs, volumes, etc. DigitalOcean lacks on of these in Kubernetes currently. On AWS for example, the cloud provider will automatically fulfill persistent volume requests with an EBS volume.

If you look at this example, you can see how all of these come together to run RethinkDB. Without the stable storage provided by the cloud, you can use either host storage, which means your pods will need to be pinned to specific hosts via a label query. I assume this is what you used to do with fleet, as fleet doesn’t have any storage semantics other than mounting host storage. You can also specify a volume to your pods that is a RAM disk and use the clustering in RethinkDB to replicate, but not have it be durable on disk.

tl;dr: having a DO cloud provider would help you out a lot, and it’s being discussed to add one to Kubernetes, in which time we would add it to Tectonic.

-  Rob

[Arve Knudsen]

Thanks Rob!

I was just installing Rook in my cluster, do you think this would be pointless as the storage layer? Should I go with PersistentVolumes/Volume Claims instead?

I already have the understanding that I should use StatefulSet to control my RethinkDB cluster.

Thanks for providing that example! It provides me with an understanding as to how I can implement StatefulSet with volume claims, in order to do what I want. I guess the question is how to connect volume claims to DigitalOcean volumes, seeing as there's no cloud provider support for it in Kubernetes yet.

Best,

Arve

[Rob Szumski]

I believe the idea of Rook is to give you storage that you can use to fulfill persistent volume claims. I’m not familiar enough with the project to know how, but I would assume it either understands PVC directly, or you have to mount Rook volumes somewhere on the host and then connect those as PVs to be claimed.

As far as DO, the limitation is getting the volume attached to the pod as it moves around, which is how the other cloud providers work. I’m not sure if they support detaching and reattaching after instance boot.

[Arve Knudsen]

Thanks for the tips! 🙂 It's given me a lot to think about