Flatcar, Kubernetes, and the deprecation of Docker as runtime

[Justin Hopper]

Hello!

I have looked around a bit, but am surprised I could not find an answer to this question that I'm sure will be asked a lot more often starting today:

With the announcement from the Kubernetes team that as of v1.20, Docker will be deprecated as a runtime, what is the plan for Flatcar and its use of Docker?

I've only been using Flatcar for the last couple of months, for a POC for my employer, but I've been meaning to look at moving to pure containerd and away from Docker as container runtime since I first started working with Flatcar.  The announcement from k8s just makes it more of a priority.

I'm not completely clear on Flatcar's setup with Docker and containerd.  But using kubeadm, it seems it will check for the docker socket in /var/run/docker.sock and it's going to expect to use Docker if that's present.  containerd is installed, could I direct kubeadm to simply use the containerd socket somewhere, and thereby avoid using Docker for Kubernetes? (assuming that socket exists somewhere)

It seems I could get this going in a number of different ways, but I'd instead like to know the official direction Flatcar wants to go for moving away from Docker for Kubernetes, and then figure out the best way to switch to pure containerd without Docker.

Thanks!

Justin

[Vincent Batts]

Hi Justin,

Good question. There are a couple of approaches here. First and primarily, Flatcar has a utility called torcx (doc

 and project) that is used primarily to provide alternate versions of docker for a Flatcar host based on boot parameters.

We plan on providing a containerd torcx image that can be booted into. This would be a while before being default because a number of users and tools for a while will likely expect to call docker or its daemon API.

As noted, there is docker and containerd present in a default boot. The containerd socket is not in its _standard_ spot, but instead at /var/run/docker/libcontainerd/docker-containerd.sock.

That being said, many projects still have opinions on exactly which version of a runtime like containerd is being used. For that purpose, there is still the possibility to bootstrap your own specific version into /opt/, but that is largely an exercise left up to the reader.

vb

--
You received this message because you are subscribed to the Google Groups "Flatcar Container Linux User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatcar-linux-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/flatcar-linux-user/5f53a298-8cc6-4833-a9c0-659dff738461n%40googlegroups.com.

--

Vincent Batts

CTO

---
Kinvolk GmbH | Adalbertstr.6a, 10999 Berlin | tel: +491755589364
Geschäftsführer/Directors: Alban Crequy, Chris Kühl, Iago López Galeiras
Registergericht/Court of registration: Amtsgericht Charlottenburg
Registernummer/Registration number: HRB 171414 B
Ust-ID-Nummer/VAT ID number: DE302207000

[Justin Hopper]

Hello Vincent,

Thank you for the response.  And the direction of Flatcar does make sense wrt the need to stay backwards compatible and conformant with what your customer base expects.

After some testing, it turned out to be pretty simple to:

  - deactivate the default Docker and containerd processes in systemd

  - install the latest containerd components in /opt

  - start a custom systemd unit for the new containerd

  - modify kubeadm config to use the newly available containerd socket

Since we use a golden image process to build the images for our clusters, this should work fine to add this to our image provisioning process.  So I think we are mostly good on this front atm.

Justin

[Rodrigo Campos]

On Fri, Dec 4, 2020 at 12:42 AM Justin Hopper <justin....@gmail.com> wrote:
>
> Hello Vincent,
>
> Thank you for the response. And the direction of Flatcar does make sense wrt the need to stay backwards compatible and conformant with what your customer base expects.
>
> After some testing, it turned out to be pretty simple to:
> - deactivate the default Docker and containerd processes in systemd
> - install the latest containerd components in /opt
> - start a custom systemd unit for the new containerd
> - modify kubeadm config to use the newly available containerd socket

Awesome! Can you please share some code snippets of that so other
users reading this thread can more easily benefit?


Thanks,

--
Rodrigo Campos

[Kai Lüke]

Hi,

you may also modify the configuration shipped by Flatcar:

Create a /etc/systemd/system/containerd.service.d/10-config-location.conf
drop-in file:

[Service]
ExecStart=
ExecStart=/usr/bin/env PATH=${TORCX_BINDIR}:${PATH}
${TORCX_BINDIR}/containerd --config /etc/containerd/containerd.toml

Then copy /run/torcx/unpack/docker/usr/share/containerd/config.toml to
/etc/containerd/containerd.toml and edit as needed, finally run: sudo
systemctl daemon-reload; sudo systemctl restart containerd

The above should be enough to enable the cri plugin and delete the
socket address line to use the default socket location.

Provisioning custom binaries is also an option but one has to be
careful about the PATH. Here is an example where the PATH is set for
the systemd units, but not for interactive use, meaning manually
running commands is not safe unless export PATH=... is done first.
Maybe I can change the preparation service to set up bind-mounts so
that no PATH modification is needed.

This here replicates the config currently shipped by Flatcar but with
custom binaries.

systemd:
units:
- name: prepare-docker.service
enabled: true
contents: |
[Unit]
Description=Download docker binaries
ConditionPathExists=!/opt/bin/docker
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
RemainAfterExit=true
Restart=on-failure
ExecStartPre=/usr/bin/mkdir -p /opt/bin
ExecStartPre=/usr/bin/curl -sSfL -o /opt/docker.tgz
https://download.docker.com/linux/static/stable/x86_64/docker-19.03.13.tgz
ExecStartPre=/usr/bin/tar -v --extract --file /opt/docker.tgz
--directory /opt/ --no-same-owner
ExecStartPre=/usr/bin/rm /opt/docker.tgz
ExecStartPre=/usr/bin/sh -c "mv /opt/docker/* /opt/bin/"
ExecStart=/usr/bin/rmdir /opt/docker
[Install]
WantedBy=multi-user.target
- name: docker.socket
enabled: true
contents: |
[Unit]
PartOf=docker.service
Description=Docker Socket for the API
[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
[Install]
WantedBy=sockets.target
- name: docker.service
enabled: false
contents: |
[Unit]
Description=Docker Application Container Engine
After=containerd.service docker.socket network-online.target
prepare-docker.service
Wants=network-online.target
Requires=containerd.service docker.socket prepare-docker.service
[Service]
Type=notify
EnvironmentFile=-/run/flannel/flannel_docker_opts.env
Environment=DOCKER_SELINUX=--selinux-enabled=true
# the default is not to use systemd for cgroups because the
delegate issues still
# exists and systemd currently does not support the cgroup
feature set required
# for containers run by docker
Environment=PATH=/opt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
ExecStart=/opt/bin/dockerd --host=fd://
--containerd=/run/containerd/containerd.sock $DOCKER_SELINUX
$DOCKER_OPTS $DOCKER_CGROUPS $DOCKER_OPT_BIP $DOCKER_OPT_MTU
$DOCKER_OPT_IPMASQ
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to
accounting overhead
# in the kernel. We recommend using cgroups to do
container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups
of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
- name: containerd.service
enabled: false
contents: |
[Unit]
Description=containerd container runtime
After=network.target prepare-docker.service
Requires=prepare-docker.service
[Service]
Delegate=yes
Environment=PATH=/opt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
ExecStart=/opt/bin/containerd --config /etc/containerd/config.toml
KillMode=process
Restart=always
# (lack of) limits from the upstream docker service unit
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
[Install]
WantedBy=multi-user.target
storage:
files:
- path: /etc/containerd/config.toml
filesystem: root
mode: 0644
contents:
inline: |
# persistent data location
root = "/var/lib/containerd"
# runtime state information
state = "/run/docker/libcontainerd/containerd"
# set containerd as a subreaper on linux when it is not
running as PID 1
subreaper = true
# set containerd's OOM score
oom_score = -999
# CRI plugin listens on a TCP port by default
disabled_plugins = ["cri"]
# grpc configuration
[grpc]
address = "/run/containerd/containerd.sock"
# socket uid
uid = 0
# socket gid
gid = 0
[plugins.linux]
# shim binary name/path
shim = "containerd-shim"
# runtime binary name/path
runtime = "runc"
# do not use a shim when starting containers, saves on memory but
# live restore is not supported
no_shim = false
# display shim logs in the containerd daemon's log output
shim_debug = true


Regards,
Kai

> --
> You received this message because you are subscribed to the Google Groups "Flatcar Container Linux User" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to flatcar-linux-u...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/flatcar-linux-user/CACaBj2Z%3DBVRrSuM3PuBee2CHV0jta%2BdTf9ehOOrRtgnJOrDrAg%40mail.gmail.com.

[Justin Hopper]

On Fri, Dec 4, 2020 at 2:53 AM Rodrigo Campos <rod...@kinvolk.io> wrote:

On Fri, Dec 4, 2020 at 12:42 AM Justin Hopper <justin....@gmail.com> wrote:
>
> Hello Vincent,
>
> Thank you for the response.  And the direction of Flatcar does make sense wrt the need to stay backwards compatible and conformant with what your customer base expects.
>
> After some testing, it turned out to be pretty simple to:
>   - deactivate the default Docker and containerd processes in systemd
>   - install the latest containerd components in /opt
>   - start a custom systemd unit for the new containerd
>   - modify kubeadm config to use the newly available containerd socket

Awesome! Can you please share some code snippets of that so other
users reading this thread can more easily benefit?

Here's what I've done so far, and it seems to be suiting our needs (kubeadm bootstrap for k8s clusters).  Feel free to let me know if there's a more optimal way to do anything.

This is from our Ansible playbook for building Flatcar images that are ready for our clusters to use:

Stop and disable Docker

- name: stop docker
  raw: sudo systemctl stop docker

- name: disable docker
  raw: sudo systemctl disable docker

Install and setup crictl

- name: scp crictl.yaml file to host
  local_action: "command scp -o StrictHostKeyChecking=no -i {{ ansible_ssh_private_key_file }} -P {{ ansible_port }} {{ role_path }}/templates/etc/crictl.yaml {{ ansible_user }}@{{ ansible_host }}:/tmp/crictl.yaml"
 
- name: move crictl.conf file to /etc
  raw: sudo mv /tmp/crictl.yaml /etc/crictl.yaml

- name: create opt bin dir
  raw: sudo mkdir /opt/bin

- name: download and install crictl tarzip v{{ crictl_version }}
  raw: curl -L https://github.com/kubernetes-sigs/cri-tools/releases/download/v{{ crictl_version }}/crictl-v{{ crictl_version }}-linux-amd64.tar.gz | sudo tar -C /opt/bin -xz

Install containerd at newer version of 1.4.1

- name: disable default containerd
  raw: sudo systemctl stop containerd

- name: disable default containerd
  raw: sudo systemctl disable containerd

- name: download and install containerd tarzip v{{ containerd_version }}
  raw: curl -L https://github.com/containerd/containerd/releases/download/v{{ containerd_version }}/containerd-{{ containerd_version }}-linux-amd64.tar.gz | sudo tar -C /opt -xz
 
- name: scp containerd.service file to host
  local_action: "command scp -o StrictHostKeyChecking=no -i {{ ansible_ssh_private_key_file }} -P {{ ansible_port }} {{ role_path }}/templates/var/run/systemd/system/containerd.service {{ ansible_user }}@{{ ansible_host }}:/tmp/containerd.service"
 
- name: move containerd.service file to /etc/systemd/system/
  raw: sudo mv /tmp/containerd.service /etc/systemd/system/containerd.service

- name: reload systemctl
  raw: sudo systemctl daemon-reload

- name: start and enable containerd
  raw: sudo systemctl enable --now containerd

Lastly, here's the crictl.yaml file:

runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: true

And the containerd.service file:

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target

[Service]
ExecStart=/opt/bin/containerd
Restart=always
RestartSec=5
Delegate=yes
KillMode=process
OOMScoreAdjust=-999

LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity

[Install]
WantedBy=multi-user.target

-----------------------------------------------------------

Some things to note:

  - Using Ansible with Flatcar is a little rough, since python is not installed, so that's why I'm using all the raw calls. But it works fine. The Azure Flatcar image actually does have a python interpreter stashed away in on of their directories, but I decided not to use it.

  - I skipped setting up a config file for containerd, as it seems that the default values set for it work just fine.  There are loads and loads of options, an overwhelming amount almost.  So let me know if anybody has any suggestions for values that are must-have for a config file.

  - There's a lot of other things our Ansible playbook covers, but I pulled just the ones relating to containerd and docker for the above list.

Thanks,

Justin

[Justin Hopper]

Hello Kai,

Thanks for these suggestions.  It looks like you are doing a custom Docker install, and one of the things I'm trying to accomplish is bypassing/disabling Docker completely.  We want to run Docker-less.  I'm a true believe in less is more :)

Some of your config for containerd might be useful for our setup though, so I'll watch for replies to my previous post about what we have for our containerd setup.

Thanks,

Justin

To view this discussion on the web visit https://groups.google.com/d/msgid/flatcar-linux-user/CAL6HQvHGV0decq6MqA1p_rngnsg7i%3D14xN1pTNNiOZSRw40vJg%40mail.gmail.com.

[Kai Lüke]

Hello everyone,

we now have docs published on enabling the CRI plugin and using
containerd with Kubernetes:
https://kinvolk.io/docs/flatcar-container-linux/latest/container-runtimes/switching-from-docker-to-containerd-for-kubernetes/
You can choose to keep Docker working or not, depending on the socket
path value.
In the future Flatcar releases the CRI plugin will be enabled and the
workaround of a custom containerd configuration won't be necessary
anymore.

Regards,
Kai