Running on GCE with etcd3 and flannel backend

[Osman Ali]

I am going through the documentation for running coreos on GCE and have some questions regarding use.

1. How do I use etcd v3 on GCE to setup the flannel backend?

I have followed the instructions here:

https://coreos.com/flannel/docs/latest/gce-backend.html

But that only allows for the creation of etcd Version: 2.3.7.

Is there any updated documentation?

2. How do I setup etcd v3 on GCE using a cloud config file?

I have tried to create a an upgraded cluster using a cloud config file as follows:

#cloud-config

coreos:

  etcd:

    advertise_client_urls: "http://$INTERNAL_IP:2379"

    listen_client_urls: "http://0.0.0.0:2379"

However, that file does not validate in the online cloud config validator:

deprecated key "advertise_client_urls" (etcd2 options no longer work for etcd)

deprecated key "listen_client_urls" (etcd2 options no longer work for etcd)

When I create a new coreos instance on GCE I get the following when I run etcd2:

2017-05-29 09:00:12.518926 I | etcdmain: etcd Version: 2.3.7

2017-05-29 09:00:12.518979 I | etcdmain: Git SHA: fd17c91

2017-05-29 09:00:12.518983 I | etcdmain: Go Version: go1.7.6

2017-05-29 09:00:12.518986 I | etcdmain: Go OS/Arch: linux/amd64

2017-05-29 09:00:12.518990 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1

2017-05-29 09:00:12.518994 W | etcdmain: no data-dir provided, using default data-dir ./default.etcd

2017-05-29 09:00:12.522628 I | etcdmain: listening for peers on http://localhost:2380

2017-05-29 09:00:12.522731 I | etcdmain: listening for peers on http://localhost:7001

2017-05-29 09:00:12.522802 I | etcdmain: listening for client requests on http://localhost:2379

2017-05-29 09:00:12.522872 I | etcdmain: listening for client requests on http://localhost:4001

2017-05-29 09:00:12.522918 I | etcdmain: stopping listening for client requests on http://localhost:4001

2017-05-29 09:00:12.522926 I | etcdmain: stopping listening for client requests on http://localhost:2379

2017-05-29 09:00:12.522932 I | etcdmain: stopping listening for peers on http://localhost:7001

2017-05-29 09:00:12.522937 I | etcdmain: stopping listening for peers on http://localhost:2380

2017-05-29 09:00:12.522942 C | etcdmain: cannot access data directory: mkdir default.etcd: permission denied

3. How do I manage etcd versions as they are both setup as services?

On any GCE instance, when I run:

export ETCDCTL_API=3

etcdctl version

The api does not use the updated version.

It says here:

https://github.com/coreos/docs/blob/master/etcd/etcd-live-cluster-reconfiguration.md

That using a cloud config file to configure etcd creates a drop in but I do not see a drop in the location indicated:

/etc/systemd/system/etcd-member.service.d/20-clct-etcd-member.conf

In /usr/lib/systemd/system/ there are the etcd-member.service and etcd2.service

Etcd2.service is as follows:

[Unit]

Description=etcd2

Conflicts=etcd.service

[Service]

User=etcd

Type=notify

Environment=ETCD_DATA_DIR=/var/lib/etcd2

Environment=ETCD_NAME=%m

ExecStart=/usr/bin/etcd2

Restart=always

RestartSec=10s

LimitNOFILE=40000

TimeoutStartSec=0

[Install]

WantedBy=multi-user.target

etcd-member.service is as follows:

[Unit]

Description=etcd (System Application Container)

Documentation=https://github.com/coreos/etcd

Wants=network.target

Conflicts=etcd.service

Conflicts=etcd2.service

[Service]

Type=notify

Restart=on-failure

RestartSec=10s

TimeoutStartSec=0

LimitNOFILE=40000

Environment="ETCD_IMAGE_TAG=v3.1.6"

Environment="ETCD_NAME=%m"

Environment="ETCD_USER=etcd"

Environment="ETCD_DATA_DIR=/var/lib/etcd"

Environment="RKT_RUN_ARGS=--uuid-file-save=/var/lib/coreos/etcd-member-wrapper.uuid"

ExecStartPre=/usr/bin/mkdir --parents /var/lib/coreos

ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/lib/coreos/etcd-member-wrapper.uuid

ExecStart=/usr/lib/coreos/etcd-wrapper $ETCD_OPTS

ExecStop=-/usr/bin/rkt stop --uuid-file=/var/lib/coreos/etcd-member-wrapper.uuid

[Install]

WantedBy=multi-user.target

[Rob Szumski]

Answers below.

1. How do I use etcd v3 on GCE to setup the flannel backend?

Is there any updated documentation?

Once you have an etcd v3 cluster, you will need to use/export this variable for etcdctl: ETCDCTL_API=3. The documented command should work fine after that.

2. How do I setup etcd v3 on GCE using a cloud config file?

I have tried to create a an upgraded cluster using a cloud config file as follows:

3. How do I manage etcd versions as they are both setup as services?

Here’s an example template. This uses the “etcd-member” service, which runs etcd as a container with the flags specified. It allows you to specify the container version as well, to give you more control over the service.

Hope that helps.

 - Rob

[Osman Ali]

Thanks! This was very useful. Managed to solve for a single instance of etcd v3 as follows:

Used etcd-member.service with the following drop in:

[Service]

Environment="ETCD_ADVERTISE_CLIENT_URLS=http://$INTERNAL_IP:2379"

Environment="ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379"

I manually inputted the internal ip value.

Then:

ETCDCTL_API=3 etcdctl set /coreos.com/network/config '{"Network":"10.40.0.0/16", "Backend": {"Type": "gce"}}'

Ran with latest version of flannel and was able to acquire the subnet least.

Next steps will be to setup etcd v3 cluster and add then secure cluster.

When I run etcdctl -v, I still get:

etcdctl version 2.3.7

Even after I run export ETCDCTL_API=3, the value is the same for the version.

Does this indicate that I am not using v3 or just the backward compatibility?

Thanks.

[Rob Szumski]

Glad you got it working. The version thing is normal since you can use etcdctl with both APIs. It just happens to have a 2.x version.

[Osman Ali]

Perfect. Thank you very much.