etcd3 how can we get it running on coreos?

[Bryant Zimmerman]

Why is the etcd version 3 binary not in the current stable release of coreos? The kub docs hint that etcd v3 should be used is this not a good idea?

How do we best upgrade the current stable release of coreos with the v3 binary?

Currently have downloaded the binary to /opt/bin but we are having limited luck getting it to start and use environment variables. The only way we have had any success getting a v3 cluster running is by adding the settings to launch command..

Any feed back would be appreciated.

Thanks

Bryant

[Euan Kemp]

There are docs for running etcd3 here:
https://coreos.com/etcd/docs/latest/getting-started-with-etcd.html#setting-up-etcd

If you know of other places in our docs that need updating, an issue or
pr would be appreciated!

> The kub docs hint that etcd v3 should be used is this not a good
> idea?

Kubernetes indeed recommends v3, and for 1.5+ clusters it should be
used. There's still ongoing work for 1.6 to make the documentation
clearer and make the cluster-deployment tools (e.g. kops) use etcd3 too.
Some of that's being tracked here:
https://github.com/kubernetes/features/issues/44

> --
> You received this message because you are subscribed to the Google
> Groups "CoreOS User" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to coreos-user...@googlegroups.com
> <mailto:coreos-user...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Bryant Zimmerman]

Euan

Thanks for the reply..

The docs link you gave for the etcd v3 is a good start and what we used to get things working, but what we found is that the etcd-member.service points to /bin/etcd  but that version of the file in the current stable release is the old 0.4.x version which is not the v3 binary. We can find no copy of the v3 binary in the current stable coreos installation. That is way we were so confused.

We had to manually download the binary from the GitHub  we pulled v3.1.3, and had to place it into another location /opt/bin as we could not replace the copy in /var/bin folder.  My concern is this puts us out of band for updates.

Is there an easy way to over write the older copy, and ensure that updates to coreos will have the updated etcd v3 support in the future?

You also ask for issues or pr's (what is a pr?) where do we open issues and pr ? We have found a number of issues during this process. Getting them updated would save outers and us from confusion in the future.

[David Michael]

On Tue, Mar 21, 2017 at 2:45 PM, Bryant Zimmerman
<bryant.z...@gmail.com> wrote:
> Euan
>
> Thanks for the reply..
>
> The docs link you gave for the etcd v3 is a good start and what we used to
> get things working, but what we found is that the etcd-member.service points
> to /bin/etcd

Where do you see that? The etcd-member.service distributed with
Container Linux does not reference an etcd binary. It uses the etcd
v3 rkt container.

> We had to manually download the binary from the GitHub we pulled v3.1.3,
> and had to place it into another location /opt/bin as we could not replace
> the copy in /var/bin folder. My concern is this puts us out of band for
> updates.

Yes, and that is unnecessary. The etcd-member.service file will be
updated with the OS to pull new versions of the container by default.

> Is there an easy way to over write the older copy, and ensure that updates
> to coreos will have the updated etcd v3 support in the future?

The /usr partition is read-only, so you can't overwrite anything on
it. (The closest option is bind mounts or overlays, but don't do
that.) The OS will automatically update the etcd v3 version as long
as you don't override the ETCD_IMAGE_TAG variable in
etcd-member.service.

> You also ask for issues or pr's (what is a pr?) where do we open issues and
> pr ? We have found a number of issues during this process. Getting them
> updated would save outers and us from confusion in the future.

Euan was referring to GitHub pull requests. General documentation is
at the URL below, but you might find other documentation in the
specific projects' repositories.

https://github.com/coreos/docs/

Some projects have their own issue trackers on GitHub, but the general
place to file bugs is:

http://issues.coreos.com/

Thanks.

David

> On Tuesday, March 21, 2017 at 3:24:00 PM UTC-4, Bryant Zimmerman wrote:
>>
>> Why is the etcd version 3 binary not in the current stable release of
>> coreos? The kub docs hint that etcd v3 should be used is this not a good
>> idea?
>>
>> How do we best upgrade the current stable release of coreos with the v3
>> binary?
>> Currently have downloaded the binary to /opt/bin but we are having limited
>> luck getting it to start and use environment variables. The only way we have
>> had any success getting a v3 cluster running is by adding the settings to
>> launch command..
>>
>> Any feed back would be appreciated.
>>
>> Thanks
>> Bryant
>

> --
> You received this message because you are subscribed to the Google Groups
> "CoreOS User" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to coreos-user...@googlegroups.com.

[Bryant Zimmerman]

Show trimmed content

David

Thank you for your response. I am now a bit more perplexed. How do we verify the version and rkt container for etcd on coreos?

It makes a bit more sense what we were seeing if it was running in a container, but this likely means we don't have our cloudconfig correctly set.

All the etcd examples we found for the cloudconfig shows either etcd or etcd2, but no etcdv3 examples from the container.

Is there any examples you could point us to on how to setup the cloudconfig to configure the etcd v3 rkt container. We initially are trying to setup a 3 node etcd cluster so we can get k8 up and running.

I found examples here https://github.com/coreos/matchbox/tree/master/examples, but none look to show using bare metal and the cloudconfig.yaml

Thanks

Bryant

[Alex Crawford]

On 03/23, Bryant Zimmerman wrote:
> How do we pull down the rkt container for etcd v3 on to coreos?

etcd-member.service does this for you. You just need to start the
service and it will download and run etcd. You should be able to treat
it the same as etcd.service or etcd2.service.

> All the etcd examples we found for the cloudconfig shows either etcd or
> etcd2, but no etcdv3 examples from the container.

coreos-cloudinit does not have support for etcd-member.service. This is
because we are in the process of sunsetting that project in favor of
Container Linux Configs. This new system does support
etcd-member.service [1]. If you are interested in learning more about
these new configs, you can read more in our docs [2].

> Is there any examples you could point us to on how to setup the cloudconfig

> to configure the etcd v3 rkt container. We initially are trying to setup a

> 3 node etcd cluster so we can get k8 up and running.

I know there are examples floating around but I can't find them at the
moment. I can look again in a bit if you are unable to find them.

-Alex

[1]: https://github.com/coreos/container-linux-config-transpiler/blob/v0.2.0/doc/examples.md#etcd
[2]: https://coreos.com/os/docs/latest/overview-of-ct.html

[Rick Moran]

Bryant,

    I spent some time fiddling around on some new clusters on both bare metal and on digital ocean.

You can see an example of what I came up with here:

https://gist.github.com/DrHayt/ea382f4d13af0e5adca3e85f915d5146

Hopefully this is self explanatory.

On bare metal I put this file called user_data in like /var/lib/coreos I think.

Thanks,

Rick

[Robin Gilh]

I just saw your cloud-config and I don't understand how you manually copied your certs to be recognized by etcd. Mine always throws an error on startup.

> etcdmain: open /etc/ssl/certs/etcd/ca.pem: no such file or directory