How to totally reset etcd

[she...@36kr.com]

Dear all,

   I encountered problems with re-joining cluster due to etcd v0.4.6 fails. There are many similar posts in the forum. I found that:

1. etcd service uses /var/lib/etcd/* to store peer info

2. Even using new discovery.io and restart CoreOS, etcd can still recover certain data from /var/lib/etcd which leads to problem.

   Is there any way to totally reset etcd data? ex: remove files in /var/lib/etcd.  Hope etcd 2.0 will solve most of problems.

--

ShenLei

[Brian Kelley]

I have had dev clusters have issues and have had to perform a reset numerous times, and yes stopping etcd, fleet, and then removing the /var/lib/etcd directory and rebooting the machine has work consistently.

[John Griessen]

Yes,

I have started with 0.4 and switched to 2.0.8+ and use bare metal for my testing purposes, and there are
few docs for that and yet it can be driven by cloud-config and reboots IFF you do these steps:
make changes to cloud-config:
sudo vi /var/lib/coreos-install/user_data
sudo rm -rf /var/lib/etcd2/*
sudo rm -f /etc/systemd/system/etcd*
sudo reboot

The deleting of /etc/systemd/system/etcd* is not always necessary -- but since it is almost undocumented and
sometimes IS necessary -- I do it every time I am not starting from scratch with an image.

Things are much smoother with 2.0: I can remove a node and add node and have a cluster of nodes that
were not in the first cluster, but keep that first identity alive, no problems. I even could kill all nodes
and recover without effort, just boot up a few minutes later, unlike 0.4.6.

John Griessen

[Stéphane Lacasse]

Thank you, Thank you, Thank you,

 I have been messing around 3 days to set my first 3-nodes CoreOS cluster with a working etcd. This method did get it working as obviously there were always some leftovers of prior configuration. Documentation is not helpful yet to know where are all the knobs and levers. 

[Rob Szumski]

etcd stores data on disk at /var/lib/etcd2/. Removing all files in that directory will reset etcd, including clustering info. You will need to do this on all of your nodes and then re-cluster everything. 

If you used cloud-config to configure your discovery token, peer addrs, etc, you will need to update your cloud-config file with the new clustering info and new token.

 - Rob

On Dec 20, 2015, at 11:33 AM, Stéphane Lacasse <teckn...@gmail.com> wrote:

Thank you, Thank you, Thank you,

 I have been messing around 3 days to set my first 3-nodes CoreOS cluster with a working etcd. This method did get it working as obviously there were always some leftovers of prior configuration. Documentation is not helpful yet to know where are all the knobs and levers. 

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Juan L]

DUDE you are a life saver!!!!! Here are some polished instructions for AWS CoreOS instances

1. First remove the below

1. • sudo rm -rf /var/lib/etcd2/* 
   • sudo rm -f /etc/systemd/system/etcd* 

1. • sudo shutdown now
2. Go to CoreOs instance and add a new discovery url for each user data 

1. • NOTE only do # of nodes that are down - sometimes the remaining machine will add it’s self once quorum is established
   • Example, for a 3 node cluster do a discovery for 2 - https://discovery.etcd.io/new?size=2
2. Start 1 at a time
3. • verify node is added to discover url

The 2 nodes will create 

quorum then the 3rd node will add it’s self. if not, do the same as above.

If the 3rd node doesn’t add its self, add it manually from a good member - etcdctl member add node4 http://10.240.0.4:2380