// , How do I completely remove a node from a Consul cluster?

[Nate B]

  // , I'm running into a bit of a problem with nodes joining each other's cluster even when I don't want them to.

I've used /agent/leave, /catalog/deregister, and a few other things, to no avail.

Trying to not get the clusters to join each other seems like trying to pry apart Romeo and Juliet, at this point.

How do I keep one Consul Cluster KV store and Consul Leader from overwriting the Consul Leader & KV store of another Consul Cluster?

Consul parting is such sweet sorrow,

--Nathan Basanese

P.S. I posted about it here:
    https://serverfault.com/questions/852520/how-do-i-completely-remove-a-node-from-a-consul-cluster

But maybe it's better to discuss this sort of issue in the Consul mailing list than a Q&A site.

[James Phillips]

Hi Nathan,

Using different datacenters is the most basic way to prevent them from
joining. TLS + gossip encryption +
https://www.consul.io/docs/agent/options.html#verify_server_hostname
can be used to enforce it.

If you've turned down the rogue server, you should be able to
force-leave it from the datacenter where it isn't supposed to be (you
have to do it from there as you noted, and that API takes the node
name, not IP). That'll also remove it from the Raft configuration,
otherwise you have to wait 72 hours for it to get reaped. If somehow
you can't force-leave it, you can remove it from Raft using
https://www.consul.io/docs/commands/operator/raft.html#remove-peer,
which should get the servers to forget about it.

-- James

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/consul/issues
> IRC: #consul on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Consul" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to consul-tool...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/consul-tool/ad513034-0db0-43e8-aefa-f00dea3cf36b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Nate B]

  // , I did use different datacenters, but that hasn't prevented the old datacenter from immediately assuming that a Consul Server in the new DC, prod0, is its new Consul Leader.

I also tried the force-leave option, but that failed to prevent assigning a new leader outside the configured dc.

I will try this
https://www.consul.io/docs/commands/operator/raft.html#remove-peer
suggestion and see if that resolves the issue.

If the issue is reproducible, afterward, it may be a bug in Consul or Raft, though I hope not.

[Nate B]

[nathan-basanese-zsh8@prd0consulserver3 ~]$ consul operator raft list-peers
Node        ID                   Address              State     Voter  RaftProtocol
(unknown)   192.176.100.3:8300  192.176.100.3:8300  follower  true   <=1
(unknown)   192.176.100.2:8300  192.176.100.2:8300  follower  true   <=1
...
[nathan-basanese-zsh8@prd0consulserver3 ~]$ consul operator raft remove-peer 192.176.100.2:8300
Error removing peer: an address or id is required for the peer to remove

Maybe there is a bug in the raft protocol, or something, but I just cannot get that peer removed.

[Nate B]

  // , In addition, I also get the following on all the Alpha0 servers, although they correctly list their members, and don't list any members from production:

[nathan-basanese-zsh8@alpha0consulserver2.alpha0 ~]% consul operator raft list-peers
Error getting peers: Failed to retrieve raft configuration: Unexpected response code: 500 (No cluster leader)

[George Chilumbu]

Lets say your service config is located in /etc/consul.d/services.json, and the service-name is db-test, then you can deregister this service by running the command 

curl -s --upload-file /etc/consul.d/services.json http://127.0.0.1:8500/v1/agent/service/deregister/db-test

So just an example, my services.json file in /etc/consul.d/services.json looks like this:

{

    "services": [

        {

            "name": "db-test",

            "tags": [

                "writes",

                "reads"

            ],

            "port": 3306,

            "checks": [

                {

                    "script": "/etc/consul.d/scripts/mysql.sh",

                    "interval": "5s"

                }

            ]

        }

    ]

}

Not sure if this is exactly what you are looking for. But give it a try.

[George Chilumbu]

Lets say your service config is located in /etc/consul.d/services.json, and the service-name is db-test, then you can deregister this service by running the command 

curl -s --upload-file /etc/consul.d/services.json http://127.0.0.1:8500/v1/agent/service/deregister/db-test

So just an example, my services.json file in /etc/consul.d/services.json looks like this:

{

    "services": [

        {

            "name": "db-test",

            "tags": [

                "writes",

                "reads"

            ],

            "port": 3306,

            "checks": [

                {

                    "script": "/etc/consul.d/scripts/mysql.sh",

                    "interval": "5s"

                }

            ]

        }

    ]

}

Not sure if this is exactly what you are looking for. But give it a try.

[James Phillips]

Hi Nate,

Sorry for the late reply, but you need -address, so:

consul operator raft remove-peer -address=192.176.100.2:8300

On Mon, Jul 3, 2017 at 12:09 PM, Nate B <nbas...@gmail.com> wrote:
> // , In addition, I also get the following on all the Alpha0 servers,
> although they correctly list their members, and don't list any members from
> production:
>

> [nathan-bas...@alpha0consulserver2.alpha0 ~]% consul operator raft

> https://groups.google.com/d/msgid/consul-tool/4117fbb3-0bc1-4e8b-a530-1fcc26c72b00%40googlegroups.com.

[Nate B]

// , That actually worked, thanks.

> You received this message because you are subscribed to a topic in the Google Groups "Consul" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/consul-tool/Mnf71dzLi-k/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to consul-tool...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/CAGoWc07iJbsp%2BuJPyfUZeLMvADPoxsYtB8%2Bxe%3DVuBsZyG3Ogmg%40mail.gmail.com.