Cleaning up misconfigured hosts

[Pablo Fischer]

Hello folks,

Over the past we had some misconfigured hosts. Such misconfiguration as:

- Using the wrong consul servers (prod / development) on some hosts.
- Using the wrong datacenter on the hosts.

All those hosts have been fixed since maybe over the last 1-2 months,
however we still see some messages of them on _any_ consul agent, even
if they are on different environment.

For example:

2015/09/17 23:23:09 [WARN] memberlist: ignoring alive message for
'foo101': Member 'foo101' part of wrong datacenter 'meh'

etc..

foo101 got his datacenter fixed a long time ago but we still see in
the logs. So question: is there a way to clean them up?

Thanks!
--
Pablo

[Armon Dadgar]

Hey Pablo,

What version is the cluster running on? Newer versions do a much better job of

filtering out messages from mixed clusters as well as guarding against that from

happening in the first place. The simplest answer may be to just upgrade the version

of Consul everywhere.

Best Regards,

Armon Dadgar

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/CAJRPuoytvO3hiRg7q71bpre_g1kWiztpD1%2BuU3zv939TqKKqKQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Pablo Fischer]

Hey Armon,

Apologies for replying so late.

The version we are running on production is the last one, 0.5.2,

$ consul version
Consul v0.5.2

I'm personally not too worried about those logs, but I'm worried that
I think there is a misconfiguration and once an agent connects to the
wrong host it will just keep trying to talk to it.

As mentioned, we have two environments, dev and prod. On the dev
environment I'm able to see some hosts from prod. Even worse, when I
DNS query the local agent it seems to return hosts only from the prod
environment not the dev.

When I check the peers.json I see the the production and the dev
consul servers. One thing tho, I'm using the same datacenter on both
environments but each environment with different set of consul
servers.

Is there a way to tell consul to reconfigure its peers? Or is the only
way to: 1) stop consul, 2) remove the data directory, 3) start it
again?

--
Pablo

[Armon Dadgar]

Hey Pablo,

Its typically easier to use iptables rules to drop all traffic between the two clusters.

This will cause each cluster to think machines in the other cluster have failed.

You can then use `force-leave` to cleanly separate the two clusters.

Hope that helps!