Consul serf health check - removes node from cluster

[mike r]

Hello all, Im running a 3 server cluster (consul 0.9.2)

I have a client node running on a Splunk server, with a splunk service healthcheck,

The Splunk node is registered with Consul and shows 2 healthcheckes as Green (serf health, splunk service)

if I stop Consul service on the Splunk node, the entire node disappears from the Cluster

I dont want it to disappear but appear as Critical so I know something is up on that Splunk box, am I missing a setting somewhere?

From Consul docs, it says something like this is available, 

An astute reader may notice that if a Consul agent dies, then no edge triggered updates will occur. From the perspective of other nodes, all checks will appear to be in a steady state. However, Consul guards against this as well. The gossip protocol used between clients and servers integrates a distributed failure detector. This means that if a Consul agent fails, the failure will be detected, and thus all checks being run by that node can be assumed failed. This failure detector distributes the work among the entire cluster while, most importantly, enabling the edge triggered architecture to work.

Server config,

{

  "bind_addr": "10.185.20.180",

  "data_dir": "/var/consul",

  "server": true,

  "ui": true,

  "bootstrap": true,

  "retry_join": ["10.185.20.179", "10.185.20.180", "10.185.20.181"],

  "datacenter": "mrx",

  "enable_script_checks": true,

  "encrypt": "UAvkAzdjGfQ7J2NlgkrJMA==",

  "enable_syslog": true,

  "addresses": {

    "dns": "10.185.20.180",

    "http": "10.185.20.180",

    "https": "10.185.20.180"

  },

  "dns_config": {

    "allow_stale": true,

    "max_stale": "30s",

    "node_ttl": "30s",

    "enable_truncate": true,

    "only_passing": true

  },

  "acl_datacenter": "mrx",

  "acl_down_policy": "extend-cache",

  "acl_default_policy": "deny",

  "acl_master_token": "dbef8b5a-6110-4575-bf61-dda1c21ca339"

}

Splunk Client config

{

  "bind_addr": "10.185.20.173",

  "data_dir": "/var/consul",

  "ui": false,

  "server": false,

  "retry_join": ["10.185.20.179", "10.185.20.180", "10.185.20.181"],

  "datacenter": "mrx",

  "encrypt": "UAvkAzdjGfQ7J2NlgkrJMA==",

  "enable_syslog": false,

  "enable_script_checks": true,

  "pid_file": "/var/consul/consul.pid",

  "acl_token": "548bb56f-33c9-622a-4351-1a04851ebb1a",

  "acl_datacenter": "mrx"

}