Create Session Lock

[Damon Sutherland]

I am attempting to acquire a lock for failover. 

On my consul registered node, I have registered a custom health check (let's call it "A") that runs every 15s and the serfHealth check. While the service that acquires the lock is not running, this check is critical (as expected).  When I try to create a session, I am unable to do so because health check "A" is already in a critical state (from the Consul log ... http: Request /v1/session/create, error: rpc error: Check 'A' is in critical state).  

Any way around this?  Do I need to go to a TTL based key invalidation scheme, or is there a way to say create session but ignore the health checks for X seconds?

Thanks

[James Phillips]

Hi Damon,

There's currently no way to get the session if it depends on a check that's in the critical state. Managing health checks for standby services can be a little tricky.

A TTL-based check is a good way to solve this. The standby service can maintain the TTL to indicate it's alive and willing to take over, which is useful, and then the leader can maintain it which will give up the lock if it doesn't meet the TTL deadline. A detail here though is that depending on how things are set up you could have your leader and standby nodes with passing health checks for this service, which means that things like Consul's DNS will route traffic to both of them. You could solve this a number of ways, such as adding a tag to the active server, or adding an additional health check associated with the service that becomes healthy just for the leader.

-- James

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/e458fc51-986f-4c36-9edb-f49764ddcbe4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Damon Sutherland]

Thanks. I appreciate the response. As you suggested, I went with the TTL approach.

Damon

You received this message because you are subscribed to a topic in the Google Groups "Consul" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/consul-tool/UO_va4iwpuk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/CAGoWc07G00osLBKbg6Qap4T-B4yWwM-yy_3eNh4DkdVSVnCR6A%40mail.gmail.com.

[Gandalf Corvotempesta]

Hi,

could you make some example about this?

I'm trying to configure exactly the same, where multiple services are running on the cluster, but only one should be the leader.

I've configured a service check that is in critical state (as expected) but I'm unable to block leader election from the machine with failed service.

In example: a mysql server running (master-slave) on multiple nodes. One mysql is failed, but node is still up and thus compete to leader election. Obviously, this node must not became the leader, as there is a failed service on it.

How can I prevent this ?