Using Consul to manage singleton services

[mic...@dynamine.net]

Greetings,

I have a use case where I would like to have a highly-available singleton service: exactly one host can be running said service (we'll call it singleserv).  If singleserv is terminated on that host, or the host dies, another host must take over responsibility for running singleserv.

I'm currently using Redis to elect the managing host (using the Redis-Semaphore Ruby Gem), but Consul looks like an interesting alternative.

I read the documentation and performed a few simulations, but what's escaping me is how to manage session IDs: If the node that holds the lock associated with the session is deemed failed, after the lock-delay expires, the session is destroyed.  This phenomenon is not mentioned in the Leader Election guide, which seems to suggest that this is not a problem:

Watching for changes is done by doing a blocking query against key. If we ever notice that the Session of the key is blank, then there is no leader, and we should retry acquiring the lock.

But you can't acquire a lock associated with a destroyed session!

To acquire the lock, all candidate nodes must know its Session ID. When we're first setting up the candidate nodes for singleserv, we can generate a new session ID manually and communicate the ID to them out of band.  That's a bit annoying but workable.

But if that session is destroyed due to a node failure, to recover singleserv, manual intervention is required: someone must generate a new session ID and communicate the new Session ID to the remaining candidate nodes so that they can successfully perform an election.  In the meantime, singleserv is down, which is obviously bad for service availability.

What are your recommendations?

Thanks,

--Michael

[Armon Dadgar]

Hey Michael,

I guess the documentation is a bit unclear about this. The session IDs should not be shared

between nodes. Sessions are specifically tied to a particular node. In this case, each instance

of singleserv should have it’s own Session ID. Only the lock is used to share between them.

Take a look at the consul-replicate project which uses sessions to perform this same

type of mutual exclusion: https://github.com/hashicorp/consul-replicate

Let me know if that helps clear it up!

Best Regards,

Armon Dadgar

--
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael S. Fischer]

Hi Armon,

At first I was puzzled by your response, but I tried the experiment
again, this time having each candidate generate its own session ID and
calling the agent as follows:

curl -X PUT -d '{"owner":"candidate_name"}'
http://localhost:8500/v1/kv/service/singlesrv?acquire=candidate_session_id

I'll submit a PR with a proposed documentation update in the hopes
that it will help clarify matters.

Thanks,

--Michael