Redis HA cluster job spec?

[Michael Weber]

Hi,

it is easy to find examples for basic (standalone) redis setups with nomad, but does anybody run a production Redis HA cluster on top of Nomad?  E.g.,

• N Redis masters with K slaves each
• placement constraints such that 
• corresponding masters and slaves are not on the same machine
• no more than K instances in the same failure domain (chassis, rack, etc.)
• stable cluster entry point (via DNS, consul) exposed to clients (e.g., the set of masters)

[Bill Anderson]

So first things first, you'll need to be more accurate in your Redis description due to Salvatore's (poor) naming choice. There is Redis HA, which is single master/1+ slaves and is managed by a Redis Sentinel cluster, and Redis Cluster, which is multiple masters with one or more slaves each. Next, because there are two primary components to this question (Consul and Nomad), I'll break them into individual sections.

# Consul 

## Redis HA

For the first scenario, Redis in an HA setup, the setup is pretty straightforward until you want Consul to know the difference between master and slaves.  I've not tested it yet (mostly because Nomad isn't really a good choice for anything beyond an ephemeral single Redis cache-only instance) but you would probably need to do something like this:

1. Write a check that passes if master as determined by querying Sentinel for current master and comparing to self IP
   
2. Write a check that passes if slave as determined by querying Sentinel for current master and checking for current IP in the set.
   
3. Create a service for redis-master using the pass-if-master health check
4. Create a service for redis-slave using the pass-if-slave health check
5. Define a "Prepared Query" which creates a DNS of "redis-master.blah.blah" using a condition that specifies the needed service
   

Which all looks perfectly cromulent. What I'd do instead is:

1. Write/use a sentinel event handling script to create/update Redis as an "external service" in Consul. It would need to create/update a "redis-master" service and a "redis-slave-pool" service.
2. Configure the pod in Sentinel to use the handler

This gets you automated updates to Consul's external service definitions for both master and slave. The sentinel-event handler will trigger when a failover occurs. I've done something similar to this, but w/o the External service bit (I updated sentinel KV which was watched by a consul-template+HAProxy setup) and you can find a framework for developing such a thing in Go at https://github.com/sentinel-tools/eventilator. I'll probably add the Consul External Service update bit at some point (ie. when I have both the time and motivation ;) ).

## Redis Cluster

As bad as the previous section sounds, this one is kind of worse because of the way Redis Cluster (RC hereafter) operates. To use RC you MUST have client support because each master is not interchangeable. RC shards the keys across the masters, so you need to know which master based on the key name. You have your bank of, say 7, masters and the *client* must either pull down the slot layout and/or handle and store the redirects the master will give you if you talk to the wrong master. 

So in order to "bootstrap" a client you need to be able to talk to ant given master, but once you have that the client will need to keep itself updated via direct communication with the existing nodes. That should be fairly easy to by following the first itemized list above. A given master or slave can change its role w/o Nomad or Consul knowing about it because in RC if a master is lost the slaves will select one of their own to promote. Thus you have to periodically query all instances and build out a list of those that have the role of slave and those that have the role of master. Unfortunately, I suspect, that means you'd have a two lists of always failing service checks because each instance can only pass one of the two checks. It sounds fairly ugly to me. But I think it can be done, given those warts. Alternatively you could write a third service that is run as a "daemon" job which queries the cluster to get the list of masters and slaves, then uses Consul's External Service mechanism similarly to the Sentinel option but on a scheduled basis (either the service sleeps and works, or you run a scheduled Nomad task that queries and updates, then exits).

As you can see, doing this in Consul is non-trivial.

# Nomad

This is where it gets really ugly. The Consul portion itself is comparatively easy. Nomad isn't really designed to handle spinning up things that manage themselves but don't bootstrap themselves. Again, because there are two main scenarios here, I'll describe how Nomad can't do either nicely.

## Redis HA

To set this up you need to select a master and know it's connectivity information. You can't know that ahead of time in Nomad. You *could*, conceivably, write a job that gets the list of spun-up instances, selects one at random, tells all the others it is the master, then tells the sentinel cluster about it. But you'd have to have it ensure it only acts if there are no masters AND the pod is not configured in Sentinel. After that Nomad has to continue stay out of the pod's configuration because Sentinel will handle it, except in the case of bringing up a new instance that needs to be told who the current master is.

## Redis Cluster (RC)

This requires a lot of setup, such that it comes with a ruby script to set up a cluster. So at best when you initially build the cluster you _could_ have a task which spins up the fresh instances and then runs the script when done. Similarly to the the HA setup you'd also need something that ran on a new node that told it what it was and who else was in the already-existing cluster. But then you would also have to reconfigure the slot layout and migrate things, so that may not be as workable as you'd like.

It is conceivable that you could use a container that uses the Consul election/lock API combined with queuing for fellow nodes to implement a bootstrap mechanism for the Redis HA/Sentinel setup, but that is more involved than I'm willing to write down in an email (plus I'd have to test the crap out of it). ;) But for RC it requires so much manual bits that while you could possibly do the same in a container, you could not handle replacement or expansion nodes. I think it would be handled better using Terraform, though even there you'll still need some custom work.

It is both sad and funny that almost every "platform" uses starting up a single Redis instance as their example. I get it, doing that is really dead simple. But running anything Redis that goes beyond that singular instance is non-trivial. Personally I'd love it Redis itself had a means to use consul to self-bootstrap or to auto-join-as-slave using Consul built in. But, alas, I don't see Salvatore implementing it anytime soon. I did talk to him a couple years ago about Redis using Consul as a configuration storage to make this easier and he liked the _idea_, but doing the actual implementation is a different ball game - and not a trivial one at that.

As far as constraints for deploying instances across failure domains, the inbuilt filtering based on tags/node attributes and the group mechanism should work just like any other job.

[Balaji J]

Hi Mike,

I am looking for similar solution using nomad for redis.

Were you able to get some working solution of using nomad to manage redis-cluster?

Pls share if so.

Thanks,

...Balaji.J

[AS]

I'm also looking for the same config.

[Vasilios Tzanoudakis]

I am looking for the same example too.

You can also look at Resec project for similar consul based redis HA 

https://github.com/YotpoLtd/resec/blob/master/README.md

Vasilios Tzanoudakis

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/a80aeb33-f669-4af1-8567-792ee16d1805%40googlegroups.com.