Redis: failover and cluster-support

[Andreas]

First I would like to thank Markus Ellinger for contributing Redis support too!

I tried the session-manager with multiple Redis configurations, especially with a small cluster (3 masters, 3 slaves) and a single instance with a single slave.

Using the cluster isn’t working (“JedisMovedDataException”), I expected that.

Accessing the single instance works well, but a failover to the slave isn’t working too. (“JedisConnectionException”).

I’m no expert on Jedis. As far as I can see you have to use a Redis Sentinal in order to support failover and a JedisCluster-Client to support cluster config.

 

My questions are:

Is it true that cluster and failover aren’t supported yet? Or am I doing something wrong?

Are there any plans on supporting clusters and/or failovers?

 

And allow me one remark:

If my assumptions are correct, the sentence “Alternatively, you can use a Redis server and Redis' built-in replication.” on the SetupAndConfiguration-page is misleading ;-).

Cheers, Andreas

[Martin Grotzke]

Hi Andreas,

there has just been an issue regarding Redis support, where Markus has
improved the Redis related documentation:
https://github.com/magro/memcached-session-manager/issues/316

Do you still see room for improvement?

Cheers,
Martin

> a Redis server and Redis' *built-in replication*.” on the

> SetupAndConfiguration-page is misleading ;-).
>
>
> Cheers, Andreas
>

> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "memcached-session-manager" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to memcached-session-...@googlegroups.com
> <mailto:memcached-session-...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

--
inoio gmbh - http://inoio.de
Schulterblatt 36, 20357 Hamburg
Amtsgericht Hamburg, HRB 123031
Geschäftsführer: Dennis Brakhane, Martin Grotzke, Ole Langbehn

[Markus Ellinger]

Hi Andreas,

right, only a single Redis Node is supported at the moment, mainly since in my own projects there was no need for more. Anyway, I would be interested in your views. Could you explain your use case for having multiple nodes?

If it is for load balancing I doubt there are many applications where you need more than one Redis server per Java app for the sole purpose of session management, and even then I would think that a setup where you just partition the frontends should be just simpler (let e.g. Tomcat instances 1 to 20 talk to Redis 1 and instances 21 to 40 talk to Redis 2).

If it is for failover, I am no expert in Redis either, but it seems to me that it should be easy to setup two Redis instances where the master Redis constantly streams to the slave, then, when the master fails, change DNS to point to the slave. This is how Amazon Elasticache for Redis does it. This has the advantage that, unlike with the built-in memcached cluster support, MSM does not need to copy every session to two Redisses. Also, since Redis supports serializing to disk, also a “cheap“ failover mode can be implemented where only one Redis instance is running but when this instance crashes you can spin up a replacement instance which at least still has the old data (however, there will be a short downtime while the other instance is spinning up).

Best regards
Markus

[Markus Ellinger]

Hi Andreas,

I researched this topic a bit more. As a result, I removed the ambiguous phrase in the docs you mentioned and also added a disclaimer to the page that Redis Sentinel (which seems to be the preferred way to implement Redis failover nowadays) is not supported at the moment.

I would still be interested in your use case and how I can be of any help in implementing it. The Redis client library we use (Jedis) indeed has built-in support for Jedis Sentinel, but I would need to change the code a bit and expose a configuration setting for it. If I add support for Redis Sentinel, would you be willing to test it thoroughly?

Best regards
Markus

[Andreas]

Hi Markus,

the use case is quite simple (or at least short).

“Requirements”:

-        - multiple web-applications

-        - high-availability

-        - no single point of failure

-        - sticky session isn’t possible for several reasons (!)

In my first post I forgot to mention that additionally only one redis node can be configured –  memcachedNodes:redis://example.com:123,redis://example.com:124 isn't supported … right? In a non-sticky setup this leads to a single point of failure (redis).

Anyway – our preferred solution is a single redis cluster as session store for all applications, which makes it much easier for us to scale the session store.

We are currently evaluating several alternatives. We probably won’t choose an experimental solution – this means I probably won’t be able to test the sentinel. Sorry. I’ll let you know, if the situation changes.

Anyway – thx a lot so far for the quick responses. I’ll do my best to be quicker next time ;-).

Cheers, Andreas

[Markus Ellinger]

Hi Andreas,

thanks for your response, I am slowly getting an impression of what you want to achieve. When you say "you are currently evaluating several alternatives", may I ask what those alternatives are? If you know other solutions for high availabiity with Redis and Tomcat, I would be interested in them, as, to be honest (and I hope Martin will forgive me that I say that), I would probably not have written the Redis support for MSM would there have been a real alternative available. So I can actually say that I myself already went through a similar process you are now going through and in the end decided to use MSM, even though I had to write Redis support for it. Of course, when by "alternative" you mean e.g. just writing the app in PHP instead of Java then it would be a different situation ;-)

As for your actual requirements, it seems to me that you are looking for two aspects:

1. Automatic load balancing between multiple Redis nodes.

2. Automatic fail over when a node fails

It might be tempting to say "oh, we just set up a Redis cluster of 5 nodes, then, when one node fails, the others take over", but that's an oversimplification and it just doesn't work like this in practice. For example, to do load balancing with a sharding architecture, you need to actively partition the key space. That is, you might have a cluster with 5 nodes, but this just means that all sessions whose session ID starts with, say, "0" or "1" are stored on Redis node 1, all sessions which start with "2" or "3" are stored on Redis node 2, and so on. You can't just save the session on Redis node 1, then, immediately, expect it to "magically appear" on all other nodes. Also, you now still don't have automatic failover. If Redis node 1 fails, all sessions where the session ID begins with "0" or "1" are lost -- unless you setup a backup node where you streamed all the data to. So one architecture would be to have n nodes and for each node, have a backup node (slave) which gets used when the master node fails. Of course, there are many more ways to set this up.

With Redis, there seem to be two independent ways of setting up an "intelligent" cluster with load-balancing and failover properties:

- Redis Sentinel

- Redis Cluster

A nice short information about the differences can be found here: https://fnordig.de/2015/06/01/redis-sentinel-and-redis-cluster/ The Redis site also has lots of docs on both solutions.

Personally, I find it a bit strange that they created two independent solutions and are nowhere advocating one over the other. At least from a marketing perspective in my opinion this just worsens things for the prospective customer, but then again, what do I know...

My take on it is that, if I had the need, I would probably try to set up a "Redis Cluster" since it seems like the more automated and more capable solution.

Then, of course there is the possibility to work with "raw" Redis servers and set up some failover script which e.g. changes DNS entries. Presumably, this is what Amazon does with Elasticache (and what I currently use since our application uses Elasticache -- hence the reason why I didn't bother to implement some other method for failover or load-balancing).

Actually, I am not sure what you mean by an "experimental solution". Do you mean MSM, or Redis, or Redis Cluster....?

Actually, MSM with Redis support is currently used in production for dozens of websites with thousands of users a day. MSM currently uses the Jedis library to "talk" to Redis servers. Jedis is a mature software library which is probably used in hundreds or thousands of open source and closed source projects. Jedis has support for both Redis Sentinel and Redis Cluster, but we do not currently expose this support. I am not sure how mature the support for Redis Cluster is (they write, it's still "under development", but I'm not sure if this really means it's unstable or if it means it's just still lacking some would-be-nice-to-have features), then again, this is free software, we just may want to ask them what they would recommend.

Best regards

Markus

[Andreas]

Hi Markus,

Alternatives to MSM: I tried several session managers with redis (chexagon, jcoleman, pivotal), all of them have restrictions which made it impossible to use them in my context. MSM is definitely the most advanced solution out there and (in my opinion) your decision to use MSM was the right one. We additionally have the option to use spring session, which is of course a total different and more “heavy” approach than a plain tomcat session manager.

Failover in Redis: I’m aware that a redis-slave is needed to achieve automatic failover – setting up slaves isn’t a big deal in redis (even slaves of slaves), so this would be fine for me.

Experimental Solution: With “experimental solution” I meant the support of MSM for redis sentinel referring to your question “If I add support for Redis Sentinel, would you be willing to test it thoroughly?”. I understood, you could add support for redis sentinel to MSM, but testing (of your changes in MSM) should be done by me – this is unfortunately not possible at the moment.

Nevertheless – you probably would make many people happy in contributing redis sentinel/cluster support to MSM ;-).

Thx again, Cheers
Andreas

Am Dienstag, 11. Oktober 2016 15:47:19 UTC+2 schrieb Andreas:

[rdp]

Hi Andreas ,

Good day 

I understand this is a old post but I have similar requirement as you . It will be nice to know what solution you implemented finally.

I have successfully implement memcached session manager with memcache nodes in test environments. It works fine with Sticky and Non Sticky configuration

Before we push it to prod I have following concern , I would like to know if you was able to address thm

1. Memcached session library in non sticky configuration selects primary node randomly and backup is created on immediately following node.

    e.g If I have 6 memcache nodes and msm picked n4 as primary node, backup will be created on n5.

   Concern: If due to some reason n4 and n5 crash then user session is lost.

   Expectation : Backup to be created on all other nodes n1, n2, n3, n5 ,n6 . Were you able to achieve this using msm+ redis

2. Using msm you can point to only on redis server , if that server crashes there will be some down time. Were you able to get around this issue

3. If you used msm+redis were you able to configure redis to do real time replication on all nodes.

4. Does msm+redis combination only support Non Sticky configuration .

It will be great if you can share you experience on how you go it right and how the performance is

Regards,