Redis Cluster at scale

[Nicholas Feinberg]

Hello! I'm looking into switching a large existing Redis fleet to use Redis Cluster, and I'm wondering if there are any known users of Redis Cluster at scale. The largest I could find was Flickr, and their use of Redis seems quite small.

Has anyone used Redis Cluster across 500+ nodes / 10TB+ of data? Are there known performance limitations that would kick in at that scale?

Thanks!

[Benjamin Sergeant]

I used it on a cluster of 10 nodes ... so 2 digits ;p I don't think that count as large fleet.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/3691d7ca-9b7c-40c7-8401-b166d40b1e52n%40googlegroups.com.

[Mehedi Hasan]

Frankly,  with the most popular use cases of Redis as cache, its unusual that normal companies out there will need a cluster of 500 nodes. Its not ElasticSearch. However, FANG gangs may need this, but if they do, I am pretty sure they have already customized Redis/InMemoryCache in their own way.

I definitely hope I will be wrong soon. New use cases brought by Redis in recent times, will help it to grow at that size. That will also need some basic architectural change in Redis Shards and write scalabilities.           

To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/D0FF50AE-18DC-4E16-A0E0-79E58D72D96C%40gmail.com.

[Tuco]

We had used redis cluster in a couple of scenarios for a high traffic adtech stack, the largest cluster being around 140 nodes, with 70 primary and 70 secondaries. Another cluster at the same time was 120 nodes(60 primaries and 60 secondaries). 

The presentation is https://www.youtube.com/watch?v=ITPDIJxnCOI

We were consistently able to get sub-ms latencies. 

Only thing is have appropriate monitoring, alerting setup. Evenly distribute data, ie don't use {} in keys, 

Also, avoid lua scripting, that's my personal opinion, the more lua scripting you use, the more CPU it will use, and considering redis is singlethreaded, it will be a problem.

I cannot overstate the importance of alerting, monitoring, graphs etc. One of the most overlooked portion from info all is the info stats data, and the CPU utilization by command and by redis in total. This will exactly tell you which command is causing the redis to go slow, if it goes slow. When the CPU utilisation goes above a threshold, its time to scale up the redis cluster. 

Also, its important to load test the system.

On Saturday, July 11, 2020 at 10:08:56 AM UTC+5:30, Mehedi Hasan wrote:

Frankly,  with the most popular use cases of Redis as cache, its unusual that normal companies out there will need a cluster of 500 nodes. Its not ElasticSearch. However, FANG gangs may need this, but if they do, I am pretty sure they have already customized Redis/InMemoryCache in their own way.

I definitely hope I will be wrong soon. New use cases brought by Redis in recent times, will help it to grow at that size. That will also need some basic architectural change in Redis Shards and write scalabilities.           

On Sat, Jul 11, 2020, 5:21 AM Benjamin Sergeant <bser...@gmail.com> wrote:

I used it on a cluster of 10 nodes ... so 2 digits ;p I don't think that count as large fleet.

On Jul 10, 2020, at 2:30 PM, Nicholas Feinberg <nich...@liftoff.io> wrote:

Hello! I'm looking into switching a large existing Redis fleet to use Redis Cluster, and I'm wondering if there are any known users of Redis Cluster at scale. The largest I could find was Flickr, and their use of Redis seems quite small.

Has anyone used Redis Cluster across 500+ nodes / 10TB+ of data? Are there known performance limitations that would kick in at that scale?

Thanks!

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.

To unsubscribe from this group and stop receiving emails from it, send an email to redi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/3691d7ca-9b7c-40c7-8401-b166d40b1e52n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.

To unsubscribe from this group and stop receiving emails from it, send an email to redi...@googlegroups.com.

[Tuco]

The presentation is https://www.slideshare.net/RedisLabs/redisconf2017-redis-in-high-traffic-adtech-stack

[Xingbo Wang]

ElastiCache supports up to 250 nodes Redis cluster, according to their document. https://aws.amazon.com/elasticache/redis/?nc=sn&loc=2&dn=1

This blog also talks about how to right sizing your cluster. https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/nodes-select-size.html

To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/68a5f6a5-2af2-4cd8-9fa1-1df4335f9577o%40googlegroups.com.

--

Best Regards,

Xingbo

[Nicholas Feinberg]

That's very helpful. Thanks to everyone very much!

You received this message because you are subscribed to a topic in the Google Groups "Redis DB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/redis-db/xS543fozeuU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to redis-db+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/CAF5Skho4XsjLGSS5HwYFsUjfP3niThS6h8hr1D_GFdOUCWZsBA%40mail.gmail.com.

[Benjamin Sergeant]

If you search on YouTube for redis cluster talk I remember watching 2 at redis conf, one about Box and another one about Twitch, both of them are what I would say large setup.

To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/CACn8%3DFnA9%2BWk_SMP0W0s%3DDPpMtOX5eox12h66mQmWUfqt-172Q%40mail.gmail.com.

[Greg Andrews]

Perhaps you're already aware of this, but just in case:  The term "Redis cluster" has two meanings:

1. Redis Cluster is the name given to the sharding configuration of Redis.  It has 3 or more master instances, and the keys are divided ("sharded") among them.  Each master instance has one or more replica instances to take over if/when that master fails.  The allocation of keys to different masters, and replicas being promoted to replace failed masters, are both handled automatically by the Redis code.  Redis Sentinel is not used for failover.
   
2. A Redis cluster is a term many people use to refer to a group of Redis instances that are not using the sharding configuration.  This is a "traditional" group of machines with a single master instance containing all of the keys, and one or more replica instances (also containing all the keys).  There is no sharding - each master and replica has all the keys managed by the group of servers.  Redis Sentinel is often used for failover from a broken master to one of the replicas, but people can use other software instead of Sentinel if they want.
   

Your question sounded like you're interested in implementing the sharded configuration of Redis on a large scale.  Because some people use the term "cluster" to mean a group of Redis instances without sharding, some of the info suggested for you might not be for the sharded config you have in mind.  Just FYI.

On Saturday, July 11, 2020 at 11:12:42 PM UTC-7, Nicholas Feinberg wrote:

That's very helpful. Thanks to everyone very much!

To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/68a5f6a5-2af2-4cd8-9fa1-1df4335f9577o%40googlegroups.com.

--

Best Regards,

Xingbo

--
You received this message because you are subscribed to a topic in the Google Groups "Redis DB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/redis-db/xS543fozeuU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to redi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/CAF5Skho4XsjLGSS5HwYFsUjfP3niThS6h8hr1D_GFdOUCWZsBA%40mail.gmail.com.

[Nicholas Feinberg]

Yes, thank you for the clarification, Greg! I was in fact referring to Redis Cluster with a capital C, not to a Redis cluster (which we already run). Apologies for any misunderstanding!

To unsubscribe from this group and all its topics, send an email to redis-db+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/195a8b0e-7bc9-4ef4-b28c-a07d8c85b13bo%40googlegroups.com.

[Benjamin Sergeant]

Redis Use in the Real World

• Some Facebook online games have a very high number of score updates. Executing these operations is trivial when using a Redis sorted set, even if there are millions of users and millions of new scores per minute.
• Twitter stores the timeline for all users within a Redis cluster.
• Pinterest stores the user follower graphs in a Redis cluster where data is sharded across hundreds of instances.
• Github uses Redis as a queue.

I found this in a 'rancher' tutorial / I couldn't find any other references so I don't know how much is true.

To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/CACn8%3DFnZE-_R%3DRbY44a0tF%2BO9WSKixDRPonVk0WewP2vzzmQgg%40mail.gmail.com.