Redis cluster linear scale solution

[Yuhe Chen]

I performed a scalability test of redis. I run three tests as listed below. In each test, I push the redis-server process cpu to 85% and evaluate the total OPS (operation per seconds) the client can send.

1 redis-server instance: 350,000 OPS
3 shard cluster: 540,000 OPS
3 redis-server instances: 1,000,000 OPS

First test use a single redis-server process. The OPS is pretty impressive. I have high performance server with 40 CPU cores and 100 GB RAM. Bandwidth is also not a problem.

The second test use a 3 shards cluster (3 redis-server processes on the same server). I expect to see the OPS triples when run 3 redis-server instance. But it's far less than that. I turned of AOF and Disc write. There is no slave shards in the cluster. Could not figure out what else prevent the linear scaling.

The third test is that I start 3 redis-server instance without joint them into a cluster. The client perform hashing on the key and dispatch command to one of the redis-server instance. The overall OPS is close to triple the single redis-server OPS. It's much better than in a cluster.

Did anybody else notice the performance issue of open source redis cluster? It should scale linearly according to the online documentation. That means, if you have N redis-server in a cluster, then you should get N times the capacity.

Is there any configuration parameter that I can tweak to improve the OPS?

[Yuhe Chen]

The redis open source cluster provide very nice feature which can be useful to achieve elasticity. It can add/remove nodes in the cluster and do resharding (rebalancing). I really would like to have a cluster that scale linearly or a least close to that.

Running independent redis-server instances can scare. However, it requires to much effort to achieve what redis cluster provided. For instance, when I find out existing server can not handle increase traffic. Then I have to add new redis-server instances. I have to reconfigure my clients and restart them as some keys will have to go the the newly added redis-servers.

I know the enterprise version cluster from redislabs might achieve both linear scaling and elasticity. What other options do I have if I would like to stick to the open source redis?

[AlexanderB]

Hi Yuhe, 

While redis-cluster itself should scale in a roughly linear fashion as more nodes are added up to a point, the redis-cluster client implementations are a bit more immature, and additional there is a bit more overhead talking to a redis-cluster. 

What I would suggest doing is to run another few tests on the redis-cluster, testing with 6, or even 12 nodes. I would predict that you'll see roughly linear performance increases as compared to the existing redis-cluster 3 node cluster, and it's just the additional redis-cluster overhead and/or a less optimized code path in the client for redis-cluster this is accounting for the difference more so than a lacking of linearity. 

[ma...@andyh.io]

The reason(probably) you cannot see linear scalability is you are getting REDIRECT from cluster since the benchmark tool does not send the key to the node that is responsible for the slot.

I suggest you test it with a client that has cluster support.

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Yuhe Chen]

Hi, Alexander, Hi, Andy

I appreciate the inputs from both of you. The client I use support cluster and asynchronous socket. It can read the hash slots configuration from cluster and dispatch the redis command accordingly. So REDIRECT does not happen in my test.

I also tested different number of shards in the cluster (ignore the numbers on the Y axis. the actual redis OPS is x18 times of the number). The chart is quite flat.

I understand there are communication between shards in the cluster. But I suppose they should be minimal as long as there is no failover or rebalancing happening.

What else do you think that may cause the overhead and can be turned off?

[andyh]

I still don't think the server side is the issue. The overhead of sending requests to the server also needs investigation. From what my experience, send requests to all the nodes concurrently might be the most difficult part in the client implementation, especially using pipeline.

If you still question Redis Cluster, what you can do is test the servers with the methodology you used in the test3 (there should not be any difference) and compare the numbers. 

Oh, I just noticed that you did not mention the CPU usage at the server side in your second test case, if the CPU usage is high, which means Redis Cluster does have some overhead and affecting the performance. But if it's the other way around, it means the requests did not reached to the server side. 

--

You received this message because you are subscribed to the Google Groups "Redis DB" group.

To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe@googlegroups.com.

To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

--

andyh

Andy Huang (Huangkejun)

[Yuhe Chen]

The cpu usage of each redis-server process is about 85% for each test. The C++ client support redis cluster and works in asynchronous socket mode. It's pretty fast. By using multi-thread and multiple redis connection, I can generate enough redis operations to flood any number of redis instances (subject to the bandwidth limits).  

I also tested 6 redis-server instance without cluster enabled.  The OPS of 1, 3 and 6 instances are close to linear scaling without cluster mode enabled. If you compare it with the second graph (cluster mode), then you notice the big difference of the performance.

I know that most people use redis as cache or in-memory DB. The OPS is not big focus of their application. By using redis cluster or multiple instances on multiple servers, then are able to scale in terms of the amount of data stored. But they may not worry if the OPS scales linearly.

My application instead rely on the high OPS scaling. t's like an realtime data analytic ETL tool. For event collected, it for instance request data from redis and do some enrichment on the event record. It's a real-time processing will handle every high event rate, e.g 1 million events per seconds. I do not know if there are many other users use redis in this way.

I know some big companies do not use redis opensource cluster. But instead do they own sharding and cluster management. Will be curious to know who are really using the open source cluster to do high performance processing.

On Thursday, June 8, 2017 at 10:03:25 AM UTC-4, iandyh wrote:

I still don't think the server side is the issue. The overhead of sending requests to the server also needs investigation. From what my experience, send requests to all the nodes concurrently might be the most difficult part in the client implementation, especially using pipeline.

If you still question Redis Cluster, what you can do is test the servers with the methodology you used in the test3 (there should not be any difference) and compare the numbers. 

Oh, I just noticed that you did not mention the CPU usage at the server side in your second test case, if the CPU usage is high, which means Redis Cluster does have some overhead and affecting the performance. But if it's the other way around, it means the requests did not reached to the server side. 

On Thu, Jun 8, 2017 at 10:28 PM, Yuhe Chen <yhche...@gmail.com> wrote:

Hi, Alexander, Hi, Andy

I appreciate the inputs from both of you. The client I use support cluster and asynchronous socket. It can read the hash slots configuration from cluster and dispatch the redis command accordingly. So REDIRECT does not happen in my test.

I also tested different number of shards in the cluster (ignore the numbers on the Y axis. the actual redis OPS is x18 times of the number). The chart is quite flat.

I understand there are communication between shards in the cluster. But I suppose they should be minimal as long as there is no failover or rebalancing happening.

What else do you think that may cause the overhead and can be turned off?

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.

To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.

To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Yuhe Chen]

BTW, my team is hiring now. A great team and the project. It will fit you if you likes technical challenges. It's an open environment and you will get a lot of opportunities to contribute to individual components and the architect.

We are looking for engineers with experience on big data analytic (hadoop, flink, etc.), distributed system design, system scaling, NFV, database, networking and high performance realtime system design.

The position is open in both Great Boston area and Italy office.

http://www.empirix.com/about-empirix/careers/?gh_jid=656452

[Salvatore Sanfilippo]

Hello, just to say that I replied in the Github issue. Cheers.

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at https://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.



--

Salvatore 'antirez' Sanfilippo
open source developer - Redis Labs https://redislabs.com

"If a system is to have conceptual integrity, someone must control the
concepts."
— Fred Brooks, "The Mythical Man-Month", 1975.

[Tuco]

Hi Yuhe, 

we are using the open source redis cluster, at scale. The data size is around 2 TB master data, with 150+ master nodes, and replication factor of 1.

we feel pretty confident with redis cluster.

although we dont work with million OPS, but one of our clusters had reached ~2 million QPS with pipelining.

A few things which i would suggest.

1) Use a cluster aware client (we use Jedis/Lettuce in java)

2) Some of the cluster commands are slow(we had ~60 master nodes in one cluster and the slowest command was cluster nodes taking around 3 ms, which was used by Lettuce to continuously refresh the network topology). We ended up asking Lettuce to refresh its cache with cluster topology every 2 hrs or in case it gets a moved or ask redirection event. Make sure your client does not hit commands like "cluster nodes" or "cluster slots" frequently enough.

3) we had found out that more often than not, the number of clients was the issue. The cluster ops almost linearly increased with increase in clients(although for us the clients were less).

4) keep a note on the "info  cpu". If you do only the basic operations and no processing(no lua), then ideally you are fully utilizing the redis instances only if "used_cpu_sys" + "used_cpu_user" is equal to N in an N second interval. we have N as 300 as we monitor and put the data on graph every 300 secs. Most probably, your CPUs will not be utilized, and the issue is not redis but the network, or most probably your clients which are not fast enough.

5) You could also try to run redis instances in a cluster on separate machines with good connectivity.

6) Also, disable automatic save if running multiple instances on same machine(so that not all instances start saving at the same time and slow down redis), instead do a BGSAVE for each instance one by one and waiting for a few seconds after each save(but i think you already disabled the saves)

I had presented our use case where we use open source redis cluster with our dataset and multiple stacks in the redis conf 2017. The presentation is here. 

[Salvatore Sanfilippo]

Hello Tuco,

about the following point:

> 6) Also, disable automatic save if running multiple instances on same
> machine(so that not all instances start saving at the same time and slow
> down redis), instead do a BGSAVE for each instance one by one and waiting
> for a few seconds after each save(but i think you already disabled the
> saves)

What about making Redis aware of other instances running in the same
host via a lock file or something like that, so that every instance
delays BGSAVE / AOF rewrites if there is already an active process
doing it?

Cheers,
Salvatore

[Tuco]

Hi Salvatore, 

right, it can be done using that, probably its the right way....

we have an idea of approx how long an rdb takes to save..so we have the shell script to save sleep for that time before saving the next instance.

Thanks

Tuco

[Salvatore Sanfilippo]

Yep... I know the scenario, it is often used to trigger AOF rewrites
via cron or alike :-)
I opened an issue about adding this feature to Redis:
https://github.com/antirez/redis/issues/4055

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at https://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.