Redis pub/sub latency testing

[Eugene Park]

Hello,

I am looking to use Redis pub/sub as communication layer for a distributed system so I have done some latency testing described below:

http://stackoverflow.com/questions/26319304/redis-of-channels-degrading-latency-how-to-prevent-degradation

Anyone have any insights on how the number of channels affect latency? Why do performance degrade so fast? I would imagine Redis to handle up to 500 channels (1000 connections in 1-to-1 pub/sub) without any degradation

[Felix Gallo]

Are you using one gevent-python process on one core of one machine in order to run this test?

F. 

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Andy McCurdy]

I believe the slowdowns you’re seeing are a result of your threading model. You’re spawning 600 threads (300 producers and 300 consumers). That’s going to create lots of context switching.

I wrote a different version of your test using 1 publisher, 1 consumer and 1000 channels. After sending 1000 messages on each channel, I saw an average latency of 0.0005 seconds per message. The server, pub.py and sub.py were all run on my laptop.

Here’s the code I used: https://gist.github.com/andymccurdy/6111c03e27ba70565d0a

--

[Eugene Park]

Yes, I am. How do I simulate an environment where there are multiple pub-sub communications going on a single machine?

[Eugene Park]

Ah I see. Yes, that's what I was afraid of. I saw your code and it seems to be sending one message at a time.  How do I simulate an environment where there are multiple pub-sub communications going on a single machine and observer latency?

Here are more data. 

	Test 1: 10 channels	Test 2: 50 channels	Test 3: 100 channels	Test 4: 150 channels	Test 5: 200 channels	Test 6: 250 channels	Test 7: 300 channels
Total # of publishes	6000	30000	60000	90000	120000	150000	180000
Latency average	0.001182	0.001910	0.005344	0.002550	0.003421	0.003874	0.005913
Latency median average	0.000747	0.001261	0.001433	0.001643	0.001936	0.002143	0.002377
Latency median median	0.001641	0.001251	0.001374	0.001641	0.001951	0.0021479	0.002360
Max latency	0.138365	0.144210	5.421915	0.138365	0.801732	0.454593	1.904010
Min latency	0.000217	0.000258	0.000259	0.000217	0.000223	0.000214	0.0002192
category:count where     # 0 : latency < .0001 sec     # 1 : .0001 < latency < .0005     # 2 : .0005 < latency < .001     # 3 : .001   < latency < .005     # 4 : .005   < latency < .01     # 5 : .01     < latency < .05     # 6 : .05     < latency < .1     # 7 : .1       < latency < .5     # 8 : .5       < latency < 1     # 9 : 1        < latency < 5     # Z : latency > 5 sec	0:0 1:2385 2:1542 3:1996 4:13 5:61 6:3 7:0 8:0 9:0 Z:0	0:0 1:6205 2:6237 3:16610 4:703 5:215 6:16 7:14 8:0 9:0 Z:0	0:0 1:10058 2:11205 3:35189 4:2682 5:574 6:154 7:64 8:21 9:40 Z:13	0:0 1:13014 2:15121 3:54806 4:5428 5:1361 6:233 7:37 8:0 9:0 Z:0	0:0 1:12588 2:17334 3:76344 4:10891 5:2257 6:268 7:282 8:36 9:0 Z:0	0:0 1:14189 2:19359 3:91665 4:18103 5:5882 6:330 7:472 8:0 9:0 Z:0	0:0 1:14305 2:21086 3:111403 4:24367 5:7813 6:366 7:400 8:39 9:221 Z:0

-         - It looks like the majority of latency falls between 0.001 seconds to 0.005 seconds for any number of channels

-         -  Min latency is very consistent yet max latency is all over the place

-

[Josiah Carlson]

You are running into issues benchmarking because you believe that there could/should be a latency difference between 1 subscribers and 10000 subscribers, and you are seeking out benchmarks to determine how much there is a slowdown. Assuming a single listener on each channel: there is no slowdown going from 1 subscriber to 10000 subscribers or more. Why? As long as you aren't using pattern subscriptions, determining where a message from a publish call will go is an O(1) hash lookup operation, followed by a write to the single subscriber that gets the message. The only huge difference when going from 1 subscriber to 10000 subscribers is usually that that there are more messages being sent to Redis, which means that a command coming in may have a nonzero wait time before being executed in Redis' single-threaded architecture.

If on the other hand, only one publish is ever being executed, then it doesn't really matter how many subscribers you have (except for the additional kernel and server overhead for client information and buffers).

But if you are *really* trying to measure expected latency of pubsub on a loaded server, you only need one publisher, one subscriber, and one load generator. Why? I claim that pubsub behavior is not that much different from most commands, except for the fact that it can send data to one or more unrelated sockets that handled a read (sort of like clients waiting on blocking list pop operations), so measuring latency of one publish/subscribe pair of connections (and many messages), with artificial load on Redis, should be reasonably representative.

You could use::

1. a single subscriber in a single process, no threads for you! (seriously though, don't use Python threads for benchmarking unless you are benchmarking thread performance)

2. a single publisher in a single process, different process from your subscriber, and also not threaded, sending individual messages one at a time, as fast as it can

3. redis-benchmark with varying -n, -c, and -P values to simulate heavy load, using "GET" on a key with a value like what you are already using, which further simulates a similar amount of work that publish/subscribe would actually do (minus the writing to different client)

 - Josiah

[Eugene Park]

>>>The only huge difference when going from 1 subscriber to 10000 subscribers is usually that that there are more messages being sent to Redis, which means that a command coming in may have a nonzero wait time before being executed in Redis' single-threaded architecture.

I think you mean publishers? The more publishers the more message being sent to Redis. The more message Redis have to process. This is what I want to measure; how the load affects latency. 

And for subscribers, I believe the author of Redis says its not the number of subscribers overall but number of subscribers to a channel that will affect latency.

My model for Redis pubsub is n publishers to 1 subscriber per channel. I wanted to know how many publishers I can have.

But i think its fair to say Redis pubsub is extremely low latency. I tested the same way but bumping Redis VM from 2 CPU to 4 CPU (i know cpu doesnt matter for redis), from 4GB RAM to 16GB RAM and it became lightening fast. 

I wonder if i was memory bounded....

Thank you so much for your explanation. It makes sense to me :D

[Josiah Carlson]

On Thu, Oct 16, 2014 at 7:33 AM, Eugene Park <eugen...@gmail.com> wrote:

>>>The only huge difference when going from 1 subscriber to 10000 subscribers is usually that that there are more messages being sent to Redis, which means that a command coming in may have a nonzero wait time before being executed in Redis' single-threaded architecture.

I think you mean publishers? The more publishers the more message being sent to Redis.

No and yes. When you have a lot of subscribers, you usually also have a lot of publishers. But a publisher is just a client that sends messages to subscribers via the "publish" command. You can have 1 publisher send 1 million messages a minute, or you can have 10000 publishers sending 1 message a minute total (or anywhere in between). The 1 publisher sending 1 million messages will cause more overhead in Redis than the 10000 publishers sending 1 message (if you ignore initial connection overhead).

Similarly, 1 subscriber receiving 1 million messages in 1 minute will be more overhead for Redis than 10000 subscribers that receive 1 message in 1 minute.

The more message Redis have to process. This is what I want to measure; how the load affects latency. 

And for subscribers, I believe the author of Redis says its not the number of subscribers overall but number of subscribers to a channel that will affect latency.

Indeed, because the number of subscribers on a channel is the number of memory copies that must occur to get that data to those clients. This is partly why the 'publish' command returns the number of clients that received the data.

My model for Redis pubsub is n publishers to 1 subscriber per channel. I wanted to know how many publishers I can have.

The number of publishers don't matter. It's the number of messages that matter, because after connection, the major overhead for latency and processing in Redis is the number of commands being sent to Redis. Certainly some commands have far larger overhead than other commands, but for publish/subscribe with a single recipient, the volume of publish calls over a time period, as well as inherent network latency, will be the determining factor for latency.

But i think its fair to say Redis pubsub is extremely low latency. I tested the same way but bumping Redis VM from 2 CPU to 4 CPU (i know cpu doesnt matter for redis), from 4GB RAM to 16GB RAM and it became lightening fast.

If you are running on AWS or something similar, this is not surprising, as usually you get better priority, network bandwidth, etc., with the bigger VMs. Also, if you were running your benchmark software on the same machine as Redis, more CPUs = less CPU contention.

I wonder if i was memory bounded....

That is unlikely.

 

Thank you so much for your explanation. It makes sense to me :D

You are welcome :)

 - Josiah