Node.js and Redis Cluster (using pub/sub)

[Guus Goossens]

Hello,

We are building a big new system on Redis. Messages are put in a queue (list) and answers are published on their message ID.

Only 1 instance is used (with 3 extra slaves with redis-sentinel for fail over). But everything is stored and published on 1 master. Because we are not sure how to scale pub/sub other then using more instances and connecting from our applications to all instances and publishing/subscribing on them. This does not look like a good solution.

Redis cluster should take care of this as far as i know. My questions is: do i need a new a different client (using https://github.com/mranney/node_redis) that can handle Redis Cluster? I think i read that i do but i cant find any (that seem production ready).

How are people doing this? any other recommendations? In the current situation when the master goes down message will not be delivered/lost until a new master has been chosen and the apps update their connection.

Thanks in advance!

Guus

[Josiah Carlson]

On Tue, Dec 2, 2014 at 2:59 AM, Guus Goossens <gu...@q42.nl> wrote:

Hello,

We are building a big new system on Redis. Messages are put in a queue (list) and answers are published on their message ID.

Only 1 instance is used (with 3 extra slaves with redis-sentinel for fail over). But everything is stored and published on 1 master. Because we are not sure how to scale pub/sub other then using more instances and connecting from our applications to all instances and publishing/subscribing on them. This does not look like a good solution.

Whether or not it is a good solution will depend on a lot of factors that you have not provided. The biggest problem you will have with what your current design is that every write/publish to the master is broadcast to 3 slaves. So for every 1 byte in, you are sending out 3+ bytes. If your network and master are fast enough, this probably isn't a big deal, but that will depend on your total write volume.

Note that if your write volume is on the order of <10k messages/second, your current design is fine. Heck, I'd even suggest switching to lists and just use BLPOP operations on the master instead of a subscribe on a slave, at least then messages would generally not be dropped (especially if you have a min-slaves-to-write line in your config setting).

Redis cluster should take care of this as far as i know.

It will take care of the publish broadcast to all cluster nodes, and it will generally handle failover, but you may still lose your published message during a node tailure/reconnection process.

 

My questions is: do i need a new a different client (using https://github.com/mranney/node_redis) that can handle Redis Cluster? I think i read that i do but i cant find any (that seem production ready).

I'm not a Node guy, so many folks will probably do better at answering this question.

How are people doing this? any other recommendations? In the current situation when the master goes down message will not be delivered/lost until a new master has been chosen and the apps update their connection.

I use lists when I care about the response, and publish/subscribe when I'm okay with losing data. If you're not okay with losing data, don't use publish/subscribe.

 - Josiah

Thanks in advance!

Guus

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Pawel]

 

My questions is: do i need a new a different client (using https://github.com/mranney/node_redis) that can handle Redis Cluster? I think i read that i do but i cant find any (that seem production ready).

I'm not a Node guy, so many folks will probably do better at answering this question.

If you cluster with Sentinel, you can use https://www.npmjs.org/package/redis-sentinel, it will in turn use https://www.npmjs.org/package/redis (which is same as https://github.com/mranney/node_redis).

[Guus Goossens]

Thanks for the reply, Josiah and Pawel.

Sorry for my late reply, somehow i did not get the email updates from this group... :-/ (the checkbox is on)

Pawel: This is what we are using now. But my understanding is that it will not work for Redis Cluster. Am i right? Can i not use Redis Cluster without a client that specifically supports this?

Josiah:

Messages can be lost in the scenario of a failure. However the system should recover and minimize downtime. Any messages lost during downtime do not have to be recovered.

My doubt is that we are using a single instance of Redis. So if that goes down the whole system stops working (single point of failure).

So how do you scale Pub/Sub?

And are you saying i should be careful using 3 slaves and not using them because this generates more traffic?

Thanks for the help! we really appreciate it!

[Josiah Carlson]

On Fri, Dec 5, 2014 at 1:43 AM, Guus Goossens <gu...@q42.nl> wrote:

Thanks for the reply, Josiah and Pawel.

Sorry for my late reply, somehow i did not get the email updates from this group... :-/ (the checkbox is on)

Pawel: This is what we are using now. But my understanding is that it will not work for Redis Cluster. Am i right? Can i not use Redis Cluster without a client that specifically supports this?

Josiah:

Messages can be lost in the scenario of a failure. However the system should recover and minimize downtime. Any messages lost during downtime do not have to be recovered.

These are your requirements, and pubsub with Redis fits these requirements. Great!

 

My doubt is that we are using a single instance of Redis. So if that goes down the whole system stops working (single point of failure).

If you are using your master with 3 slaves, this is only the case during failure/failover.

 

So how do you scale Pub/Sub?

You don't have a problem of scaling pubsub, you have a problem of scaling a task processing system that uses pubsub to report results back to a requesting client. And going deeper, your actual problem is that you have all of your task items and results flowing through a single Redis master. If you want to actually scale this, you need to partition your tasks and results reporting. That's it. Find some way to partition your work items to multiple Redis shards, either have your workers request items from multiple shards, or run a worker for each shard on your task processor.

If you are worried about HA, run 1 master and 1 slave for each shard.

And are you saying i should be careful using 3 slaves and not using them because this generates more traffic?

If you are okay losing data during failure as a result of pubsub being what it is, then having 3 slaves is about 2 slaves too many. Stick with 1 master and 1 slave per shard. If possible, build a piece that automatically starts up a replacement server in case of server failure. If you can't, and you need it to work at 3AM while you are sleeping, then 2 slaves is fine.

By sticking with master/slave instead of master/3 slaves, you can run twice as many shards for the same money, getting twice as much total throughput.

 - Josiah

[Guus Goossens]

Your input is very helpful, Josiah. Thanks a lot!

We thought, to reduce the nr of instances, that we could run the sentinels and redis programs on the same server. And we understood that you need at least 2 or 3 sentinels for it to function correctly.

Our setup now is this: (4 servers in 2 zones)

       Zone A     |      Zone B

------------------------------------

Master + sentinel | Slave + sentinel

Slave  + sentinel | Slave + sentinel

Does this make sense? This way if zone A fails entirely Zone B has 2 sentinels and can agree to make 1 of the 2 remaining slaves the new master.

We do need it to work at 3AM when we are sleeping :)

So onto sharding; my understanding is that i should have 2 masters and my applications (node.js) should connect to both (more configuration and code).

Then they can use something like crc32 to determine what key/event goes to what server.

That way only 50% of my message should fail at the same time, unless both masters go down of course.

I've read a bunch of articles on this online but it still sounds pretty complicated, requiring a lot of configuration, code and places for bugs.

If you have the time to help us with this but we need to take this into a PM as to not spam everyone with updates thats ok. Let me know.

Cheers,

Guus