Scaling sidekiq horizontally

[Nicholas Wieland]

Hi *, I have a server dedicated to sidekiq + redis and I need to scale out to process more jobs *and* add fault tolerance (if this server goes down I'm dead right now :)

I already know I can connect sidekiq to a pool of redis servers and this is already very good, my questions is: can I have 2 different servers with an instance of sidekiq and a redis each? How would I need to configure redis replication in this case?

What I would like to achieve is having 2 servers processing jobs with sidekiq and be safe if one on these servers go down and just process less jobs.

How can I do this?

TIA,

  ngw

[Mike Perham]

Sidekiq is tied to Redis so Sidekiq's scaling strategy is just that: how do you scale Redis?  You'll find that you hit an upper limit of about 5000 jobs/sec on good hardware.  Large Sidekiq users shard their Redis usage across many Redis servers so they can go beyond that.

You can push jobs to a particular Redis shard by creating a connection pool pointing to each shard and using Sidekiq::Client directly:

client = Sidekiq::Client.new(redis_pool)

client.push(...)

client.push_bulk(...)

etc...

and of course you'll dedicate one or more Sidekiq processes to work on jobs for a given shard.  Each shard is completely independent.

I won't repeat the Redis options, just point you here so you can get them straight from the source:

http://redis.io/documentation

--
You received this message because you are subscribed to the Google Groups "Sidekiq" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sidekiq+u...@googlegroups.com.
To post to this group, send email to sid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/d1138fec-9f4f-47a2-a473-9e1a22051710%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mike Perham – Contributed Systems

Smart, effective open source infrastructure for your apps.

[Nicholas Wieland]

Thanks Mike, this was very helpful.

I managed to distribute redis using redis-sentinel an everything works as expected.

I have one last question: now that I have 2 sidekiqs on 2 different servers, how can I use sidekiq/web to monitor the queue of both sidekiqs?

When I started 2 sidekiqs on the same machine (my mac) I could see the other sidekiq under "processes", but on my staging env I can't see anything.

I'm not entirely sure how to debug this ...

  ngw

[Mike Perham]

You need to have a different Sidekiq::Web process for each shard.  Like the Sidekiq process, Sidekiq::Web only knows about and views a single Redis instance.