Sidekiq Client error handling

28 views
Skip to first unread message

Abhishek D

unread,
Apr 13, 2016, 5:14:01 AM4/13/16
to Sidekiq
I would like to know if there is a way to rescue redis connection errors, so that I can use a alternative datastore to push jobs. We use client side middleware to set current thread user etc, so rescuing .perform_async wouldn't be of much help. Also, Client side middleware doesn't seem to support error handling. Is there a workaround somehow that I can rescue for redis connection errors inside client middleware? The approach that I could think of was implement a heartbeat to redis, set a global flag to push jobs to an alternative datastore inside middleware. Any help is appreciated.

Thanks.

Mike Perham

unread,
Apr 13, 2016, 9:24:51 AM4/13/16
to sid...@googlegroups.com
Sidekiq Pro's reliable_push feature does essentially what you are asking - it rescues redis errors and saves the payload in-memory.

You can do everything you want with some patching of the Client API.  You are right that the client middleware doesn't "wrap" the Redis call so you can't use middleware for the error handling.

On Wed, Apr 13, 2016 at 2:14 AM, Abhishek D <natraj....@gmail.com> wrote:
I would like to know if there is a way to rescue redis connection errors, so that I can use a alternative datastore to push jobs. We use client side middleware to set current thread user etc, so rescuing .perform_async wouldn't be of much help. Also, Client side middleware doesn't seem to support error handling. Is there a workaround somehow that I can rescue for redis connection errors inside client middleware? The approach that I could think of was implement a heartbeat to redis, set a global flag to push jobs to an alternative datastore inside middleware. Any help is appreciated.

Thanks.

--
You received this message because you are subscribed to the Google Groups "Sidekiq" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sidekiq+u...@googlegroups.com.
To post to this group, send email to sid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/78d43738-64e0-4b4c-bce1-65c266fe294b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Mike Perham – CEO, Contributed Systems
Smart, effective open source infrastructure for your apps.

Abhishek D

unread,
Apr 18, 2016, 5:13:58 AM4/18/16
to Sidekiq
Thanks for getting back, I apologize for the late reply here. I looked at reliable_push feature to find out that it supports only 10,000 jobs. Our throughput is even higher than that for 5 seconds. We have an enterprise version. I will connect with you over email

Mike Perham

unread,
Apr 18, 2016, 11:45:31 AM4/18/16
to sid...@googlegroups.com
That 10,000 limit is *per client process* and can be changed by updating the constant in your initializer.  I assume you don't have a single puma process pushing 1000 jobs/sec.  If you have 50 puma processes, you have, in aggregate, a 500,000 job buffer.


For more options, visit https://groups.google.com/d/optout.

Abhishek D

unread,
Apr 18, 2016, 12:02:50 PM4/18/16
to Sidekiq
Thanks for information here, I'll check that.
Reply all
Reply to author
Forward
0 new messages