Runaway errors using Rollbar's Sidekiq integration

[Mike Pastore]

Hi folks,

We process about 4K jobs/day. We had a situation two weeks ago where a handful of errors related to external connectivity issues (graph.facebook.com and api.rollbar.com were unreachable due to a Linode outage) caused an explosive "echo chamber," leading to approximately 110K job failures on 330K jobs over a two- or three-hour period (Rollbar's config.sidekiq_threshold was set to 3). 

I emailed Rollbar support and they suggested setting a rate limit to prevent going over quota in the future. However, I'm interested in learning how this might be prevented in the first place. Is there a simple way to tell Sidekiq to pause job processing for a period of time if the error rate exceeds a given threshold? Or more generally a way to pause Sidekiq if an external resource (a database, the internet, etc.) is not available?

Thank you in advance,

Mike Pastore

mi...@oobak.org

[Mike Perham]

Hi Mike, generally a system cannot reliably watch itself.  I prefer to leave that monitoring to the application so you'd need to implement something yourself.  You can use the Sidekiq API to check for retry set size and script some response, but the possible inputs and logic are so app-specific; I don't think I could provide anything useful in Sidekiq itself.  Sidekiq Pro has an API to pause a given queue, you could pause all queues like:

Sidekiq::Queue.all.each(&:pause!)

Let us know if you figure out something reliable and useful.

Mike

--
You received this message because you are subscribed to the Google Groups "Sidekiq" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sidekiq+u...@googlegroups.com.
To post to this group, send email to sid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/a3e6fb3c-6c56-4d4d-a52d-a2401cee4296%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mike Perham – CEO, Contributed Systems

Smart, effective open source infrastructure for your apps.