What's a good way to persist metadata across a job retry lifecycle?

[James Hu]

Hi!

I'm looking to track the number of times an exception is raised across Sidekiq retries for the purposes of throttling error reporting to my exception tracker until the exception has been raised enough times. 

For example, I work with lots of API calls and I want to reduce the noise of my timeout errors by only reporting the exception after it's tried 5 times. However, I can't simply just rely on `retry_count` since another exception could've been raised 5 times before and then for the 6th time if the timeout error is raised, it would be a false positive.

Is there a recommended way to persist a "session id" of some sorts across job retries? I can track the number of times an exception has been raised in Redis but I'm finding it hard to track by an ID of some sort that persists across the lifecycle of the retries. 

Thanks so much for Sidekiq!

[Mike Perham]

That's a really tricky question, no one's ever asked that one. You could use Redis to store the unique set of error messages associated with a JID that have already been reported to your error service. That's effectively a session for the given JID.

--
You received this message because you are subscribed to the Google Groups "Sidekiq" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sidekiq+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/7c599b04-f5bd-43b0-be29-e13e06ed49adn%40googlegroups.com.

--

Mike Perham – CEO, Contributed Systems

Smart, effective open source infrastructure for your apps.

[James Hu]

Thanks for the quick response! 

I'm still having a hard time trying to wrap my head around connecting these JIDs together. From my tests I can see that there's new JID generated for each job retry. Just to clarify, assuming I want to report to my exception handler on the 5th time Timeout::Error is raised:

{ "jid": "000", "error_class": "Timeout::Error", "retry_count": 1 }  => ignore since 1st time

{ "jid": "111", "error_class": "Timeout::Error", "retry_count": 2 }  => ignore since 2nd time

{ "jid": "222", "error_class": "Timeout::Error", "retry_count": 3 }  => ignore since 3rd time

{ "jid": "333", "error_class": "Sidekiq::Shutdown", "retry_count": 4 }  => ignore since not Timeout:Error

{ "jid": "444", "error_class": "Sidekiq::Shutdown", "retry_count": 5 }  => ignore since not Timeout:Error

{ "jid": "555", "error_class": "Sidekiq::Shutdown", "retry_count": 6 }  => ignore since not Timeout:Error

{ "jid": "666", "error_class": "Timeout::Error", "retry_count": 7 }  => ignore since 4th time

{ "jid": "777", "error_class": "Timeout::Error", "retry_count": 8 }  => report since 5th time

or am I mistaken that JID is actually persisted somehow across these retries?

[Mike Perham]

A Sidekiq::Worker job which retries should have the same JID each time, as it's the same job. Are you using ActiveJob? It does not persist the same JID across retries.

To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/d6f0ac5c-5917-44cf-a428-2ed719329284n%40googlegroups.com.