Sidekiq looses all jobs and previous stats. Redis not showing a restart

[Angus Scown]

Sidekiq has been a very reliable part of our system for a few years now.  This week, several times sidekiq has reset all the counters and lost all the scheduled jobs. 

Redis has not restarted. 

I do not see anything in the production rails log or in the sidekiq log specifically that correlates .   

Any ideas on what else I should be looking at to see what's causing the issue.  My first step is to move redis off the same machine as sidekiq in case it's a side effect issue.  

thanks. 

Angus

[Mike Perham]

It's really hard to say and a tough one to debug.  Typically it's some piece of code running "flushall" or similar.

--
You received this message because you are subscribed to the Google Groups "Sidekiq" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sidekiq+u...@googlegroups.com.
To post to this group, send email to sid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/bfd5cd75-6d3d-446e-add2-c225c832347e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mike Perham – CEO, Contributed Systems

Smart, effective open source infrastructure for your apps.

[Angus Scown]

My thoughts initially too but I've had AOF persistence on in redis during one of this instances and no FLUSHALL appear in the aof. I'll keep digging.  

Thanks. 

[Angus Scown]

A huge thanks for the prompt on FLUSHALL.  Sparked a deeper analysis  and found an external process had the ability to flushall on the incorrect redis instance.  Wasn't in AOF logs as they were getting re-written periodically (to reduce size) hence hiding the FLUSHALL.