Sidekiq Running the Same Job Twice??

[mi...@pillpack.com]

I discovered a very strange problem today when I noticed that Sidekiq had one of my jobs twice on two different servers. We've run millions of jobs and never seen this before.

We have 6 sidekiq machines, each with 5 workers. One one machine, I see the job enqueued once (and only once):

[ActiveJob] [ProcessClaimResponse] [79f5071a-05b6-4f4b-a16a-4a032e087cdf] Enqueued Claims::SendClaim (Job ID: 37612ffb-e3d6-4b1b-9909-6750ea02a5ba) to Sidekiq(high) with arguments: "584055cdb2d91a15950a7dc7", "BillingClaim", "default", ... etc (I cut out the other parameters here)

And then I see it run on two different machines at exactly the same time.

First, on one machine:

[ActiveJob] [Claims::SendClaim] [37612ffb-e3d6-4b1b-9909-6750ea02a5ba] Performing Claims::SendClaim from Sidekiq(high) with arguments: "584055cdb2d91a15950a7dc7", "BillingClaim", "default", etc.

...

[ActiveJob] [Claims::SendClaim] [37612ffb-e3d6-4b1b-9909-6750ea02a5ba] Performed Claims::SendClaim from Sidekiq(high) in 1115.38ms

And on another machine:

[ActiveJob] [Claims::SendClaim] [37612ffb-e3d6-4b1b-9909-6750ea02a5ba] Performing Claims::SendClaim from Sidekiq(high) with arguments: "584055cdb2d91a15950a7dc7", "BillingClaim", "default", etc.

...

[ActiveJob] [Claims::SendClaim] [37612ffb-e3d6-4b1b-9909-6750ea02a5ba] Performed Claims::SendClaim from Sidekiq(high) in 116.65ms

As you can see, the job with ID 37612ffb-e3d6-4b1b-9909-6750ea02a5ba appears to have been run twice!

Both logs have a "Performed" message so I am assuming that means that the job ran to completion on both machines and did not fail on one and retry on the other. The database timestamps (the jobs write to a postgres db) are identical (within 1 second) for both jobs. 

I'm not sure if this is an issue with Sidekiq, with Redis, or with ActiveJob. 

I'm at a loss. Does anyone have any ideas as to what could be going on?