Quart.Net is Sometimes Running Overlapping Tasks

[Eric]

I am using Quartz.Net 3.0.7 to manage a scheduler.  In my test environment I have two instances of the scheduler running.  I have a test process that runs for exactly 2 hours before ending.  Quartz is configured to start the process every 10 seconds and I am using the DisallowConcurrentExecution attribute to prevent multiple instances of the task from running at the same time.  80% of the time this is working as expected.  Quartz will start up the process and prevent any other instances of the task from starting until after the initial one has completed.  If I stop one of the two services hosting Quart, then the other instance picks up the task at the next 10 second mark.

 

However, after keeping these two Quartz services running for 48 uninterrupted hours, I have discovered a couple of times where things went horribly wrong.  At times host B will start up the task, even though the task is still in the middle of its 2 hour execution on host A.  At one point I even found the process had started up 3 times on host B, all within a 10 minute period.  So, for a two hour period, the one task had three instances running simultaneously.  After all three finished, Quartz went back to the expected schedule of only having one instance running at a time.

 

If these overlapping tasks were happening 100% of the time, I would think there is something wrong on my end, but since it seems to happen only about 20% of the time, I am thinking it must be something in the Quartz implementation.  Is this by design or is it a bug?  If there is an event I can capture from Quart.Net to tell me that another instance of a task has started up, I can listen for that and stop the existing task from running.  I just need to make sure that DisallowConcurrentExecution is getting obeyed and prevent a task from running multiple instances concurrently.  Thanks.

 

[Eric]

Edit:

I added logic that uses context.Scheduler.GetCurrentlyExecutingJobs to look for any jobs that have the same JobDetail.Key but a different FireInstanceId when my task starts up.  If I find another currently executing job, I will prevent this instance from doing anything.  I am finding that in the duplicate concurrent scenario, Quartz is reporting that there are no other jobs currently executing with the same JobDetail.Key.  Should that be possible?  Under what case would Quartz.Net start an IJob, lose track of it as an executing job after a few minutes, but allow it to continue executing without cancelling the CancellationToken? 

[Eric]

Edit2:

I found an instance in my logs where Quartz started a task as expected.  Then, one minute later, Quartz tried to start up 9 additional instances, each with a different FireInstanceId.  My custom code blocked the 9 additional instances, because it can see that the original instance was still going, by calling GetCurrentlyExecutingJobs to get a list of running jobs.  I double checked and the ConcurrentExecutionDisallowed flag is true on all of the tasks at runtime, so I would expect that Quartz would prevent the duplicate instances.  This sounds like a bug.  Am I expected to handle this manually or should I expect Quartz to get this right?

[Eric]

Edit3:

I am definitely looking at two different problems.  In both cases Quartz.Net is launching my IJob instance with a new FireInstanceId while there is already another FireInstanceId running for the same JobKey.  In one scenario I can see that both FireInstanceIds are active by calling GetCurrentlyExecutingJobs.  In the second scenario calling GetCurrentlyExecutingJobs shows that the first FireInstanceId is no longer running, even though I can see from my logs that the original instance is still running.  Both of these scenarios result in multiple instance of my IJob running at the same time, which is not acceptable.  It is easy enough to tackle the first scenario by calling GetCurrentlyExecutingJobs when my IJob starts, but the second scenario is harder.  I will have to ping GetCurrentlyExecutingJobs on an interval and stop the task if it’s FireInstanceId has disappeared from the active list.  Has anyone else really not noticed this behavior?

[Eric]

Edit 4:

I have an IJob implementation that takes 2 hours to complete and it is flagged as DisallowConcurrentExecution = true.  9 out of 10 times my job runs its course and no more instances are created until the original one finishes, but about 1 out of 10 times, an extra instance of my IJob gets started before the last one finished.  When things are working properly, I can see my FireInstanceId in FIRED_TRIGGERS for the entire 2 hours.  But when there are multiple tasks running, I see that only one of them is listed in FIRED_TRIGGERS.  So, what would cause a job to stop showing up in FIRED_TRIGGERS, and when it does disappear from that table, why is the cancellation token from the job context not getting cancelled?  If the token would just cancel, then I could stop my job myself.

[Eric]

I found that if I set this option, that I no longer have overlapping executing jobs.  I still wish that Quartz would cancel the job’s cancellation token, though, if it lost track of the executing job.

 

QuartzProperties.Add("quartz.jobStore.clusterCheckinInterval", "60000");