Job Scheduler "Stuck", due to "An error occured while marking executed job complete"

Gary German

unread,

Jan 10, 2011, 1:18:19 PM1/10/11

to quar...@googlegroups.com

Recently, a few our Quartz instances have become "stuck", with no new jobs getting downloaded or executed. It looks like this is due to an internal update in the Scheduler that is failing (see exception info below).

We first tried stopping and restarting our Windows Service (which stops and then restarts the Quartz.Net scheduler). But the exceptions continued, with the scheduler still throwing exceptions and "stuck" from our point of view.

Our only solution was to delete all the records in our Quartz_ tables. This freed up whatever was repeatedly blocking.

Since doing this, our servers have been stable, with no further problems. But I'm worried that this is a "data issue" that could come back to haunt us.

Any ideas?

Here's the exception we captured:

JobSchedulerListener says: Scheduler Error.

Message: An error occured while marking executed job complete (will continue attempts). job= 'admin.PhoneHomeForJobs'

Exception: at Quartz.Impl.AdoJobStore.UpdateLockRowSemaphore.ExecuteSQL(ConnectionAndTransactionHolder conn, String lockName, String expandedSQL)

at Quartz.Impl.AdoJobStore.DBSemaphore.ObtainLock(DbMetadata metadata, ConnectionAndTransactionHolder conn, String lockName)

at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock(String lockName, ITransactionCallback txCallback)

at Quartz.Impl.AdoJobStore.JobStoreSupport.TriggeredJobComplete(SchedulingContext ctxt, Trigger trigger, JobDetail jobDetail, SchedulerInstruction triggerInstCode)

at Quartz.Core.QuartzScheduler.NotifyJobStoreJobComplete(SchedulingContext ctxt, Trigger trigger, JobDetail detail, SchedulerInstruction instCode)

at Quartz.Core.JobRunShell.CompleteTriggerRetryLoop(Trigger trigger, JobDetail jobDetail, SchedulerInstruction instCode)

Marko Lahma

unread,

Jan 10, 2011, 1:44:43 PM1/10/11

to quar...@googlegroups.com

Hi,

you don't give us much to go on. What's the internal exception, the
root cause of update failure? Are you sure that you service restart
properly kill the whole process? What database, which Quartz.NET
version? How many jobs and triggers? You are running Quartz.NET
clustered?

-Marko

> --
> You received this message because you are subscribed to the Google Groups
> "Quartz.NET" group.
> To post to this group, send email to quar...@googlegroups.com.
> To unsubscribe from this group, send email to
> quartznet+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/quartznet?hl=en.
>

Gary German

unread,

Jan 10, 2011, 3:48:16 PM1/10/11

to quar...@googlegroups.com

Re: "What's the internal exception, the root cause of update failure? "

It looks like the root cause is some sort of timeout (see more detailed
event logs below).

Exception: Failure obtaining db row lock: Timeout expired. The timeout
period elapsed prior to completion of the operation or the server is not
responding.
The statement has been terminated.

But, this exception never seems to clear itself up either. Once it starts
happening, our only recourse seems to be to manually delete all entries in
our Quartz_ tables. That seems to clear up the blockage.

Re: "Are you sure that you service restart properly kill the whole process?"

The request to stop the service processed OK, as did the request to start
the service. The scheduler gets shut down during the stop procedure, and
started during the start procedure. Those both seemed to go OK, but after
restart the problem persisted.

Re: What database, which Quartz.NET version?

SQL Server 2008.

Quartz dll version 1.0.2.2 (upgrading to 1.0.3 shortly).

Re: How many jobs and triggers?

Hardly any - perhaps one repeating job, and maybe one or two other jobs.
Scheduler is set up to allow 5 threads.

Re: You are running Quartz.NET clustered?

No. Single instances, and we feed jobs to them via a "PhoneHomeForJobs" job
(it's a job that phones home via a web service call).

Event log entries from when the service was failing:

JobSchedulerListener says: Scheduler Error. Message: An error occured
while marking executed job complete (will continue attempts). job=
'admin.PhoneHomeForJobs'
Exception: at
Quartz.Impl.AdoJobStore.UpdateLockRowSemaphore.ExecuteSQL(ConnectionAndTrans

actionHolder conn, String lockName, String expandedSQL)

at Quartz.Impl.AdoJobStore.DBSemaphore.ObtainLock(DbMetadata metadata,
ConnectionAndTransactionHolder conn, String lockName)
at
Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock(String
lockName, ITransactionCallback txCallback)
at
Quartz.Impl.AdoJobStore.JobStoreSupport.TriggeredJobComplete(SchedulingConte

xt ctxt, Trigger trigger, JobDetail jobDetail, SchedulerInstruction

triggerInstCode)
at
Quartz.Core.QuartzScheduler.NotifyJobStoreJobComplete(SchedulingContext
ctxt, Trigger trigger, JobDetail detail, SchedulerInstruction instCode)
at Quartz.Core.JobRunShell.CompleteTriggerRetryLoop(Trigger trigger,
JobDetail jobDetail, SchedulerInstruction instCode)

'JobSchedulerListener says: Scheduler Error. Message: An error occured

while marking executed job complete (will continue attempts). job=
'admin.PhoneHomeForJobs'

Exception: Failure obtaining db row lock: Timeout expired. The timeout
period elapsed prior to completion of the operation or the server is not
responding.
The statement has been terminated.