1.0.3.3 - Trigger getting stuck in BLOCKED State
Stephen Tunney
single trigger, and it is the same one every single time, getting
stuck in the BLOCKED state. No other trigger in the system gets
stuck.
My job is a IStatefulJob. Key behaviours of my job include:
-Updating JobDataMap values (strings)
-Creating/Updating/Deleting other Job Schedules using the Quartz API
*only*
Here is the code I use to create the job and trigger, and finally
schedule them in the scheduler, each of the JobDataMap keys has the
[Serializable] attribute, and implements ISerializable properly (fully
unit tested each):
jobDetail = new JobDetail(jobName, JOB_GROUP_NAME,
typeof(Job_SentryJobDatabaseWatcher));
jobDetail.RequestsRecovery = true;
jobDetail.JobDataMap[Job_SentryJobDatabaseWatcher.JOBDETAIL_Sentry_CONNECTION_STRING]
= m_sqlServerConnectionString;
jobDetail.JobDataMap[Job_SentryJobDatabaseWatcher.JOBDETAILS_EMAIL_NOTIFIER]
= m_emailer;
jobDetail.JobDataMap[Job_SentryJobDatabaseWatcher.JOBDETAILS_LOGGER] =
m_logger;
jobDetail.JobDataMap[Job_SentryJobDatabaseWatcher.JOBDETAIL_IOC_CONTAINER]
= m_iocContainer;
trigger = new CronTrigger(triggerName, JOB_GROUP_NAME, @"0 0/2 * ? * *
*");
trigger.MisfireInstruction =
MisfireInstruction.CronTrigger.FireOnceNow;
trigger.Priority = 100;
_scheduler.ScheduleJob(jobDetail, trigger);
Here is the current row in the Triggers table for this malfunctioning
CronTrigger:
TRIGGER_NAME Trigger_SentryDBWatcher
TRIGGER_GROUP DBWatchers
JOB_NAME Job_SentryDBWatcher
JOB_GROUP DBWatchers
IS_VOLATILE 0
DESCRIPTION NULL
NEXT_FIRE_TIME 6.34183E+17
PREV_FIRE_TIME 6.34183E+17
PRIORITY 100
TRIGGER_STATE BLOCKED
TRIGGER_TYPE CRON
START_TIME 6.34183E+17
END_TIME 0
CALENDAR_NAME NULL
MISFIRE_INSTR 1
JOB_DATA NULL
Here is my current configuration, I am connecting to a SQL 2008 Std
Edition (x64) instance:
<quartz>
<add key="quartz.scheduler.instanceName"
value="DefaultQuartzJobScheduler" />
<add key="quartz.scheduler.instanceId" value="AUTO" />
<add key="quartz.jobStore.clustered" value="true" />
<add key="quartz.jobStore.clusterCheckinInterval" value="15000" />
<add key="quartz.threadPool.type"
value="Quartz.Simpl.SimpleThreadPool, Quartz" />
<add key="quartz.jobStore.useProperties" value="false" />
<add key="quartz.jobStore.type"
value="Quartz.Impl.AdoJobStore.JobStoreTX, Quartz" />
<add key="quartz.jobStore.driverDelegateType"
value="Quartz.Impl.AdoJobStore.SqlServerDelegate, Quartz" />
<add key="quartz.jobStore.tablePrefix" value="QRTZ_" />
<add key="quartz.jobStore.lockHandler.type"
value="Quartz.Impl.AdoJobStore.UpdateLockRowSemaphore, Quartz" />
<add key="quartz.jobStore.misfireThreshold" value="120000" />
<add key="quartz.jobStore.dataSource" value="default" />
<add key="quartz.dataSource.default.connectionString" value="Data
Source=QUARTZBOX;Database=Quartz_A;UID=XXXXXX;PWD=XXXXXXXXX;" />
<add key="quartz.dataSource.default.provider" value="SqlServer-20" />
<!-- Customizable values per Node -->
<add key="quartz.threadPool.threadCount" value="2" />
<add key="quartz.threadPool.threadPriority" value="Normal" />
</quartz>
Here is the exception that I am seeing every single time this error
occurs, please note that the previous execution (at 12:40:00) executed
and completed successfully:
2010-08-24 12:42:00,000
[DefaultQuartzJobScheduler_QuartzSchedulerThread] DEBUG
Quartz.Impl.AdoJobStore.UpdateLockRowSemaphore [(null)] - Lock
'TRIGGER_ACCESS' is desired by:
DefaultQuartzJobScheduler_QuartzSchedulerThread
2010-08-24 12:42:00,000
[DefaultQuartzJobScheduler_QuartzSchedulerThread] DEBUG
Quartz.Impl.AdoJobStore.UpdateLockRowSemaphore [(null)] - Lock
'TRIGGER_ACCESS' is being obtained:
DefaultQuartzJobScheduler_QuartzSchedulerThread
2010-08-24 12:42:00,015
[DefaultQuartzJobScheduler_QuartzSchedulerThread] DEBUG
Quartz.Impl.AdoJobStore.UpdateLockRowSemaphore [(null)] - Lock
'TRIGGER_ACCESS' given to:
DefaultQuartzJobScheduler_QuartzSchedulerThread
2010-08-24 12:42:12,875 [QuartzScheduler_DefaultQuartzJobScheduler-
QUARTZBOX_A634182574917031250_MisfireHandler] DEBUG
Quartz.Impl.AdoJobStore.JobStoreTX [(null)] - MisfireHandler: scanning
for misfires...
2010-08-24 12:42:20,062
[DefaultQuartzJobScheduler_QuartzSchedulerThread] ERROR
Quartz.Impl.AdoJobStore.JobStoreTX [(null)] - Couldn't rollback
ADO.NET connection. This SqlTransaction has completed; it is no longer
usable.
System.InvalidOperationException: This SqlTransaction has completed;
it is no longer usable.
at System.Data.SqlClient.SqlTransaction.ZombieCheck()
at System.Data.SqlClient.SqlTransaction.Rollback()
at
Quartz.Impl.AdoJobStore.JobStoreSupport.RollbackConnection(ConnectionAndTransactionHolder
cth) in C:\Users\stunney\Desktop\Share\Quartz.NET-1.0.3\src\Quartz\Impl
\AdoJobStore\JobStoreSupport.cs:line 4031
2010-08-24 12:42:21,406 [QuartzScheduler_DefaultQuartzJobScheduler-
QUARTZBOX_A634182574917031250_ClusterManager] ERROR
Quartz.Impl.AdoJobStore.JobStoreTX [(null)] - Couldn't rollback
ADO.NET connection. This SqlTransaction has completed; it is no longer
usable.
System.InvalidOperationException: This SqlTransaction has completed;
it is no longer usable.
at System.Data.SqlClient.SqlTransaction.ZombieCheck()
at System.Data.SqlClient.SqlTransaction.Rollback()
at
Quartz.Impl.AdoJobStore.JobStoreSupport.RollbackConnection(ConnectionAndTransactionHolder
cth) in C:\Users\stunney\Desktop\Share\Quartz.NET-1.0.3\src\Quartz\Impl
\AdoJobStore\JobStoreSupport.cs:line 4031
2010-08-24 12:42:54,546 [QuartzScheduler_DefaultQuartzJobScheduler-
QUARTZBOX_A634182574917031250_MisfireHandler] DEBUG
Quartz.Impl.AdoJobStore.JobStoreTX [(null)] - Found 0 triggers that
missed their scheduled fire-time.
2010-08-24 12:43:29,546
[DefaultQuartzJobScheduler_QuartzSchedulerThread] DEBUG
Quartz.Impl.AdoJobStore.UpdateLockRowSemaphore [(null)] - Lock
'TRIGGER_ACCESS' returned by:
DefaultQuartzJobScheduler_QuartzSchedulerThread
2010-08-24 12:43:29,562 [QuartzScheduler_DefaultQuartzJobScheduler-
BIZSYS-MONITOR2634182574917031250_ClusterManager] ERROR
Quartz.Impl.AdoJobStore.JobStoreTX [(null)] - ClusterManager: Error
managing cluster: Couldn't commit ADO.NET transaction. Timeout
expired. The timeout period elapsed prior to completion of the
operation or the server is not responding.
Quartz.JobPersistenceException: Couldn't commit ADO.NET transaction.
Timeout expired. The timeout period elapsed prior to completion of
the operation or the server is not responding. --->
System.Data.SqlClient.SqlException: Timeout expired. The timeout
period elapsed prior to completion of the operation or the server is
not responding.
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at
System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject
stateObj, UInt32 error)
at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult
asyncResult, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
at System.Data.SqlClient.TdsParserStateObject.ReadByte()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior,
SqlCommand cmdHandler, SqlDataReader dataStream,
BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject
stateObj)
at
System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[]
buffer, TransactionManagerRequestType request, String transactionName,
TransactionManagerIsolationLevel isoLevel, Int32 timeout,
SqlInternalTransaction transaction, TdsParserStateObject stateObj,
Boolean isDelegateControlRequest)
at
System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest
transactionRequest, String transactionName, IsolationLevel iso,
SqlInternalTransaction internalTransaction, Boolean
isDelegateControlRequest)
at System.Data.SqlClient.SqlInternalTransaction.Commit()
at System.Data.SqlClient.SqlTransaction.Commit()
at
Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionAndTransactionHolder
cth, Boolean openNewTransaction) in C:\Users\stunney\Desktop\Share
\Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
4053
--- End of inner exception stack trace ---
at
Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionAndTransactionHolder
cth, Boolean openNewTransaction) in C:\Users\stunney\Desktop\Share
\Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
4065
at Quartz.Impl.AdoJobStore.JobStoreSupport.DoCheckin() in C:\Users
\stunney\Desktop\Share\Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore
\JobStoreSupport.cs:line 3669
at Quartz.Impl.AdoJobStore.JobStoreSupport.ClusterManager.Manage()
in C:\Users\stunney\Desktop\Share\Quartz.NET-1.0.3\src\Quartz\Impl
\AdoJobStore\JobStoreSupport.cs:line 4282 [See nested exception:
System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The
timeout period elapsed prior to completion of the operation or the
server is not responding.
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at
System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject
stateObj, UInt32 error)
at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult
asyncResult, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
at System.Data.SqlClient.TdsParserStateObject.ReadByte()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior,
SqlCommand cmdHandler, SqlDataReader dataStream,
BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject
stateObj)
at
System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[]
buffer, TransactionManagerRequestType request, String transactionName,
TransactionManagerIsolationLevel isoLevel, Int32 timeout,
SqlInternalTransaction transaction, TdsParserStateObject stateObj,
Boolean isDelegateControlRequest)
at
System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest
transactionRequest, String transactionName, IsolationLevel iso,
SqlInternalTransaction internalTransaction, Boolean
isDelegateControlRequest)
at System.Data.SqlClient.SqlInternalTransaction.Commit()
at System.Data.SqlClient.SqlTransaction.Commit()
at
Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionAndTransactionHolder
cth, Boolean openNewTransaction) in C:\Users\stunney\Desktop\Share
\Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
4053]
2010-08-24 12:43:41,500
[Tsavo_DefaultQuartzJobScheduler_QuartzSchedulerThread] ERROR
Quartz.Core.ErrorLogger [(null)] - An error occured while firing
trigger 'DBWatchers.Trigger_SentryDBWatcher'
Quartz.JobPersistenceException: Couldn't commit ADO.NET transaction.
Timeout expired. The timeout period elapsed prior to completion of
the operation or the server is not responding. --->
System.Data.SqlClient.SqlException: Timeout expired. The timeout
period elapsed prior to completion of the operation or the server is
not responding.
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at
System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject
stateObj, UInt32 error)
at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult
asyncResult, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
at System.Data.SqlClient.TdsParserStateObject.ReadByte()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior,
SqlCommand cmdHandler, SqlDataReader dataStream,
BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject
stateObj)
at
System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[]
buffer, TransactionManagerRequestType request, String transactionName,
TransactionManagerIsolationLevel isoLevel, Int32 timeout,
SqlInternalTransaction transaction, TdsParserStateObject stateObj,
Boolean isDelegateControlRequest)
at
System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest
transactionRequest, String transactionName, IsolationLevel iso,
SqlInternalTransaction internalTransaction, Boolean
isDelegateControlRequest)
at System.Data.SqlClient.SqlInternalTransaction.Commit()
at System.Data.SqlClient.SqlTransaction.Commit()
at
Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionAndTransactionHolder
cth, Boolean openNewTransaction) in C:\Users\stunney\Desktop\Share
\Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
4053
--- End of inner exception stack trace ---
at
Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionAndTransactionHolder
cth, Boolean openNewTransaction) in C:\Users\stunney\Desktop\Share
\Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
4065
at
Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock(String
lockName, ITransactionCallback txCallback) in C:\Users\stunney\Desktop
\Share\Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore
\JobStoreSupport.cs:line 4242
at
Quartz.Impl.AdoJobStore.JobStoreSupport.TriggerFired(SchedulingContext
ctxt, Trigger trigger) in C:\Users\stunney\Desktop\Share
\Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
3250
at Quartz.Core.QuartzSchedulerThread.Run() in C:\Users\stunney
\Desktop\Share\Quartz.NET-1.0.3\src\Quartz\Core
\QuartzSchedulerThread.cs:line 388 [See nested exception:
System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The
timeout period elapsed prior to completion of the operation or the
server is not responding.
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at
System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject
stateObj, UInt32 error)
at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult
asyncResult, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
at System.Data.SqlClient.TdsParserStateObject.ReadByte()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior,
SqlCommand cmdHandler, SqlDataReader dataStream,
BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject
stateObj)
at
System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[]
buffer, TransactionManagerRequestType request, String transactionName,
TransactionManagerIsolationLevel isoLevel, Int32 timeout,
SqlInternalTransaction transaction, TdsParserStateObject stateObj,
Boolean isDelegateControlRequest)
at
System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest
transactionRequest, String transactionName, IsolationLevel iso,
SqlInternalTransaction internalTransaction, Boolean
isDelegateControlRequest)
at System.Data.SqlClient.SqlInternalTransaction.Commit()
at System.Data.SqlClient.SqlTransaction.Commit()
at
Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionAndTransactionHolder
cth, Boolean openNewTransaction) in C:\Users\stunney\Desktop\Share
\Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
4053]
2010-08-24 12:43:41,515
[Tsavo_DefaultQuartzJobScheduler_QuartzSchedulerThread] INFO root
[(null)] - Couldn't commit ADO.NET transaction. Timeout expired. The
timeout period elapsed prior to completion of the operation or the
server is not responding.
2010-08-24 12:43:43,312
[QuartzScheduler_Tsavo_DefaultQuartzJobScheduler-BIZSYS-
MONITOR2634182574917031250_ClusterManager] DEBUG
Quartz.Impl.AdoJobStore.JobStoreTX [(null)] - ClusterManager: Check-in
complete.
Marko Lahma
1.0.2 or was it introduced by update to new version? How many workers
do you have in your cluster?Are you putting a whole IoC container to
your database or what is that data you use?
-Marko
> --
> You received this message because you are subscribed to the Google Groups "Quartz.NET" group.
> To post to this group, send email to quar...@googlegroups.com.
> To unsubscribe from this group, send email to quartznet+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/quartznet?hl=en.
>
>
Stephen Tunney
This did indeed occur in 1.0.2
I currently have 1 instance running in my cluster, soon to be 7.
The "IoCContainer" that I am using is just a tiny little class that stores the only the filename of the location for my castle Windsor configuration data. When serializing I only keep the filename, the actual Kernel/Container object is new'd when I deserialize the object so that the db is kept as small as possible.
I have attached a copy of the IoCContainer implementation for your review.
Stephen
-Marko
> te[] buffer, TransactionManagerRequestType request, String
> transactionName, TransactionManagerIsolationLevel isoLevel, Int32
> timeout, SqlInternalTransaction transaction, TdsParserStateObject
> stateObj, Boolean isDelegateControlRequest)
> at
> System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon
> (TransactionRequest transactionRequest, String transactionName,
> IsolationLevel iso, SqlInternalTransaction internalTransaction,
> Boolean
> isDelegateControlRequest)
> at System.Data.SqlClient.SqlInternalTransaction.Commit()
> at System.Data.SqlClient.SqlTransaction.Commit()
> at
> Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionAnd
> TransactionHolder cth, Boolean openNewTransaction) in
> C:\Users\stunney\Desktop\Share
> \Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
> 4053
> --- End of inner exception stack trace ---
> at
> Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionAnd
> TransactionHolder cth, Boolean openNewTransaction) in
> C:\Users\stunney\Desktop\Share
> \Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
> 4065
> at Quartz.Impl.AdoJobStore.JobStoreSupport.DoCheckin() in C:\Users
> \stunney\Desktop\Share\Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore
> \JobStoreSupport.cs:line 3669
> at Quartz.Impl.AdoJobStore.JobStoreSupport.ClusterManager.Manage()
> in C:\Users\stunney\Desktop\Share\Quartz.NET-1.0.3\src\Quartz\Impl
> \AdoJobStore\JobStoreSupport.cs:line 4282 [See nested exception:
> System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The
> timeout period elapsed prior to completion of the operation or the
> server is not responding.
> at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
> exception, Boolean breakConnection)
> at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
> at
> System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserState
> Object
> stateObj, UInt32 error)
> at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult
> asyncResult, TdsParserStateObject stateObj)
> at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
> at System.Data.SqlClient.TdsParserStateObject.ReadByte()
> at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior,
> SqlCommand cmdHandler, SqlDataReader dataStream,
> BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject
> stateObj)
> at
> System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(By
> te[] buffer, TransactionManagerRequestType request, String
> transactionName, TransactionManagerIsolationLevel isoLevel, Int32
> timeout, SqlInternalTransaction transaction, TdsParserStateObject
> stateObj, Boolean isDelegateControlRequest)
> at
> System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon
> (TransactionRequest transactionRequest, String transactionName,
> IsolationLevel iso, SqlInternalTransaction internalTransaction,
> Boolean
> isDelegateControlRequest)
> at System.Data.SqlClient.SqlInternalTransaction.Commit()
> at System.Data.SqlClient.SqlTransaction.Commit()
> at
> Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionAnd
> TransactionHolder cth, Boolean openNewTransaction) in
> C:\Users\stunney\Desktop\Share
> \Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
> 4053]
> 2010-08-24 12:43:41,500
> [Tsavo_DefaultQuartzJobScheduler_QuartzSchedulerThread] ERROR
> Quartz.Core.ErrorLogger [(null)] - An error occured while firing
> trigger 'DBWatchers.Trigger_SentryDBWatcher'
> Quartz.JobPersistenceException: Couldn't commit ADO.NET transaction.
> Timeout expired. The timeout period elapsed prior to completion of
> the operation or the server is not responding. --->
> System.Data.SqlClient.SqlException: Timeout expired. The timeout
> period elapsed prior to completion of the operation or the server is
> not responding.
> at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
> exception, Boolean breakConnection)
> at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
> at
> System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserState
> Object
> stateObj, UInt32 error)
> at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult
> asyncResult, TdsParserStateObject stateObj)
> at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
> at System.Data.SqlClient.TdsParserStateObject.ReadByte()
> at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior,
> SqlCommand cmdHandler, SqlDataReader dataStream,
> BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject
> stateObj)
> at
> System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(By
> te[] buffer, TransactionManagerRequestType request, String
> transactionName, TransactionManagerIsolationLevel isoLevel, Int32
> timeout, SqlInternalTransaction transaction, TdsParserStateObject
> stateObj, Boolean isDelegateControlRequest)
> at
> System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon
> (TransactionRequest transactionRequest, String transactionName,
> IsolationLevel iso, SqlInternalTransaction internalTransaction,
> Boolean
> isDelegateControlRequest)
> at System.Data.SqlClient.SqlInternalTransaction.Commit()
> at System.Data.SqlClient.SqlTransaction.Commit()
> at
> Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionAnd
> TransactionHolder cth, Boolean openNewTransaction) in
> C:\Users\stunney\Desktop\Share
> \Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
> 4053
> --- End of inner exception stack trace ---
> at
> Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionAnd
> TransactionHolder cth, Boolean openNewTransaction) in
> C:\Users\stunney\Desktop\Share
> \Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
> 4065
> at
> Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock(Stri
> ng lockName, ITransactionCallback txCallback) in
> C:\Users\stunney\Desktop
> \Share\Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore
> \JobStoreSupport.cs:line 4242
> at
> Quartz.Impl.AdoJobStore.JobStoreSupport.TriggerFired(SchedulingContext
> ctxt, Trigger trigger) in C:\Users\stunney\Desktop\Share
> \Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
> 3250
> at Quartz.Core.QuartzSchedulerThread.Run() in C:\Users\stunney
> \Desktop\Share\Quartz.NET-1.0.3\src\Quartz\Core
> \QuartzSchedulerThread.cs:line 388 [See nested exception:
> System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The
> timeout period elapsed prior to completion of the operation or the
> server is not responding.
> at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
> exception, Boolean breakConnection)
> at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
> at
> System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserState
> Object
> stateObj, UInt32 error)
> at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult
> asyncResult, TdsParserStateObject stateObj)
> at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
> at System.Data.SqlClient.TdsParserStateObject.ReadByte()
> at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior,
> SqlCommand cmdHandler, SqlDataReader dataStream,
> BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject
> stateObj)
> at
> System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(By
> te[] buffer, TransactionManagerRequestType request, String
> transactionName, TransactionManagerIsolationLevel isoLevel, Int32
> timeout, SqlInternalTransaction transaction, TdsParserStateObject
> stateObj, Boolean isDelegateControlRequest)
> at
> System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon
> (TransactionRequest transactionRequest, String transactionName,
> IsolationLevel iso, SqlInternalTransaction internalTransaction,
> Boolean
> isDelegateControlRequest)
> at System.Data.SqlClient.SqlInternalTransaction.Commit()
> at System.Data.SqlClient.SqlTransaction.Commit()
> at
> Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionAnd
> TransactionHolder cth, Boolean openNewTransaction) in
Stephen Tunney
Also, I store the IIoCContainer implementation in the JobDetails of many (200+) jobs. I do find it strange though that this only ever happens to one trigger, and it is the first trigger created by my application on app startup (if it does not already exist).
Stephen Tunney
investigation and this appears to happen when the sql server that
quartz is on is under heavy (100%) cpu load for a few minutes.
On Aug 25, 5:26 pm, Marko Lahma <marko.la...@gmail.com> wrote:
> Just some quick checks before diving deeper.. Did this happen with
> 1.0.2 or was it introduced by update to new version? How many workers
> do you have in your cluster?Are you putting a whole IoC container to
> your database or what is that data you use?
>
> -Marko
>
> On Wed, Aug 25, 2010 at 5:23 PM, Stephen Tunney
>
>
>
> <stephen.tun...@gmail.com> wrote:
> > I am seeing (just upgraded to Quartz.Net 1.0.3.3 from sourceforge) a
> > single trigger, and it is the same one every single time, getting
> > stuck in the BLOCKED state. No other trigger in the system gets
> > stuck.
>
> > My job is a IStatefulJob. Key behaviours of my job include:
> > -Updating JobDataMap values (strings)
> > -Creating/Updating/Deleting other Job Schedules using the Quartz API
> > *only*
>
> > Here is the code I use to create the job and trigger, and finally
> > schedule them in the scheduler, each of the JobDataMap keys has the
> > [Serializable] attribute, and implements ISerializable properly (fully
> > unit tested each):
> > jobDetail = new JobDetail(jobName, JOB_GROUP_NAME,
> > typeof(Job_SentryJobDatabaseWatcher));
> > jobDetail.RequestsRecovery = true;
> > \AdoJobStore\JobStoreSupport.cs:line 4031
> > 2010-08-24 12:42:21,406 [QuartzScheduler_DefaultQuartzJobScheduler-
> > QUARTZBOX_A634182574917031250_ClusterManager] ERROR
> > Quartz.Impl.AdoJobStore.JobStoreTX [(null)] - Couldn't rollback
> > ADO.NET connection. This SqlTransaction has completed; it is no longer
> > usable.
> > System.InvalidOperationException: This SqlTransaction has completed;
> > it is no longer usable.
> > at System.Data.SqlClient.SqlTransaction.ZombieCheck()
> > at System.Data.SqlClient.SqlTransaction.Rollback()
> > at
> > at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult
> > asyncResult, TdsParserStateObject stateObj)
> > at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
> > at System.Data.SqlClient.TdsParserStateObject.ReadByte()
> > at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior,
> > SqlCommand cmdHandler, SqlDataReader dataStream,
> > BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject
> > stateObj)
> > at
> > System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[]
> > buffer, TransactionManagerRequestType request, String transactionName,
> > TransactionManagerIsolationLevel isoLevel, Int32 timeout,
> > SqlInternalTransaction transaction, TdsParserStateObject stateObj,
> > Boolean isDelegateControlRequest)
> > at
> > SqlInternalTransaction internalTransaction, Boolean
> > isDelegateControlRequest)
> > at System.Data.SqlClient.SqlInternalTransaction.Commit()
> > at System.Data.SqlClient.SqlTransaction.Commit()
> > at
> > \Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
> > 4053
> > --- End of inner exception stack trace ---
> > at
> > \Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
> > 4065
> > at Quartz.Impl.AdoJobStore.JobStoreSupport.DoCheckin() in C:\Users
> > \stunney\Desktop\Share\Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore
> > \JobStoreSupport.cs:line 3669
> > at Quartz.Impl.AdoJobStore.JobStoreSupport.ClusterManager.Manage()
> > in C:\Users\stunney\Desktop\Share\Quartz.NET-1.0.3\src\Quartz\Impl
> > \AdoJobStore\JobStoreSupport.cs:line 4282 [See nested exception:
> > System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The
>
> - Show quoted text -...
>
> read more »
Blake
the BLOCKED state and staying there. I can't see any pattern to it.
I'm also using ADO.NET storage.
>
> read more »
Stephen Tunney
The same job that was being placed into perpetual BLOCKED state is now in a perpetual ACQUIRED state. Again, it is a CRON, with a misfire instruction of 1 (SmartPolicy). I recently changed the JobDataMap to only contain strings (useProperties=true in my config), and all complex objects that I used to pass in through the datamap are now assigned to public properties in my job class in the JobToBeExecuted callback in a custom JobListener (dependency injection, yay!).
This CRON is set to fire every 2 minutes, reads from a very small database, and creates new quartz jobs based on the data it finds.
Just a heads up that the BLOCKED state is not the only condition for this.
Stephen
> > > _CONNECTI ON_STRING] = m_sqlServerConnectionString;
> > > jobDetail.JobDataMap[Job_SentryJobDatabaseWatcher.JOBDETAILS_EMAIL
> > > _NOTIFIER ] = m_emailer;
> > > jobDetail.JobDataMap[Job_SentryJobDatabaseWatcher.JOBDETAILS_LOGGE
> > > R] = m_logger;
> > > jobDetail.JobDataMap[Job_SentryJobDatabaseWatcher.JOBDETAIL_IOC_CO
> > > NTAINER]
--
Marko Lahma
Could you guys elaborate these bits about your situation:
* the actual job/trigger relations
- how many triggers per job type and job instance (instance = same
job name and group)
* job type
- are types in your cases IStatefulJobs?
* estimated/average job execution time per execution
If you have just one server in cluser (and hence not a cluster, you
could just use isClustered = false). Running stateful jobs with DB can
generally cause weird situations if running is locked from other
triggers and triggers run long and cause other triggers to wait (if
triggers' target is the same job definition).
Generally I would recommend against using stateful jobs with db if possible.
-Marko
Marko Lahma
would have interest to test how system performs with trunk version of
Quartz.NET. It has some reworked internals (ported from 1.8) and other
performance enhancements (especially when using RAMJobStore). Not safe
for production (NSFP) though :-)
You should just need to compile from trunk, configuration should work
as is. The interfaces have been generified so if you have lot of
references to IScheduler API there might be some work to get tests
run...
-Marko
Stephen Tunney
-All jobs in my case are IStateful (all of my jobs inherit from a base class)
-all of my jobs run in under a minute. Most are around 2-3 seconds
-I use the stateful job to record which machine the job last ran on (just the Machine Name), and a couple of other stats.
I'm sure I could do something else instead of using IStatefulJob (History add-in)
What odd things are you seeing with the ADO jobstore and stateful jobs?
I'll try removing stateful job and see how it goes. This error happens every few hours, so it should be easy to see if it resolves the problem over the next couple of days.
Stephen
-----Original Message-----
From: quar...@googlegroups.com [mailto:quar...@googlegroups.com] On Behalf Of Marko Lahma
Sent: Wednesday, September 08, 2010 2:38 PM
To: quar...@googlegroups.com
-Marko
>> > > y _CONNECTI ON_STRING] = m_sqlServerConnectionString;
>> > > S tateObjec t stateObj, UInt32 error)
>> > > at
>> > > System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult
>> > > asyncResult, TdsParserStateObject stateObj)
>> > > at
>> > > System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
>> > > at System.Data.SqlClient.TdsParserStateObject.ReadByte()
>> > > at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior,
>> > > SqlCommand cmdHandler, SqlDataReader dataStream,
>> > > BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject
>> > > stateObj)
>> > > at
>> > > System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerReque
>> > > s t(Byte[] buffer, TransactionManagerRequestType request, String
>> > > transactionName, TransactionManagerIsolationLevel isoLevel, Int32
>> > > timeout, SqlInternalTransaction transaction, TdsParserStateObject
>> > > stateObj, Boolean isDelegateControlRequest)
>> > > at
>> > > System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransaction
>> > > Y ukon(Tran sactionRequest transactionRequest, String
>> > > transactionName, IsolationLevel iso, SqlInternalTransaction
>> > > internalTransaction, Boolean
>> > > isDelegateControlRequest)
>> > > at System.Data.SqlClient.SqlInternalTransaction.Commit()
>> > > at System.Data.SqlClient.SqlTransaction.Commit()
>> > > at
>> > > Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(Connecti
>> > > o nAndTrans actionHolder cth, Boolean openNewTransaction) in
Marko Lahma
blocked state can be result of triggers running stateful jobs that
must be blocked until completion. Are you now running 7 instances or
just one with isClustered = true. Could you also try to set clustered
to false when running the test if only one instance?
-Marko
Stephen Tunney
Blake
and 1 instance with isClustered = false.
The job getting blocked is an IStatefulJob, 2 other jobs are also
IStatefulJobs
Average execution type for the blocked job is 10 min. Job is set to
run every 30.
The job reads records from a db and makes a remote call and then
writes back to the database with updated data.
It blocks even when it doesn't run over 30 min.
What happens if the job throws an exception while executing? Will
quartz.net catch the exception and mark the job as done or will it
think the job is still running? I expect it knows the job exceptions
out, just throwing out ideas.
>
> read more »
Stephen Tunney
I kept clustering on, but ALL of my jobs are now simply IJob
implementers, and I never modify the datamap of a job after scheduling
it.
Furthermore, all of the items in my datamap are strings, and the
useProperties config param for the ADO job store is set to true.
The job completes, and the as the follow logs will show, some cluster
checkins complete, and then some trigger management finishes, and then
EXPLOSION. Could it be a collision between the background management
threads that quartz has running that perhaps go after the same
resource (transaction object) and one beats the other to calling the
Commit()/Rollback() on that object?
Here are the logs for everything just before and including the stack
trace again:
2010-09-10 01:48:00,039 [Sentry_DefaultQuartzJobScheduler_Worker-5]
INFO root [(null)] - Job executed for 00:00:00.0156250, details are
Job Name:Job_SentryDBWatcher
Job Group:DBWatchers
JDM Key:SentryDBConnectionString='Data
Source=COMPUTERA;Database=Sentry;User ID=XXXXXX;Password=XXXXXX;'
2010-09-10 01:48:00,039 [Sentry_DefaultQuartzJobScheduler_Worker-5]
DEBUG Quartz.Core.JobRunShell [(null)] - Trigger instruction :
NoInstruction
2010-09-10 01:48:00,039 [Sentry_DefaultQuartzJobScheduler_Worker-5]
'TRIGGER_ACCESS' is desired by:
2010-09-10 01:48:00,039 [Sentry_DefaultQuartzJobScheduler_Worker-5]
'TRIGGER_ACCESS' is being obtained:
2010-09-10 01:48:00,054 [Sentry_DefaultQuartzJobScheduler_Worker-5]
2010-09-10 01:48:00,054 [Sentry_DefaultQuartzJobScheduler_Worker-5]
Sentry_DefaultQuartzJobScheduler_Worker-5
2010-09-10 01:48:13,711
[QuartzScheduler_Sentry_DefaultQuartzJobScheduler-634196389862578750_ClusterManager]
Check-in complete.
[QuartzScheduler_Sentry_DefaultQuartzJobScheduler-634196389862578750_ClusterManager]
Check-in complete.
[QuartzScheduler_Sentry_DefaultQuartzJobScheduler-634196389862578750_ClusterManager]
Check-in complete.
[QuartzScheduler_Sentry_DefaultQuartzJobScheduler-634196389862578750_ClusterManager]
Check-in complete.
[QuartzScheduler_Sentry_DefaultQuartzJobScheduler-634196389862578750_ClusterManager]
Check-in complete.
[QuartzScheduler_Sentry_DefaultQuartzJobScheduler-634196389862578750_MisfireHandler]
scanning for misfires...
[QuartzScheduler_Sentry_DefaultQuartzJobScheduler-634196389862578750_MisfireHandler]
that missed their scheduled fire-time.
[QuartzScheduler_Sentry_DefaultQuartzJobScheduler-634196389862578750_ClusterManager]
ADO.NET connection. This SqlTransaction has completed; it is no longer
usable.
System.InvalidOperationException: This SqlTransaction has completed;
it is no longer usable.
at System.Data.SqlClient.SqlTransaction.ZombieCheck()
at System.Data.SqlClient.SqlTransaction.Rollback()
at
\AdoJobStore\JobStoreSupport.cs:line 4031
[Sentry_DefaultQuartzJobScheduler_QuartzSchedulerThread] ERROR
ADO.NET connection. This SqlTransaction has completed; it is no longer
usable.
System.InvalidOperationException: This SqlTransaction has completed;
it is no longer usable.
at System.Data.SqlClient.SqlTransaction.ZombieCheck()
at System.Data.SqlClient.SqlTransaction.Rollback()
at
\AdoJobStore\JobStoreSupport.cs:line 4031
Marko Lahma
again and again with little luck so far. Some more checks to do:
* SQL Server version?
* Do you create your own transactions in your jobs using
transactionscope or by other means?
* Are cluster nodes clocks synchronized?
* what happens if you increase the thread pool thread count, if you
now have two handling them?
I'm sure there are people on the list running clustered setups without
problems, if so please chime in that I don't get too paranoid :)
Blake: Unhandled exceptions are handled with forced no-retry police,
proceeding with standard scheduling. If you catch exceptions in your
jobs and rethrow as JobExecutionException (as you should) you can give
details about what to do (refire instructions).
-Marko
Stephen Tunney
I tried increasing the thread pool count from 2 to 6, and the behaviour stayed the same.
On a side note, removing IStatefulJob and turning off clustering seems to have eliminated the problem, I will let you know for sure sometime tomorrow after reviewing the logs and checking the state of the quartz database.
Stephen
----- Original Message -----
From: quar...@googlegroups.com <quar...@googlegroups.com>
To: quar...@googlegroups.com <quar...@googlegroups.com>
Stephen Tunney
scheduler for a job that is merely an IJob implementation.
And now, the Trigger for this job (Job_SentryDBWatcher) is stuck in an
"ACQUIRED" state in the database. It is a CRON job set to run every 2
minutes (0 0/2 * ? * * *).
Hope this helps, as it elminates the clustering and IStatefulJob logic
from your investigation. I really wish I could attach files to these
posts, as it would make things easier to work with.
Thanks for looking into this Marko!
2010-09-10 22:12:00,031 [Sentry_DefaultQuartzJobScheduler_Worker-4]
DEBUG Quartz.Core.JobRunShell [(null)] - Calling Execute on job
DBWatchers.Job_SentryDBWatcher
2010-09-10 22:12:00,031 [Sentry_DefaultQuartzJobScheduler_Worker-4]
DEBUG root [(null)] - AJob::Execute Called
2010-09-10 22:12:00,046 [Sentry_DefaultQuartzJobScheduler_Worker-4]
DEBUG root [(null)] - AJob::Execute Call Complete
2010-09-10 22:12:00,046 [Sentry_DefaultQuartzJobScheduler_Worker-4]
Job Name:Job_SentryDBWatcher
Job Group:DBWatchers
JDM Key:SentryDBConnectionString='Data
ID=USERXXXX;Password=password1234;'
2010-09-10 22:12:00,046 [Sentry_DefaultQuartzJobScheduler_Worker-4]
NoInstruction
'TRIGGER_ACCESS' is desired by:
2010-09-10 22:12:00,046 [Sentry_DefaultQuartzJobScheduler_Worker-4]
'TRIGGER_ACCESS' is being obtained:
2010-09-10 22:12:00,062 [Sentry_DefaultQuartzJobScheduler_Worker-4]
2010-09-10 22:12:00,062 [Sentry_DefaultQuartzJobScheduler_Worker-4]
'TRIGGER_ACCESS' returned by:
2010-09-10 22:13:54,890
[QuartzScheduler_Sentry_DefaultQuartzJobScheduler-
NON_CLUSTERED_MisfireHandler] DEBUG Quartz.Impl.AdoJobStore.JobStoreTX
Quartz.Impl.AdoJobStore.JobStoreTX [(null)] - Couldn't rollback
ADO.NET connection. This SqlTransaction has completed; it is no longer
usable.
System.InvalidOperationException: This SqlTransaction has completed;
it is no longer usable.
at System.Data.SqlClient.SqlTransaction.ZombieCheck()
at System.Data.SqlClient.SqlTransaction.Rollback()
at
Quartz.Impl.AdoJobStore.JobStoreSupport.RollbackConnection(ConnectionAndTransactionHolder
cth) in C:\Users\stunney\Desktop\Share\Quartz.NET-1.0.3\src\Quartz\Impl
\AdoJobStore\JobStoreSupport.cs:line 4031
[QuartzScheduler_Sentry_DefaultQuartzJobScheduler-
NON_CLUSTERED_MisfireHandler] DEBUG Quartz.Impl.AdoJobStore.JobStoreTX
[Sentry_DefaultQuartzJobScheduler_QuartzSchedulerThread] ERROR
Quartz.Core.ErrorLogger [(null)] - An error occured while scanning for
the next trigger to fire.
Timeout expired. The timeout period elapsed prior to completion of
the operation or the server is not responding. --->
System.Data.SqlClient.SqlException: Timeout expired. The timeout
period elapsed prior to completion of the operation or the server is
not responding.
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at
at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult
asyncResult, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
at System.Data.SqlClient.TdsParserStateObject.ReadByte()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior,
SqlCommand cmdHandler, SqlDataReader dataStream,
BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject
stateObj)
at
System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[]
buffer, TransactionManagerRequestType request, String transactionName,
TransactionManagerIsolationLevel isoLevel, Int32 timeout,
SqlInternalTransaction transaction, TdsParserStateObject stateObj,
Boolean isDelegateControlRequest)
at
SqlInternalTransaction internalTransaction, Boolean
isDelegateControlRequest)
at System.Data.SqlClient.SqlInternalTransaction.Commit()
at System.Data.SqlClient.SqlTransaction.Commit()
at
cth, Boolean openNewTransaction) in C:\Users\stunney\Desktop\Share
--- End of inner exception stack trace ---
at
cth, Boolean openNewTransaction) in C:\Users\stunney\Desktop\Share
at
Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock(String
lockName, ITransactionCallback txCallback) in C:\Users\stunney\Desktop
at
Quartz.Impl.AdoJobStore.JobStoreSupport.AcquireNextTrigger(SchedulingContext
ctxt, DateTime noLaterThan) in C:\Users\stunney\Desktop\Share
at Quartz.Core.QuartzSchedulerThread.Run() in C:\Users\stunney
\Desktop\Share\Quartz.NET-1.0.3\src\Quartz\Core
\QuartzSchedulerThread.cs:line 286 [See nested exception:
System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The
server is not responding.
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at
at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult
asyncResult, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
at System.Data.SqlClient.TdsParserStateObject.ReadByte()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior,
SqlCommand cmdHandler, SqlDataReader dataStream,
BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject
stateObj)
at
System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[]
buffer, TransactionManagerRequestType request, String transactionName,
TransactionManagerIsolationLevel isoLevel, Int32 timeout,
SqlInternalTransaction transaction, TdsParserStateObject stateObj,
Boolean isDelegateControlRequest)
at
SqlInternalTransaction internalTransaction, Boolean
isDelegateControlRequest)
at System.Data.SqlClient.SqlInternalTransaction.Commit()
at System.Data.SqlClient.SqlTransaction.Commit()
at
cth, Boolean openNewTransaction) in C:\Users\stunney\Desktop\Share
2010-09-10 22:14:07,703
[Sentry_DefaultQuartzJobScheduler_QuartzSchedulerThread] INFO root
[(null)] - Couldn't commit ADO.NET transaction. Timeout expired. The
server is not responding.
[Sentry_DefaultQuartzJobScheduler_QuartzSchedulerThread] DEBUG
'TRIGGER_ACCESS' is desired by:
2010-09-10 22:15:00,015
[Sentry_DefaultQuartzJobScheduler_QuartzSchedulerThread] DEBUG
'TRIGGER_ACCESS' is being obtained:
2010-09-10 22:15:00,031
[Sentry_DefaultQuartzJobScheduler_QuartzSchedulerThread] DEBUG
'TRIGGER_ACCESS' given to:
2010-09-10 22:15:00,062
[Sentry_DefaultQuartzJobScheduler_QuartzSchedulerThread] DEBUG
'TRIGGER_ACCESS' returned by:
2010-09-10 22:15:00,062
[Sentry_DefaultQuartzJobScheduler_QuartzSchedulerThread] DEBUG
Quartz.Simpl.SimpleJobFactory [(null)] - Producing instance of Job
'SentryJobs.SENTRY_JOB_24',
class=Jobs.Sentry.Job_StoredProcThatProducesOutputToBeEMailed
2010-09-10 22:15:00,062 [Sentry_DefaultQuartzJobScheduler_Worker-5]
INFO root [(null)] - Job to be executed, details are
Job Name:SENTRY_JOB_24
Job Group:SentryJobs
JDM Key:JobID='24'
JDM Key:SentryDBConnectionString='Data
Source=MACHINEA;Database=Sentry;User
ID=USERXXXX;Password=password1234;'
> > Check-in complete.
> > 2010-09-10 01:48:28,711
> > Check-in complete.
> > 2010-09-10 01:48:47,211
> > Check-in complete.
> > 2010-09-10 01:49:08,429
> > Check-in complete.
> > 2010-09-10 01:49:13,711
> > Check-in complete.
> > 2010-09-10 01:49:16,586
> > scanning for misfires...
> > 2010-09-10 01:49:16,586
> > that missed their scheduled fire-time.
> > 2010-09-10 01:49:48,711
> > ADO.NET connection. This SqlTransaction has completed; it is no longer
> > usable.
> > System.InvalidOperationException: This SqlTransaction has completed;
> > it is no longer usable.
> > at System.Data.SqlClient.SqlTransaction.ZombieCheck()
> > at System.Data.SqlClient.SqlTransaction.Rollback()
> > at
> > \AdoJobStore\JobStoreSupport.cs:line 4031
> > 2010-09-10 01:49:58,664
> > [Sentry_DefaultQuartzJobScheduler_QuartzSchedulerThread] ERROR
> > Quartz.Impl.AdoJobStore.JobStoreTX [(null)] - Couldn't rollback
> > ADO.NET connection. This SqlTransaction has completed; it is no longer
> > usable.
> > System.InvalidOperationException: This SqlTransaction has completed;
> > it is no longer usable.
> > at System.Data.SqlClient.SqlTransaction.ZombieCheck()
> > at System.Data.SqlClient.SqlTransaction.Rollback()
> > at
Marko Lahma
believe this is doing some DB polling?). If so, does your job check
Quartz tables ever? In the last test you made you were running simple
cron trigger with 6 thread pool threads. And it seems that you job
always finishes, say, under 5 seconds?
-Marko
Stephen Tunney
My job polls a separate set of tables for data that helps define how new jobs will be scheduled in quartz. But I never directly access the quartz tables ever. I only ever use the Quartz API for checking for the existence of triggers, jobs, and calendars.
And yes, six threads in my pool, all triggers are relatively simple Cron triggers of varying types (most run every 2, 5, 15 minutes every hour, every day). And I'd say most of the jobs finish in under 5 seconds, yes.
-----Original Message-----
From: quar...@googlegroups.com [mailto:quar...@googlegroups.com] On Behalf Of Marko Lahma
Sent: Monday, September 13, 2010 12:55 PM
To: quar...@googlegroups.com
-Marko
> Quartz.Impl.AdoJobStore.JobStoreSupport.RollbackConnection(ConnectionA
> ndTransactionHolder
> cth) in
> te[] buffer, TransactionManagerRequestType request, String
> transactionName, TransactionManagerIsolationLevel isoLevel, Int32
> timeout, SqlInternalTransaction transaction, TdsParserStateObject
> stateObj, Boolean isDelegateControlRequest)
> at
> System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon
> (TransactionRequest transactionRequest, String transactionName,
> IsolationLevel iso, SqlInternalTransaction internalTransaction,
> Boolean
> isDelegateControlRequest)
> at System.Data.SqlClient.SqlInternalTransaction.Commit()
> at System.Data.SqlClient.SqlTransaction.Commit()
> at
> Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionAnd
> TransactionHolder cth, Boolean openNewTransaction) in
> C:\Users\stunney\Desktop\Share
> \Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
> 4053
> --- End of inner exception stack trace ---
> at
> Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionAnd
> TransactionHolder cth, Boolean openNewTransaction) in
> C:\Users\stunney\Desktop\Share
> \Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
> 4065
> at
> Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock(Stri
> ng lockName, ITransactionCallback txCallback) in
> C:\Users\stunney\Desktop
> \Share\Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore
> \JobStoreSupport.cs:line 4242
> at
> Quartz.Impl.AdoJobStore.JobStoreSupport.AcquireNextTrigger(SchedulingC
> ontext ctxt, DateTime noLaterThan) in C:\Users\stunney\Desktop\Share
> \Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
> 3114
> at Quartz.Core.QuartzSchedulerThread.Run() in C:\Users\stunney
> \Desktop\Share\Quartz.NET-1.0.3\src\Quartz\Core
> \QuartzSchedulerThread.cs:line 286 [See nested exception:
> System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The
> timeout period elapsed prior to completion of the operation or the
> server is not responding.
> at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
> exception, Boolean breakConnection)
> at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
> at
> System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserState
> Object
> stateObj, UInt32 error)
> at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult
> asyncResult, TdsParserStateObject stateObj)
> at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
> at System.Data.SqlClient.TdsParserStateObject.ReadByte()
> at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior,
> SqlCommand cmdHandler, SqlDataReader dataStream,
> BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject
> stateObj)
> at
> System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(By
> te[] buffer, TransactionManagerRequestType request, String
> transactionName, TransactionManagerIsolationLevel isoLevel, Int32
> timeout, SqlInternalTransaction transaction, TdsParserStateObject
> stateObj, Boolean isDelegateControlRequest)
> at
> System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon
> (TransactionRequest transactionRequest, String transactionName,
> IsolationLevel iso, SqlInternalTransaction internalTransaction,
> Boolean
> isDelegateControlRequest)
> at System.Data.SqlClient.SqlInternalTransaction.Commit()
> at System.Data.SqlClient.SqlTransaction.Commit()
> at
> Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionAnd
> TransactionHolder cth, Boolean openNewTransaction) in
Marko Lahma
behaves? Might there be any connection problems that might cause
transaction timeout? With this non-clustered configuration will Quartz
continue to run correctly or do you see that trigger stops to fire?
Because ADO.NET connection can be reset by SQL Server due to network
error and thus creating the error in the log about not being able to
rollback or commit.
-Marko
Stephen Tunney
SQL Server/network issues should be handled appropriately by quartz, shouldn't they?
-Marko
>> Quartz.Impl.AdoJobStore.JobStoreSupport.RollbackConnection(Connection
>> A
>> ndTransactionHolder
>> cth) in
>> y te[] buffer, TransactionManagerRequestType request, String
>> transactionName, TransactionManagerIsolationLevel isoLevel, Int32
>> timeout, SqlInternalTransaction transaction, TdsParserStateObject
>> stateObj, Boolean isDelegateControlRequest)
>> at
>> System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYuko
>> n (TransactionRequest transactionRequest, String transactionName,
>> IsolationLevel iso, SqlInternalTransaction internalTransaction,
>> Boolean
>> isDelegateControlRequest)
>> at System.Data.SqlClient.SqlInternalTransaction.Commit()
>> at System.Data.SqlClient.SqlTransaction.Commit()
>> at
>> Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionAn
>> d TransactionHolder cth, Boolean openNewTransaction) in
>> C:\Users\stunney\Desktop\Share
>> \Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
>> 4053
>> --- End of inner exception stack trace ---
>> at
>> Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionAn
>> d TransactionHolder cth, Boolean openNewTransaction) in
>> C:\Users\stunney\Desktop\Share
>> \Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
>> 4065
>> at
>> Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock(Str
>> i ng lockName, ITransactionCallback txCallback) in
>> C:\Users\stunney\Desktop
>> \Share\Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore
>> \JobStoreSupport.cs:line 4242
>> at
>> Quartz.Impl.AdoJobStore.JobStoreSupport.AcquireNextTrigger(Scheduling
>> C ontext ctxt, DateTime noLaterThan) in
>> C:\Users\stunney\Desktop\Share
>> \Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line
>> 3114
>> at Quartz.Core.QuartzSchedulerThread.Run() in C:\Users\stunney
>> \Desktop\Share\Quartz.NET-1.0.3\src\Quartz\Core
>> \QuartzSchedulerThread.cs:line 286 [See nested exception:
>> System.Data.SqlClient.SqlException (0x80131904): Timeout expired.
>> The timeout period elapsed prior to completion of the operation or
>> the server is not responding.
>> at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
>> exception, Boolean breakConnection)
>> at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
>> at
>> System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStat
>> e
>> Object
>> stateObj, UInt32 error)
>> at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult
>> asyncResult, TdsParserStateObject stateObj)
>> at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
>> at System.Data.SqlClient.TdsParserStateObject.ReadByte()
>> at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior,
>> SqlCommand cmdHandler, SqlDataReader dataStream,
>> BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject
>> stateObj)
>> at
>> System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(B
>> y te[] buffer, TransactionManagerRequestType request, String
>> transactionName, TransactionManagerIsolationLevel isoLevel, Int32
>> timeout, SqlInternalTransaction transaction, TdsParserStateObject
>> stateObj, Boolean isDelegateControlRequest)
>> at
>> System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYuko
>> n (TransactionRequest transactionRequest, String transactionName,
>> IsolationLevel iso, SqlInternalTransaction internalTransaction,
>> Boolean
>> isDelegateControlRequest)
>> at System.Data.SqlClient.SqlInternalTransaction.Commit()
>> at System.Data.SqlClient.SqlTransaction.Commit()
>> at
>> Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionAn
>> d TransactionHolder cth, Boolean openNewTransaction) in
>>> > 5 0_ClusterManager] DEBUG Quartz.Impl.AdoJobStore.JobStoreTX
>>> > [(null)] - ClusterManager:
>>> > Check-in complete.
>>> > 2010-09-10 01:48:28,711
>>> > [QuartzScheduler_Sentry_DefaultQuartzJobScheduler-6341963898625787
>>> > 5 0_ClusterManager] DEBUG Quartz.Impl.AdoJobStore.JobStoreTX
>>> > [(null)] - ClusterManager:
>>> > Check-in complete.
>>> > 2010-09-10 01:48:47,211
>>> > [QuartzScheduler_Sentry_DefaultQuartzJobScheduler-6341963898625787
>>> > 5 0_ClusterManager] DEBUG Quartz.Impl.AdoJobStore.JobStoreTX
>>> > [(null)] - ClusterManager:
>>> > Check-in complete.
>>> > 2010-09-10 01:49:08,429
>>> > [QuartzScheduler_Sentry_DefaultQuartzJobScheduler-6341963898625787
>>> > 5 0_ClusterManager] DEBUG Quartz.Impl.AdoJobStore.JobStoreTX
>>> > [(null)] - ClusterManager:
>>> > Check-in complete.
>>> > 2010-09-10 01:49:13,711
>>> > [QuartzScheduler_Sentry_DefaultQuartzJobScheduler-6341963898625787
>>> > 5 0_ClusterManager] DEBUG Quartz.Impl.AdoJobStore.JobStoreTX
>>> > [(null)] - ClusterManager:
>>> > Check-in complete.
>>> > 2010-09-10 01:49:16,586
>>> > [QuartzScheduler_Sentry_DefaultQuartzJobScheduler-6341963898625787
>>> > 5 0_MisfireHandler] DEBUG Quartz.Impl.AdoJobStore.JobStoreTX
>>> > [(null)] - MisfireHandler:
>>> > scanning for misfires...
>>> > 2010-09-10 01:49:16,586
>>> > [QuartzScheduler_Sentry_DefaultQuartzJobScheduler-6341963898625787
>>> > 5 0_MisfireHandler] DEBUG Quartz.Impl.AdoJobStore.JobStoreTX
>>> > [(null)] - Found 0 triggers that missed their scheduled fire-time.
>>> > 2010-09-10 01:49:48,711
>>> > [QuartzScheduler_Sentry_DefaultQuartzJobScheduler-6341963898625787
>>> > 5 0_ClusterManager] ERROR Quartz.Impl.AdoJobStore.JobStoreTX
Marko Lahma
possible due to network errors. This is not the case though as
triggers do get stuck. Back to the drawing board....
-Marko
Mark Jones
Here's the link to the trial : http://www.red-gate.com/products/ants_performance_profiler/
Cheers
Mark
Mark Jones
SharePoint Solutions Architect - Collaboris Ltd
www.collaboris.co.uk
This message is intended only for the use of the person(s) ("the intended recipient(s)") to whom it is addressed. It may contain information which is privileged and confidential within the meaning of applicable law. If you are not the intended recipient, please contact the sender as soon as possible. The views expressed in this communication may not necessarily be the views held by The Company.
-Marko
>>> Quartz.Impl.AdoJobStore.JobStoreSupport.RollbackConnection(Connectio
>>> n
>>> A
>>> ndTransactionHolder
>>> B y te[] buffer, TransactionManagerRequestType request, String
>>> transactionName, TransactionManagerIsolationLevel isoLevel, Int32
>>> timeout, SqlInternalTransaction transaction, TdsParserStateObject
>>> stateObj, Boolean isDelegateControlRequest)
>>> at
>>> System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYuk
>>> o n (TransactionRequest transactionRequest, String transactionName,
>>> IsolationLevel iso, SqlInternalTransaction internalTransaction,
>>> Boolean
>>> isDelegateControlRequest)
>>> at System.Data.SqlClient.SqlInternalTransaction.Commit()
>>> at System.Data.SqlClient.SqlTransaction.Commit()
>>> at
>>> Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionA
>>> n d TransactionHolder cth, Boolean openNewTransaction) in
>>> C:\Users\stunney\Desktop\Share
>>> \Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:lin
>>> e
>>> 4053
>>> --- End of inner exception stack trace ---
>>> at
>>> Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionA
>>> n d TransactionHolder cth, Boolean openNewTransaction) in
>>> C:\Users\stunney\Desktop\Share
>>> \Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:lin
>>> e
>>> 4065
>>> at
>>> Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock(St
>>> r i ng lockName, ITransactionCallback txCallback) in
>>> C:\Users\stunney\Desktop
>>> \Share\Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore
>>> \JobStoreSupport.cs:line 4242
>>> at
>>> Quartz.Impl.AdoJobStore.JobStoreSupport.AcquireNextTrigger(Schedulin
>>> g C ontext ctxt, DateTime noLaterThan) in
>>> C:\Users\stunney\Desktop\Share
>>> \Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:lin
>>> e
>>> 3114
>>> at Quartz.Core.QuartzSchedulerThread.Run() in C:\Users\stunney
>>> \Desktop\Share\Quartz.NET-1.0.3\src\Quartz\Core
>>> \QuartzSchedulerThread.cs:line 286 [See nested exception:
>>> System.Data.SqlClient.SqlException (0x80131904): Timeout expired.
>>> The timeout period elapsed prior to completion of the operation or
>>> the server is not responding.
>>> at
>>> System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
>>> exception, Boolean breakConnection)
>>> at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
>>> at
>>> System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserSta
>>> t
>>> e
>>> Object
>>> stateObj, UInt32 error)
>>> at
>>> System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult
>>> asyncResult, TdsParserStateObject stateObj)
>>> at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
>>> at System.Data.SqlClient.TdsParserStateObject.ReadByte()
>>> at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior,
>>> SqlCommand cmdHandler, SqlDataReader dataStream,
>>> BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject
>>> stateObj)
>>> at
>>> System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(
>>> B y te[] buffer, TransactionManagerRequestType request, String
>>> transactionName, TransactionManagerIsolationLevel isoLevel, Int32
>>> timeout, SqlInternalTransaction transaction, TdsParserStateObject
>>> stateObj, Boolean isDelegateControlRequest)
>>> at
>>> System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYuk
>>> o n (TransactionRequest transactionRequest, String transactionName,
>>> IsolationLevel iso, SqlInternalTransaction internalTransaction,
>>> Boolean
>>> isDelegateControlRequest)
>>> at System.Data.SqlClient.SqlInternalTransaction.Commit()
>>> at System.Data.SqlClient.SqlTransaction.Commit()
>>> at
>>> Quartz.Impl.AdoJobStore.JobStoreSupport.CommitConnection(ConnectionA
>>> n d TransactionHolder cth, Boolean openNewTransaction) in
>>> C:\Users\stunney\Desktop\Share
>>> \Quartz.NET-1.0.3\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:lin
>>> e
Stephen Tunney
1.0.3.3 code base and I think I have a "fix" for the issue.
First of all, I think the error is misleading, in that the command
that was associated with the "failed" transaction actually made it
though and got comitted to the database (sql server scenario
anyways). So one solution would be to simply swallow the exception.
The other, which I have done, it to add a try/catch to
JobStoreSupport::AcquireNextTrigger(SchedulingContext ctxt, DateTime
noLaterThan):Trigger
Here is the code block as I changed it to:
/// <summary>
/// Get a handle to the next N triggers to be fired, and mark them
as 'reserved'
/// by the calling scheduler.
/// </summary>
/// <seealso cref="ReleaseAcquiredTrigger(SchedulingContext,
Trigger)" />
public virtual Trigger AcquireNextTrigger(SchedulingContext ctxt,
DateTime noLaterThan)
{
object retval = null;
try
{
if (AcquireTriggersWithinLock)
{
ExecuteInNonManagedTXLock(LockTriggerAccess, new
AcquireNextTriggerCallback(this, ctxt, noLaterThan), out retval);
}
else
{
// default behavior since Quartz 1.0.1 release
ExecuteInNonManagedTXLock(
null, /* passing null as lock name causes no
lock to be made */
new AcquireNextTriggerCallback(this, ctxt,
noLaterThan), out retval);
}
return (Trigger)retval;
}
catch (JobPersistenceException _ex)
{
if (null == retval) throw;
try
{
Log.Warn("Error getting the next trigger.
Attempting to revert trigger back to WAITING state", _ex);
//TODO: Reset trigger state back to WAITING
ExecuteInNonManagedTXLock(null, new
ReleaseAcquiredTriggerCallback(this, ctxt, (Trigger)retval));
throw;
}
catch(Exception _ex2)
{
throw new Exception(string.Format("Attempt to
revert Trigger '{0}' back to WAITING state failed.",
((Trigger)retval).FullName), _ex2);
}
}
}
And a new version of ExecuteInNonManagedTXLock that has an out param:
protected virtual void ExecuteInNonManagedTXLock(string lockName,
ITransactionCallback txCallback, out object _result)
{
bool transOwner = false;
ConnectionAndTransactionHolder conn = null;
try
{
if (lockName != null)
{
// If we aren't using db locks, then delay getting DB connection
// until after aquiring the lock since it isn't needed.
if (LockHandler.RequiresConnection)
{
conn = GetNonManagedTXConnection();
}
transOwner = LockHandler.ObtainLock(DbProvider.Metadata, conn,
lockName);
}
if (conn == null)
{
conn = GetNonManagedTXConnection();
}
_result = txCallback.Execute(conn);
CommitConnection(conn, false);
}
catch (JobPersistenceException)
{
RollbackConnection(conn);
throw;
}
catch (Exception e)
{
RollbackConnection(conn);
throw new JobPersistenceException("Unexpected runtime exception: "
+ e.Message, e);
}
finally
{
try
{
ReleaseLock(conn, lockName, transOwner);
}
finally
{
CleanupConnection(conn);
}
}
}
Now one thing to note, is that I get the timeout exception in the last
catch (_ex2) block, but the state of the trigger in question is set
back to WAITING anyways.
I have let this code run on my end for 4 days now and I don't run into
the issue anymore. I realize the code smells that exist in my
suggestions, but this is the fastest "hammer" I could come up with for
the problem at hand.
Again, I wish I could attach files to these forum posts. Marko, I can
upload the file to you if you wish to see if it passed your vigorous
testing framework.
Regards,
Stephen
On Sep 13, 3:53 pm, Marko Lahma <marko.la...@gmail.com> wrote:
> Yes, Quartz should handle that gracefully, but false alarms are also
> possible due to network errors. This is not the case though as
> triggers do get stuck. Back to the drawing board....
>
> -Marko
>
>
>
> On Mon, Sep 13, 2010 at 10:47 PM, Stephen Tunney <STun...@tsavo.com> wrote:
> > The triggers that become locked in the ACQUIRED or BLOCKED state no longer continue to fire at all.
> > SQL Server/network issues should be handled appropriately by quartz, shouldn't they?
>
> > -----Original Message-----
> > From: quar...@googlegroups.com [mailto:quar...@googlegroups.com] On Behalf Of Marko Lahma
> > Sent: Monday, September 13, 2010 3:44 PM
> > To: quar...@googlegroups.com
> > Subject: Re: [quartznet:2037] Re: 1.0.3.3 - Trigger getting stuck in BLOCKED State
>
> > Hmm.. maybe back to basics then. Have you checked how the database behaves? Might there be any connection problems that might cause transaction timeout? With this non-clustered configuration will Quartz continue to run correctly or do you see that trigger stops to fire?
>
> > Because ADO.NET connection can be reset by SQL Server due to network error and thus creating the error in the log about not being able to rollback or commit.
>
> > -Marko
>
> >> No the Quartz tables are in a completely separate database.
>
> >> My job polls a separate set of tables for data that helps define how new jobs will be scheduled in quartz. But I never directly access the quartz tables ever. I only ever use the Quartz API for checking for the existence of triggers, jobs, and calendars.
>
> >> And yes, six threads in my pool, all triggers are relatively simple Cron triggers of varying types (most run every 2, 5, 15 minutes every hour, every day). And I'd say most of the jobs finish in under 5 seconds, yes.
>
> >> -----Original Message-----
> >> From: quar...@googlegroups.com [mailto:quar...@googlegroups.com]
> >> On Behalf Of Marko Lahma
> >> Sent: Monday, September 13, 2010 12:55 PM
> >> To: quar...@googlegroups.com
> >> Subject: Re: [quartznet:2030] Re: 1.0.3.3 - Trigger getting stuck in
> >> BLOCKED State
>
> >> Are you Quartz.NET tables in the same DB as the job's target DB (I believe this is doing some DB polling?). If so, does your job check Quartz tables ever? In the last test you made you were running simple cron trigger with 6 thread pool threads. And it seems that you job always finishes, say, under 5 seconds?
>
> >> -Marko
>
Stephen Tunney
and every time I get the culprit exception the modified code kicks in
and fixes the trigger state.
Marko, can you see yourself putting this code in as a permanent patch
to your code base? I know that this fixes the ACQUIRED state issue, I
will put my config back into a clustered state and see if the BLOCKED
state issue comes back.
Regards,
Stephen
Stephen Tunney
definitely fixes the blocked and acquire state stickiness issue.
YAY!!
Lloyd Cotten
We've been running quartz.net for months now with no issue. Suddenly, on Monday this started occurring. We run this on a server using SQL Server 2005 as the job store and Windows Clustering (quartz is not clustered though, only one instance). On Monday we had trouble with the Windows Clustering trying to switch nodes, so some services, including SQL Server, was suddenly stopped, and the DNS resolution to the SQL Server was mixed up for a bit. That's the first time I have ever seen the issue described here. We resolved the underlying problem within a few minutes and I restarted the service running quartz, and all the jobs seemingly resumed execution correctly.
However, several hours later I noticed that the jobs stopped, and upon investigation some of them were stuck in the ACQUIRED or BLOCKED states. Upon restarting the service they would resume, but again after a period of time some would again get stuck. It is random (not always the same jobs). It's been happening all week. We've had no other issues with resolving the SQL Server host name or other services being stopped. My jobs are mainly concerned with file system operations (search for, copy, and ftp files), no database interaction.
One thing to note, is that I've determined that the jobs that get stuck will execute successfully (no exception) on their regular schedule (all CronTriggers), then they will get stuck before executing again. First command in the job Execute is to write to a log and last command is to write to a log.
Lloyd
Lloyd Cotten
Lloyd
Stephen Tunney
Lloyd,
What changes did you make on your side? Was it OS, Network, SQLServer (What version of each are you running)?
Inquiring minds want to know!
Stephen
Lloyd Cotten
Marko Lahma
problems come from connectivity issues or server dropping transactions
for some reason. Quartz seems to handle the db connection handling
logic correctly but as said, when things go wrong in unpredictable
ways (rollbacks don't work, connections time-out etc) there's a place
to improve.
As a software vendor I would strongly recommend checking the hardware ;)
Jokes aside, this definitely is an area where robustness should be
improved on Quartz's side.
-Marko
Stephen Tunney
Marko, does the patch look acceptable to you?
Lloyd Cotten
Stephen Tunney
Hey Lloyd,
I’m not sure the changes you made will actually work now. I needed to have the Trigger be an out param because when the exception is thrown in the first try, nextTrigger will ALWAYS be null (because the exception is thrown within the ExecuteInNonManagedTXLock(…) and hence the assignment of nextTrigger will not take place), and the check inside of the outer catch will never try and release the acquired trigger.
Also, the ExecuteInNonManagedTXLock(…) method I supplied was not meant to replace, but simply overload/compliment the one that already exists, so that I don’t modify any other functionality in the code base.
Believe me, I know the code isn’t elegant :) But it was the best stop-gap I could come up with.
Marko Lahma
try to rollback but still get committed data in DB. So without
successful commit (whether it is timeout or whatnot) we get data
committed to DB. I need to dig deeper and try to find what's the catch
here.. The patches you propose try to handle rollback of changes that
should not exist (yes, they do exist). Or this is just my analysis..
Thanks guys for your insight, I really appreciate it. Lets try to
find out what's going on here!
-Marko
Marko Lahma
JobStoreSupport.AcquireNextTrigger in your setup if it isn't too much
of a trouble? The modification here is to give up after 20 seconds
(SqlCommand timeout by default is 30 seconds). This version is to
tackle the possibility of connection/transaction timing out.
This is just a guess and I'm afraid that not a good solution as it
would indicate that your setup could take *a lot* of time trying to
find next trigger. If I understand correctly you don't have that many
triggers to handle. If you'd have many then the indexes described in
the FAQ would come handy to ensure timely updates and retrieval.
-Marko
DateTime acquireStart = DateTime.UtcNow;
do
{
try
{
Trigger nextTrigger = null;
IList keys = Delegate.SelectTriggerToAcquire(conn,
noLaterThan, MisfireTime);
// No trigger is ready to fire yet.
if (keys == null || keys.Count == 0)
{
return null;
}
foreach (Key triggerKey in keys)
{
int rowsUpdated =
Delegate.UpdateTriggerStateFromOtherState(
conn,
triggerKey.Name, triggerKey.Group,
StateAcquired, StateWaiting);
// If our trigger was no longer in the
expected state, try a new one.
if (rowsUpdated <= 0)
{
continue;
}
nextTrigger = RetrieveTrigger(conn, ctxt,
triggerKey.Name, triggerKey.Group);
// If our trigger is no longer available, try a new one.
if (nextTrigger == null)
{
continue;
}
break;
}
// if we didn't end up with a trigger to fire from
that first
// batch, try again for another batch
if (nextTrigger == null)
{
continue;
}
nextTrigger.FireInstanceId = FiredTriggerRecordId;
Delegate.InsertFiredTrigger(conn, nextTrigger,
StateAcquired, null);
return nextTrigger;
}
catch (Exception e)
{
throw new JobPersistenceException(
"Couldn't acquire next trigger: " + e.Message, e);
}
} while (DateTime.UtcNow - acquireStart < TimeSpan.FromSeconds(20));
return null;
Stephen Tunney
Having multiple threads makes this a supreme headache to test. I'm just glad I caught the symptom :) Now to cure the cold!
-----Original Message-----
From: quar...@googlegroups.com [mailto:quar...@googlegroups.com] On Behalf Of Marko Lahma
Sent: Wednesday, September 22, 2010 1:37 PM
To: quar...@googlegroups.com
Subject: Re: [quartznet:2082] Re: 1.0.3.3 - Trigger getting stuck in BLOCKED State
The most troubling thing here that I see is that we get exception, we
try to rollback but still get committed data in DB. So without
successful commit (whether it is timeout or whatnot) we get data
committed to DB. I need to dig deeper and try to find what's the catch
here.. The patches you propose try to handle rollback of changes that
should not exist (yes, they do exist). Or this is just my analysis..
Thanks guys for your insight, I really appreciate it. Lets try to
find out what's going on here!
-Marko
On Wed, Sep 22, 2010 at 8:23 PM, Stephen Tunney <STu...@tsavo.com> wrote:
> Hey Lloyd,
>
>
>
> I'm not sure the changes you made will actually work now. I needed to have
> the Trigger be an out param because when the exception is thrown in the
> first try, nextTrigger will ALWAYS be null (because the exception is thrown
> within the ExecuteInNonManagedTXLock(...) and hence the assignment of
> nextTrigger will not take place), and the check inside of the outer catch
> will never try and release the acquired trigger.
>
> Also, the ExecuteInNonManagedTXLock(...) method I supplied was not meant to
Stephen Tunney
Stephen
-Marko
return null;
>> within the ExecuteInNonManagedTXLock(...) and hence the assignment of
>> nextTrigger will not take place), and the check inside of the outer catch
>> will never try and release the acquired trigger.
>>
>> Also, the ExecuteInNonManagedTXLock(...) method I supplied was not meant to
Marko Lahma
The connection-transaction pair is opened per operation and stored
only in call tree (stack) so there shouldn't be any shared resource
conflicts. I suspect this to be some SQL Server edge case. One option
would be to use SET XACT_ABORT ON before every statement. What I
understand from the documentation is that SQL Server may commit
transaction if "no serious" error occurs. This would mean that you can
acquire a trigger and get a "non-fatal" error that would throw Quartz
of track (catch-block) and lose handle to the trigger that was ought
to be processed.
I still think that these might be symptoms to some other problem
though. Do you have the indexes created for the tables?
-Marko
Lloyd Cotten
Stephen Tunney
Blake
blocked (even after re-adding the job + trigger to the database).
More info on my environment:
No clustering, thread size of 3
SQL Server 2008 Standard
SQL Server is on the same box as Quartz.NET (shouldn't be having any
networking/connectivity problems)
1 trigger per job, all using cron
Jobs being that are getting blocked have the same cron expression.
Jobs being blocked never run successfully (they don't run once and
then block).
Quartz.NET tables are in same database as application tables.
Jobs read from application tables and never reference Quartz.NET
tables.
Jobs run successfully using the RAM store.
On Sep 22, 2:39 pm, Stephen Tunney <STun...@tsavo.com> wrote:
> Yes, I have the indexes recommended on your site implemented in the tables.
>
>
>
> -----Original Message-----
> From: quar...@googlegroups.com [mailto:quar...@googlegroups.com] On Behalf Of Marko Lahma
> Sent: Wednesday, September 22, 2010 3:21 PM
> To: quar...@googlegroups.com
> Subject: Re: [quartznet:2086] Re: 1.0.3.3 - Trigger getting stuck in BLOCKED State
>
> Thanks, much appreciated. I'm not in a hurry unlike you :-)
>
> The connection-transaction pair is opened per operation and stored
> only in call tree (stack) so there shouldn't be any shared resource
> conflicts. I suspect this to be some SQL Server edge case. One option
> would be to use SET XACT_ABORT ON before every statement. What I
> understand from the documentation is that SQL Server may commit
> transaction if "no serious" error occurs. This would mean that you can
> acquire a trigger and get a "non-fatal" error that would throw Quartz
> of track (catch-block) and lose handle to the trigger that was ought
> to be processed.
>
> I still think that these might be symptoms to some other problem
> though. Do you have the indexes created for the tables?
>
> -Marko
>
> >> The most troubling thing here that I see is that we get exception, we
> >> try to rollback but still get committed data in DB. So without
> >> successful commit (whether it is timeout or whatnot) we get data
> >> committed to DB. I need to dig deeper and try to find what's the catch
> >> here.. The patches you propose try to handle rollback of changes that
> >> should not exist (yes, they do exist). Or this is just my analysis..
>
> >> Thanks guys for your insight, I really appreciate it. Lets try to
> >> find out what's going on here!
>
> >> -Marko
>
>
> read more »
Blake
prepare for the capture I removed all rows from the Job Details table
and from the Triggers table. Then started my Quartz.NET service.
Below is the output. I apologize if it doesn't render very well.
exec sp_executesql N'INSERT INTO QRTZ_TRIGGERS (TRIGGER_NAME,
TRIGGER_GROUP, JOB_NAME, JOB_GROUP, IS_VOLATILE, DESCRIPTION,
NEXT_FIRE_TIME, PREV_FIRE_TIME, TRIGGER_STATE, TRIGGER_TYPE,
START_TIME, END_TIME, CALENDAR_NAME, MISFIRE_INSTR, JOB_DATA,
PRIORITY)
VALUES(@triggerName, @triggerGroup, @triggerJobName,
@triggerJobGroup, @triggerVolatile, @triggerDescription,
@triggerNextFireTime, @triggerPreviousFireTime, @triggerState,
@triggerType, @triggerStartTime, @triggerEndTime,
@triggerCalendarName, @triggerMisfireInstruction,
@triggerJobJobDataMap, @triggerPriority)',N'@triggerName
nvarchar(90),@triggerGroup nvarchar(5),@triggerJobName
nvarchar(90),@triggerJobGroup nvarchar(5),@triggerVolatile
int,@triggerDescription nvarchar(4000),@triggerNextFireTime
decimal(18,0),@triggerPreviousFireTime decimal(1,0),@triggerState
nvarchar(7),@triggerType nvarchar(4),@triggerStartTime
decimal(18,0),@triggerEndTime decimal(1,0),@triggerCalendarName
nvarchar(4000),@triggerMisfireInstruction int,@triggerJobJobDataMap
image,@triggerPriority int',@triggerName=N'Namespace.EmailJob,
Namespace',@triggerGroup=N'tasks',@triggerJobName=N'Namespace.EmailJob,
Namespace',@triggerJobGroup=N'tasks',@triggerVolatile=0,@triggerDescription=NULL,@triggerNextFireTime=634212846150000000,@triggerPreviousFireTime=-1,@triggerState=N'BLOCKED',@triggerType=N'CRON',@triggerStartTime=634212846130000000,@triggerEndTime=0,@triggerCalendarName=NULL,@triggerMisfireInstruction=1,@triggerJobJobDataMap=NULL,@triggerPriority=5
exec sp_executesql N'INSERT INTO QRTZ_TRIGGERS (TRIGGER_NAME,
TRIGGER_GROUP, JOB_NAME, JOB_GROUP, IS_VOLATILE, DESCRIPTION,
NEXT_FIRE_TIME, PREV_FIRE_TIME, TRIGGER_STATE, TRIGGER_TYPE,
START_TIME, END_TIME, CALENDAR_NAME, MISFIRE_INSTR, JOB_DATA,
PRIORITY)
VALUES(@triggerName, @triggerGroup, @triggerJobName,
@triggerJobGroup, @triggerVolatile, @triggerDescription,
@triggerNextFireTime, @triggerPreviousFireTime, @triggerState,
@triggerType, @triggerStartTime, @triggerEndTime,
@triggerCalendarName, @triggerMisfireInstruction,
@triggerJobJobDataMap, @triggerPriority)',N'@triggerName
nvarchar(94),@triggerGroup nvarchar(5),@triggerJobName
nvarchar(94),@triggerJobGroup nvarchar(5),@triggerVolatile
int,@triggerDescription nvarchar(4000),@triggerNextFireTime
decimal(18,0),@triggerPreviousFireTime decimal(1,0),@triggerState
nvarchar(7),@triggerType nvarchar(4),@triggerStartTime
decimal(18,0),@triggerEndTime decimal(1,0),@triggerCalendarName
nvarchar(4000),@triggerMisfireInstruction int,@triggerJobJobDataMap
image,@triggerPriority int',@triggerName=N'Namespace.BulkEmailJob,
Namespace',@triggerGroup=N'tasks',@triggerJobName=N'Namespace.BulkEmailJob,
Namespace',@triggerJobGroup=N'tasks',@triggerVolatile=0,@triggerDescription=NULL,@triggerNextFireTime=634212846150000000,@triggerPreviousFireTime=-1,@triggerState=N'BLOCKED',@triggerType=N'CRON',@triggerStartTime=634212846140000000,@triggerEndTime=0,@triggerCalendarName=NULL,@triggerMisfireInstruction=1,@triggerJobJobDataMap=NULL,@triggerPriority=5
exec sp_executesql N'UPDATE QRTZ_TRIGGERS SET TRIGGER_STATE =
@newState WHERE TRIGGER_STATE = @oldState1 OR TRIGGER_STATE =
@oldState2',N'@newState nvarchar(7),@oldState1 nvarchar(8),@oldState2
nvarchar(7)',@newState=N'WAITING',@oldState1=N'ACQUIRED',@oldState2=N'BLOCKED'
exec sp_executesql N'UPDATE QRTZ_TRIGGERS SET TRIGGER_STATE =
@newState WHERE TRIGGER_STATE = @oldState1 OR TRIGGER_STATE =
@oldState2',N'@newState nvarchar(6),@oldState1 nvarchar(14),@oldState2
nvarchar(14)',@newState=N'PAUSED',@oldState1=N'PAUSED_BLOCKED',@oldState2=N'PAUSED_BLOCKED'
exec sp_executesql N'UPDATE QRTZ_TRIGGERS SET TRIGGER_STATE = @state
WHERE JOB_NAME = @jobName AND JOB_GROUP = @jobGroup AND TRIGGER_STATE
= @oldState',N'@state nvarchar(7),@jobName nvarchar(90),@jobGroup
nvarchar(5),@oldState
nvarchar(7)',@state=N'BLOCKED',@jobName=N'Namespace.EmailJob,
Namespace',@jobGroup=N'tasks',@oldState=N'WAITING'
exec sp_executesql N'UPDATE QRTZ_TRIGGERS SET TRIGGER_STATE = @state
WHERE JOB_NAME = @jobName AND JOB_GROUP = @jobGroup AND TRIGGER_STATE
= @oldState',N'@state nvarchar(7),@jobName nvarchar(90),@jobGroup
nvarchar(5),@oldState
nvarchar(8)',@state=N'BLOCKED',@jobName=N'Namespace.EmailJob,
Namespace',@jobGroup=N'tasks',@oldState=N'ACQUIRED'
exec sp_executesql N'UPDATE QRTZ_TRIGGERS SET TRIGGER_STATE = @state
WHERE JOB_NAME = @jobName AND JOB_GROUP = @jobGroup AND TRIGGER_STATE
= @oldState',N'@state nvarchar(14),@jobName nvarchar(90),@jobGroup
nvarchar(5),@oldState
nvarchar(6)',@state=N'PAUSED_BLOCKED',@jobName=N'Namespace.EmailJob,
Namespace',@jobGroup=N'tasks',@oldState=N'PAUSED'
exec sp_executesql N'UPDATE QRTZ_TRIGGERS SET JOB_NAME =
@triggerJobName, JOB_GROUP = @triggerJobGroup, IS_VOLATILE =
@triggerVolatile, DESCRIPTION = @triggerDescription, NEXT_FIRE_TIME =
@triggerNextFireTime, PREV_FIRE_TIME = @triggerPreviousFireTime,
TRIGGER_STATE = @triggerState, TRIGGER_TYPE = @triggerType,
START_TIME = @triggerStartTime, END_TIME = @triggerEndTime,
CALENDAR_NAME = @triggerCalendarName, MISFIRE_INSTR =
@triggerMisfireInstruction, PRIORITY = @triggerPriority
WHERE TRIGGER_NAME = @triggerName AND TRIGGER_GROUP =
@triggerGroup',N'@triggerJobName nvarchar(90),@triggerJobGroup
nvarchar(5),@triggerVolatile int,@triggerDescription
nvarchar(4000),@triggerNextFireTime
decimal(18,0),@triggerPreviousFireTime decimal(18,0),@triggerState
nvarchar(7),@triggerType nvarchar(4),@triggerStartTime
decimal(18,0),@triggerEndTime decimal(1,0),@triggerCalendarName
nvarchar(4000),@triggerMisfireInstruction int,@triggerPriority
int,@triggerName nvarchar(90),@triggerGroup
nvarchar(5)',@triggerJobName=N'Namespace.EmailJob,
Namespace',@triggerJobGroup=N'tasks',@triggerVolatile=0,@triggerDescription=NULL,@triggerNextFireTime=634212846300000000,@triggerPreviousFireTime=634212846150000000,@triggerState=N'BLOCKED',@triggerType=N'CRON',@triggerStartTime=634212846130000000,@triggerEndTime=0,@triggerCalendarName=NULL,@triggerMisfireInstruction=1,@triggerPriority=5,@triggerName=N'Namespace.EmailJob,
Namespace',@triggerGroup=N'tasks'
exec sp_executesql N'UPDATE QRTZ_TRIGGERS SET TRIGGER_STATE = @state
WHERE JOB_NAME = @jobName AND JOB_GROUP = @jobGroup AND TRIGGER_STATE
= @oldState',N'@state nvarchar(7),@jobName nvarchar(94),@jobGroup
nvarchar(5),@oldState
nvarchar(7)',@state=N'BLOCKED',@jobName=N'Namespace.BulkEmailJob,
Namespace',@jobGroup=N'tasks',@oldState=N'WAITING'
exec sp_executesql N'UPDATE QRTZ_TRIGGERS SET TRIGGER_STATE = @state
WHERE JOB_NAME = @jobName AND JOB_GROUP = @jobGroup AND TRIGGER_STATE
= @oldState',N'@state nvarchar(7),@jobName nvarchar(94),@jobGroup
nvarchar(5),@oldState
nvarchar(8)',@state=N'BLOCKED',@jobName=N'Namespace.BulkEmailJob,
Namespace',@jobGroup=N'tasks',@oldState=N'ACQUIRED'
exec sp_executesql N'UPDATE QRTZ_TRIGGERS SET TRIGGER_STATE = @state
WHERE JOB_NAME = @jobName AND JOB_GROUP = @jobGroup AND TRIGGER_STATE
= @oldState',N'@state nvarchar(14),@jobName nvarchar(94),@jobGroup
nvarchar(5),@oldState
nvarchar(6)',@state=N'PAUSED_BLOCKED',@jobName=N'Namespace.BulkEmailJob,
Namespace',@jobGroup=N'tasks',@oldState=N'PAUSED'
exec sp_executesql N'UPDATE QRTZ_TRIGGERS SET JOB_NAME =
@triggerJobName, JOB_GROUP = @triggerJobGroup, IS_VOLATILE =
@triggerVolatile, DESCRIPTION = @triggerDescription, NEXT_FIRE_TIME =
@triggerNextFireTime, PREV_FIRE_TIME = @triggerPreviousFireTime,
TRIGGER_STATE = @triggerState, TRIGGER_TYPE = @triggerType,
START_TIME = @triggerStartTime, END_TIME = @triggerEndTime,
CALENDAR_NAME = @triggerCalendarName, MISFIRE_INSTR =
@triggerMisfireInstruction, PRIORITY = @triggerPriority
WHERE TRIGGER_NAME = @triggerName AND TRIGGER_GROUP =
@triggerGroup',N'@triggerJobName nvarchar(94),@triggerJobGroup
nvarchar(5),@triggerVolatile int,@triggerDescription
nvarchar(4000),@triggerNextFireTime
decimal(18,0),@triggerPreviousFireTime decimal(18,0),@triggerState
nvarchar(7),@triggerType nvarchar(4),@triggerStartTime
decimal(18,0),@triggerEndTime decimal(1,0),@triggerCalendarName
nvarchar(4000),@triggerMisfireInstruction int,@triggerPriority
int,@triggerName nvarchar(94),@triggerGroup
nvarchar(5)',@triggerJobName=N'Namespace.BulkEmailJob,
Namespace',@triggerJobGroup=N'tasks',@triggerVolatile=0,@triggerDescription=NULL,@triggerNextFireTime=634212846300000000,@triggerPreviousFireTime=634212846150000000,@triggerState=N'BLOCKED',@triggerType=N'CRON',@triggerStartTime=634212846140000000,@triggerEndTime=0,@triggerCalendarName=NULL,@triggerMisfireInstruction=1,@triggerPriority=5,@triggerName=N'Namespace.BulkEmailJob,
Namespace',@triggerGroup=N'tasks'
>
> read more »
Blake
an exception in a JobListener, causing the job to be marked as BLOCKED
but never execute. It never got written back to WAITING after that.
>
> read more »
Stephen Tunney
that the issue I am seeing is unrelated to the issues that Blake has
discovered. Though this is something that should also be handled
gracefully by Quartz internally. Perhaps catch a custom exception
type, the same way that JobExecutionException is expected from a
Job::Execute call. Just some random thoughts.