Timeout issue

[Luke]

Hi,

I have recently started using Quartz for scheduling surveys for our
team members.

Brief structure:
- Quartz 2.x running as windows service (standalone)
- MSSQL2k8 R2

- Number of jobs (currently around 30 - not much from what I have seen
other people use quartz for)
- One job - which I call collector is a job with a trigger to run
every 30 seconds. It sole purpose if there are any new jobs in the
database that need to be created and scheduled in quartz. (I think
someone else on this group used similar structure).

Problem:
- Every so often, when collector runs it gets following error:

DatacomQuartzScheduler-NON_CLUSTERED_MisfireHandler] ERROR
Quartz.Impl.AdoJobStore.JobStoreTX [(null)] - MisfireHandler: Error
handling misfires: Failure setting up connection.
Quartz.JobPersistenceException: Failure setting up connection. --->
System.Data.SqlClient.SqlException: Timeout expired. The timeout
period elapsed prior to completion of the operation or the server is
not responding.

I have checked the SQL server and it doesn't even see Quartz
attempting to make connection.
This error causes collector to stop. All current jobs that are in the
quartz do trigger fine, however no new jobs are being created since
collector fails. This is what I have checked so far:
- Network - no issues there.
- Database - checked for any log shipping, backups, anything that
could cause timeout or db going down. Nothing wrong though, everything
is working fine and time of the error doesnt match any maintenance
stuff.
- I tried increasing Timeout in the connection string to 60 seconds.
Hasn't changed anything.
- I found few posts of people getting timeout issues with connection
to SQL server, solution was to switch off TCP chimney. I switched it
off - no change.

Is there any way of capturing this and getting Quartz.net to retry?
When I resume trigger for collector it runs fine again. The worst
problem with this error is that I can't manually recreate it,
therefore I couldn't really pin point any specific code that would
cause this.

Also, big thanks to the dev team. You are doing brilliant job!

[Luke]

I managed to find where the error occurs and change it so that it
tries X amount of times before fully giving up on getting connection.
Not sure if anyone had similar issues with Quartz. Let me know if you
did:

File I modified was JobStoreSupport.cs:410

Quartz still throws an error that it couldn't connect, however it puts
thread to sleep for Y seconds and tries again and my job doesn't just
stop it carries on which is very good.

I'm vaguely familiar with Quartz code so I'm not sure if this is a
good way to tackle this problem.

[Luke]

Would be good to get some input on this but this is what I have added
to GetConnection() method.

const int MaxTries = 5;
const int SleepTime = 10000;
for (var i = 1; i <= 5; i++)
{
try
{
if (TxIsolationLevelSerializable)
{
tx =
conn.BeginTransaction(IsolationLevel.Serializable);
}
else
{
// default
tx =
conn.BeginTransaction(IsolationLevel.ReadCommitted);
}
break;

}
catch (Exception e)
{
if (i == MaxTries)
{
conn.Close();
throw new JobPersistenceException("Failure
setting up connection.", e);
}
Thread.Sleep(SleepTime);
}
}

Obviously max-tries and sleep-time can be moved to config if needed.

[yorah]

Hello Luke,

Are you able to reproduce the issue with a different computer and/or
database? The thing I'm tryng to understand here is if the connection
failure is something wrong inside Quartz.net, or if it's something in
your infrastructure.

An easy way could be to create a simple console application, which
would have an infinite loop in which it tries to open a connection and
a transaction, issue a simple request to your database, and then close
the connection. If you are able to reproduce your issue that way, then
it means you may have a problem (network, drivers, ...) on your side.
Otherwise, we can investigate more on Quartz.net side.

For your proposed fix, I will try to give you some input about it
tomorrow (I'm not on my main computer right now).

Yoram

[Luke]

It is happening on both our uat and production environment. (4
different VMs - since each environment has a pair of SQL server VM and
app server)
It isn't the best fix, I know, however it makes sure that my trigger
doesn't just freeze.

[Marko Lahma]

I believe there's something wrong with the infrastructure. What kind
of connections pool configuration are you using and do you have
connection limits on database server's side?

-Marko

> --
> You received this message because you are subscribed to the Google Groups "Quartz.NET" group.
> To post to this group, send email to quar...@googlegroups.com.
> To unsubscribe from this group, send email to quartznet+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/quartznet?hl=en.
>