Quartz and JavaMelody on Tomcat

[MiTaes]

Hi again,

I have another issue with JavaMelody running on tomcat and Quartz
scheduler. We use quartz schedulers (1.8.1 /1.8.2) configured via
Spring SchedulerFactoryBean and with JavaMelody added the tomcat
shutdown does not work anymore. We have to kill the catalina java
process. I don't know if this is an issue in Quartz (quite possible)
or with JavaMelody but any ideas would be welcome.

Thanks and Regards,
Michael

[evernat]

Hi Michael,

Is there a destroy on Spring when your webapp is undeployed before the
server shutdowns?
(for example, in a contextDestroyed method of a
ServletContextListener)
Because I think that the destroy() method of your SchedulerFactoryBean
should be called in order that it calls the shutdown() method on its
quartz scheduler.
And you did not say to your SchedulerFactoryBean to wait for jobs to
complete before shutdown, or did you?

And have you said to your SchedulerFactoryBean to
exposeSchedulerInRepository?
http://code.google.com/p/javamelody/wiki/UserGuide#12._Batch_jobs_%28if_Quartz%29

If all was ok until this point, perhaps you could add a call to
org.quartz.impl.StdSchedulerFactory.getDefaultScheduler().shutdown();
Or to make it sure: for (Scheduler scheduler :
SchedulerRepository.getInstance().lookupAll())
{ scheduler.shutdown(); }
If this last point solves the issue, then it is probably related to
the fact that the spring bean creates its own Quartz scheduler for
itself.

Please say the result to us.

Thanks, Emeric

[MiTaes]

Hi Emeric,

thank you for your response. We use Spring ContextConfigListener to
manage spring context handling so I suppose the scheduler should be
destroyed upon call to contextDestroyed(). I set the destroy-method to
"destroy" the SchedulerFactoryBean.

I enabled the exposeSchedulerInRepository and the jobs are visible in
JavaMelody.

I will try to explicitly shutdown the schedulers via
SchedulerRepository in my contextListener and will see if this works.
I will also check if the SchedulerFactoryBean is set to wait for
scheduler shutdown. It should be "true", right ?!

One curious thing though is that even with commenting out our
scheduler config in the ContextConfigListener part in web.xml Tomcat
does not shutdown. There are not jobs visible in JavaMelody in this
case.

Regards,
Michael

On Oct 28, 6:56 pm, evernat <ever...@free.fr> wrote:
> Hi Michael,
>
> Is there a destroy on Spring when your webapp is undeployed before the
> server shutdowns?
> (for example, in a contextDestroyed method of a
> ServletContextListener)
> Because I think that the destroy() method of your SchedulerFactoryBean
> should be called in order that it calls the shutdown() method on its
> quartz scheduler.
> And you did not say to your SchedulerFactoryBean to wait for jobs to
> complete before shutdown, or did you?
>
> And have you said to your SchedulerFactoryBean to

> exposeSchedulerInRepository?http://code.google.com/p/javamelody/wiki/UserGuide#12._Batch_jobs_%28...

[burkhard]

Hi, I exactly ran into the same problem as Michael (using/testing
melody 1.19 and 1.21, quartz 1.6.x and spring 2.5.6 (the project is a
little bit older)).

On 1 Nov., 11:26, MiTaes <m.taesch...@googlemail.com> wrote:
> One curious thing though is that even with commenting out our
> scheduler config in the ContextConfigListener part in web.xml Tomcat
> does not shutdown.

Confirmed.

> I will also check if the SchedulerFactoryBean is set to wait for
> scheduler shutdown. It should be "true", right ?!

Did not help.

> On Oct 28, 6:56 pm, evernat <ever...@free.fr> wrote:
> ...
> > If all was ok until this point, perhaps you could add a call to

Yes, all ok.

> I will try to explicitly shutdown the schedulers via
> SchedulerRepository in my contextListener and will see if this works.

I've done that: Now Tomcat (6.9.29) can be stopped, but he complains
in his logfiles:

01.11.2010 17:44:35 org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
... The web application ... appears to have started a thread named
[DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is
very likely to create a memory leak.

...up to Worker-10, which has not been stopped.

[evernat]

Hi,

I will try to give informations on what is done in javamelody and what
is done in spring, and I will give a jar for test.

When the filter of javamelody is initialized and if quartz classes are
present, javamelody add a listener on all quartz schedulers and in
particular on the default quartz scheduler:
http://code.google.com/p/javamelody/source/browse/trunk/javamelody-core/src/main/java/net/bull/javamelody/JobGlobalListener.java#46

This listener is used to display in javamelody the execution time, cpu
time and sql executions of each jobs.
The fact of getting the default quartz listener initializes it if
needed with the quartz.properties configuration or with the default
quartz configuration.
Most applications with quartz jobs use the default quartz scheduler
via StdSchedulerFactory.getDefaultScheduler().

Javamelody does not shutdown the quartz scheduler(s) when the webapp
is undeployed (or on server shutdown) because the destroy method of
the javamelody filter is perhaps not the right time to shutdown the
quartz schedulers and also because javamelody does not know if you
want to wait or not for jobs to complete before shutdown. So most
applications should call shutdown() on the quartz schedulers when the
webapp is undeployed (for example, in a
ServletContextListener.contextDestroyed method) otherwise the server
can't be shutdown : the quartz threads are not stopped and they are
not flagged as daemons which prevents the jvm to stop.
Until now all is good: quartz jobs in applications are executed and
the javamelody listener can monitor the jobs. Threads are stopped when
the application calls shutdown on the default scheduler or on all
schedulers.


And Spring comes. In fact, SchedulerFactoryBean in Spring does not use
the default quartz scheduler, it creates its own scheduler. And by
default, Spring does not want that the quartz scheduler is known by
quartz: Spring removes the scheduler from quartz just after its
creation, unless exposeSchedulerInRepository is set to true in
SchedulerFactoryBean. That's why exposeSchedulerInRepository should be
set to true in order for javamelody to display the jobs from Spring.

In the destroy method of SchedulerFactoryBean, the
scheduler.shutdown(waitForJobsToCompleteOnShutdown) is called in order
to stop the threads of the quartz scheduler and I suppose (without
having checked it) that this destroy is called via
org.springframework.web.context.ContextLoaderListener.contextDestroyed.

So I suppose that if the ContextLoaderListener is used in a
application, the quartz threads of the scheduler in
SchedulerFactoryBean are stopped.
But it seems that some other threads were started when javamelody
called StdSchedulerFactory.getDefaultScheduler(): those threads are
not stopped.
Those threads could certainly be stopped by the application with
StdSchedulerFactory.getDefaultScheduler().shutdown() or better with:

for (Scheduler scheduler :
SchedulerRepository.getInstance().lookupAll())
{ scheduler.shutdown(); }

[burkhard: if some quartz threads are not stopped by the
scheduler.shutdown() above, where these threads come from? Spring?]

In order that the applications using quartz via Spring
SchedulerFactoryBean do not have to call scheduler.shutdown() as above
to stop the threads, I have a proposal:
a javamelody parameter "quartz-default-listener-disabled" can be
added.
If this parameter has the value "true", javamelody would not call
StdSchedulerFactory.getDefaultScheduler() to add the listener, and so
the threads of the default quartz scheduler would not be started. The
only quartz threads will be started by SchedulerFactoryBean.

You can test that by using this jar I have just built:
http://javamelody.googlecode.com/files/javamelody-20101103.jar

and by adding the following for the javamelody filter in your web.xml:
<init-param>
<param-name>quartz-default-listener-disabled</param-name>
<param-value>true</param-value>
</init-param>

If this works for you, I will add that in the next release.

Happy hacking !
Emeric

[burkhard]

Hi Emeric, thank's for your answer/explanation - I'll check it next
week!

Cheers
Burkhard

[MiTaes]

Hi Emeric,

I tried your new jar and can confirm that it works like a charm.
Tomcat does shutdown correctly.

So I guess your assumption of Spring Quartz usage are correct and that
JavaMelody "accidently" initialized schedulers when retrieving quartz
info where Spring only created the schedulers that are configured in
the context.

Thanks for your effort and I am looking forward to the next mavenized
release of JavaMelody.

Cheers and Regards,
Michael

[php...@gmail.com]

We also run into this problem too without Spring Quartz stuff enabled. I think a much simple way  to resolve this was adding a context listener provided by Quartz to web.xml to startup/shutdown.

  <listener>

    <listener-class>org.quartz.ee.servlet.QuartzInitializerListener</listener-class>

  </listener> 

Thanks,

Jeff