Jobcenter max concurrent jobs with the same name

[raphy]

I have scheduled multiple instances of a given object in a given
project, each instance with different parameters. They were scheduled
to run in the same time.

I have found that only 2 jobs have run concurrently and the others
waited for the first to end, when one ended a waiting one has started.
Is it defined somewhere? can I control it?

Thanks

Raphy

[Greg Schueler]

Hi Raphy,

I haven't seen this behavior before, but I will investigate it.

Do you have any sample job.xml files that I could use to reproduce the issue?

--
Greg Schueler

ControlTier Software, Inc
gr...@controltier.com
650-292-9660x709

http://www.controltier.com

[raphy zarecki]

Hi Greg

I think that trying to run the same command on the same type on different objects concurrently will do the trick, just make sure that the command it self will take long enough time.

[Greg Schueler]

I am not able to reproduce this.

I made a trivial command that sleeps for a period of time, and was able to execute two jobs simultaneously on different objects of the same type.  (both manually and using scheduled jobs.)

Are you possibly executing a command that performs some sort of change on the server data model (e.g. Change-Dependencies, as invoked by the Update workflow)? 

If that is true, then the effect you are seeing could be due to a race condition happening at the server:  only one thread/request can write to the data model at a time, so other requests would wait until any other transaction is complete . 

[RaphyZ]

I am sorry if I was not clear, however 2 instances did work
concurrently, however more than that waited in a way that only 2 will
work concurrently. I was not doing nothing related to changing the server.


Try the same test with 4 or 5 concurrent commands in the job center.


Thanks

> gr...@controltier.com <mailto:gr...@controltier.com>

> 650-292-9660x709
>
> http://www.controltier.com
>
>
>
>
>
>
>
>
> --
> Greg Schueler
>
> ControlTier Software, Inc

> gr...@controltier.com <mailto:gr...@controltier.com>
> 650-292-9660x709
>
> http://www.controltier.com
>
> >

[RaphyZ]

Hi Greg


I have prepared a simple project that contains a single object which is
an updater that belongs to the local host that has a command named
Sleep that waits for 1 minute.


The command also echos the date of the start of sleep and the end of the
sleep to /opt/ctier/temp.txt


I created a job with this project and this command and run it multiple
times (more than 3 - I suggest to run it 7 times).


If no jobs are currently running in job center, you can easily see that
it runs only 3 jobs concurrently and starts the 4th when the first ends
and so on.


Please try it, and let me know if you still have problems recreating it.


This is a show stopper for us, when thinking about using job-center!


Let me know how to proceed from here.



Thanks


Raphy

[Anthony Shortland]

Hi Raphy ... what version of ControlTier are you using?

<TestJobCenterLimit.20090624121258.par>

Anthony Shortland

[ Professional Services | ControlTier Software, Inc. | mobile: 650.215.3117 aim: anthony....@me.com yahoo: anthony.shortland irc.freenode.net: #controltier skype: anthony.shortland]

[Greg Schueler]

Hi Raphy, I found the culprit in this situation:

short answer: you should modify the "quartz.properties" file located at $CTIER_ROOT/pkgs/jetty-6.1.14/webapps/jobcenter/WEB-INF/classes/quartz.properties and restart the jetty server.

change this line:

org.quartz.threadPool.threadCount = 3

Set the threadCount value to the max number of threads you want to run concurrently.

http://wiki.opensymphony.com/display/QRTZ1/ConfigThreadPool

Jobcenter uses the Quartz library for job scheduling, and it seems the "SimpleThreadPool" implementation that is used by default is pretty basic, and does not grow on demand. Clearly 3 is a pretty small number, so I will definitely update the default value that we use in the quartz.properties at installation time.  Quartz does not apparently ship with a flexible threadpool implementation, so the best solution is to change that number to something slightly higher than you anticipate needing.

This is also clearly a useful FAQ item.

Greg

650-292-9660x709

http://www.controltier.com

[RaphyZ]

Thanks!!!!

Raphy



Greg Schueler wrote:

> Hi Raphy, I found the culprit in this situation:
>
> short answer: you should modify the "quartz.properties" file located
> at
> $CTIER_ROOT/pkgs/jetty-6.1.14/webapps/jobcenter/WEB-INF/classes/quartz.properties
> and restart the jetty server.
>
> change this line:
>

> *org.quartz.threadPool.threadCount = 3*

> > <mailto:raphy....@gmail.com

> <mailto:raphy....@gmail.com>>> wrote:
> >
> >
> > I have scheduled multiple instances of a given object in a given
> > project, each instance with different parameters. They were
> scheduled
> > to run in the same time.
> >
> > I have found that only 2 jobs have run concurrently and the
> others
> > waited for the first to end, when one ended a waiting one
> has started.
> > Is it defined somewhere? can I control it?
> >
> > Thanks
> >
> > Raphy
> >
> >
> >
> >
> > --
> > Greg Schueler
> >
> > ControlTier Software, Inc
> > gr...@controltier.com <mailto:gr...@controltier.com>

> <mailto:gr...@controltier.com <mailto:gr...@controltier.com>>

[Greg Schueler]

Follow up: 

I modified the default threadCount to be 20, which will be used in the next release (3.4.6).  Bug report item: http://sourceforge.net/tracker/index.php?func=detail&aid=2811907&group_id=151079&atid=779847

Added a FAQ item to the wiki: 5.1.3 Jobcenter threadCount and concurrency

650-292-9660x709

http://www.controltier.com