threadpool.max_threads

[andrea]

How can I initialize the scheduler with a threadpool.max_threads > 20 and how much is the max value for thi parameter?
Maximum number of persistent threads in the pool does mean maximum number of scheduled jobs or maximum number of jobs running at the same time?

thank you

[Alex Grönholm]

Let's say that if your thread count climbs to over 100, you likely have
a design error in your application.
I recall seeing a test where a couple thousand threads in a single
process slowed the system to a grinding halt.
That means the system will probably hang before you reach the hard
limit, whatever that is.

And yes, maximum number of threads in the pool does mean the maximum
number of jobs running at the same time.
It'd be insane to keep a thread allocated to an idle job.

[andrea]

So if one has to schedule hundreds of scripts (with execution time that spans to several minute) with triggers time that can overlap, apscheduler may not be appropriate? Or am I missing the point?

[Alex Grönholm]

27.06.2012 16:11, andrea kirjoitti:

So if one has to schedule hundreds of scripts (with execution time that spans to several minute) with triggers time that can overlap, apscheduler may not be appropriate? Or am I missing the point?

How many concurrently executing callables do you think you'll have? What is the use case anyway? The threading limitations do not come from APScheduler, but the OS and Python itself.

[andrea]

Users can schedule their own jobs, choosing among a set of data cruncher, setting some parameters and a cron-like schedule time.
Although some limitations must be posed in some way, the number of concurrent jobs is not so obviously predictable. But I naively did believe to be far away from reaching the maximum threads limit. Clearly I was wrong. Anyway, how can I set the threadpool.max_threads in APScheduler?
I'm trying this:
Scheduler(gconfig = {'apscheduler.threadpool.max_threads':50})
But it doesn't seem to affect the max number of threads.

[Alex Grönholm]

27.06.2012 17:14, andrea kirjoitti:

Users can schedule their own jobs, choosing among a set of data cruncher, setting some parameters and a cron-like schedule time.
Although some limitations must be posed in some way, the number of concurrent jobs is not so obviously predictable. But I naively did believe to be far away from reaching the maximum threads limit. Clearly I was wrong. Anyway, how can I set the threadpool.max_threads in APScheduler?
I'm trying this:
Scheduler(gconfig = {'apscheduler.threadpool.max_threads':50})
But it doesn't seem to affect the max number of threads.

That should be the correct syntax. If it doesn't work, I'll have to investigate and find out why.