APScheduler in BlockingScheduler mode, adding jobs

[Mark Grandi]

Hello,

I'm interested in using APScheduler in a project, one of which involves a webserver using web.py. We are using apache, and apache of course uses lots of child processes to serve web requests, and each of those will have a python process as well, running different instances of the server.

Point is, I can't use APScheduler embedded in the server application as there would be multiple schedulers using the same job store, which sounds like a big nono.

So what i want is to use a simple python script/server that just runs APScheduler, using the BlockingScheduler to do jobs, and when my server needs to add a job, it does <something> and the separate python process only running APScheduler will pick up the new job and run it when its time, etc

The problem is, what is <something>? Since BlockingScheduler is ...blocking of course, how do I get it to receive new jobs?  Is it possible to have the web.py app add new jobs to the database/job store, and then the python script running only APScheduler will pick up on those and handle them correctly? Is there a timeout for the start() call where i can then check for new jobs somewhere, add them and then resume blocking? I searched around and haven't seen a solution for this use case, which is weird given the presence of BlockingScheduler, since I would of thought someone would of wanted to dynamically add jobs to a BlockingScheduler...

~Mark

[Alex Grönholm]

BlockingScheduler is only useful if you're doing nothing else in the process. Since you also need to receive new jobs from an external process, I suggest you use BackgroundScheduler or something and have a web server running which receives them.

~Mark

--
You received this message because you are subscribed to the Google Groups "APScheduler" group.
To unsubscribe from this group and stop receiving emails from it, send an email to apscheduler...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[polz...@gmail.com]

Building own web-server just to add jobs seems like an overhead, especially if it is possible to add jobs to a store directly. So, the question is: will BlockingScheduler pick up new jobs that were added by different process via Store class (RedisStore for example)?

Thanks,

Sergey

[Alex Grönholm]

16.12.2014, 15:29, polz...@gmail.com kirjoitti:

Building own web-server just to add jobs seems like an overhead, especially if it is possible to add jobs to a store directly. So, the question is: will BlockingScheduler pick up new jobs that were added by different process via Store class (RedisStore for example)?

I think it should be safe to add new jobs directly, but you have to notify the scheduler via scheduler.wakeup() so it knows to scan the job store and determine when it should wake itself up.

[polz...@gmail.com]

On Tuesday, December 16, 2014 2:50:11 PM UTC+1, Alex Grönholm wrote:

16.12.2014, 15:29, polz...@gmail.com kirjoitti:

Building own web-server just to add jobs seems like an overhead, especially if it is possible to add jobs to a store directly. So, the question is: will BlockingScheduler pick up new jobs that were added by different process via Store class (RedisStore for example)?

I think it should be safe to add new jobs directly,

What about changing/deleting jobs?

 

but you have to notify the scheduler via scheduler.wakeup() so it knows to scan the job store and determine when it should wake itself up.

 

Hm, so there is no code that can re-scan job store periodically. Is there a recommended approach for running APScheduler in a distributed way, running scheduler from one process and managing jobs from another?

Thanks,

Sergey

 

[Alex Grönholm]

16.12.2014, 16:25, polz...@gmail.com kirjoitti:

On Tuesday, December 16, 2014 2:50:11 PM UTC+1, Alex Grönholm wrote:

16.12.2014, 15:29, polz...@gmail.com kirjoitti:

Building own web-server just to add jobs seems like an overhead, especially if it is possible to add jobs to a store directly. So, the question is: will BlockingScheduler pick up new jobs that were added by different process via Store class (RedisStore for example)?

I think it should be safe to add new jobs directly,

What about changing/deleting jobs?

If the scheduler checks out and job and it is deleted before the scheduler saves the changes back to the job store, a JobLookupError is raised which is left uncaught in the scheduler thread. This will (with the current codebase) break the scheduler. It's a race condition that would be rarely triggered. The job modification/removal code in _process_jobs() could be made more fault tolerant.

Another potential issue is updating the next run time of a job. The scheduler needs to be notified of this if the run time is moved to a sooner point in time so it can adjust its wakeup time accordingly.

 

but you have to notify the scheduler via scheduler.wakeup() so it knows to scan the job store and determine when it should wake itself up.

 

Hm, so there is no code that can re-scan job store periodically. Is there a recommended approach for running APScheduler in a distributed way, running scheduler from one process and managing jobs from another?

Thanks,

Sergey