Can ProcessPoolExecutor be used with APScheduler?

[Brent Miller]

I have a use case where I need to run a couple of cpu intensive jobs on regular intervals so I'd like to have each job run in its own process instead of a thread. I'm using Python 2.7, so I've installed futures. Just as a test, I've created a simple script:

import datetime

from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor

from apscheduler.scheduler import Scheduler

def testjob():

    print "This is a test - %s" % datetime.datetime.now()

if __name__ == '__main__':

    tp = ProcessPoolExecutor(12)

    

    sched = Scheduler({"apscheduler.threadpool":tp})

    sched.start()

    sched.add_cron_job(testjob, second="*/5")

    

    raw_input("Press enter to quit\n\n")

    

    sched.shutdown()

However, when I run it I get:

Traceback (most recent call last):

  File "/usr/lib/python2.7/multiprocessing/queues.py", line 266, in _feed

    send(obj)

PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

If I replace ProcessPoolExecutor with ThreadPoolExecutor everything works fine. I'm guessing that APScheduler takes the function that I pass it in add_cron_job and is making it a member of a class, which the multiprocessing library doesn't handle. Is this something that only works in Python 3? Is there a way to get this to work in Python 2.7? (Some of the libraries that I use haven't been ported to Python 3 yet.)

Thanks,

Brent

[Alex Grönholm]

Truthfully, I hadn't tested it with ProcessPoolExecutor, I had just assumed that it would work since it uses the same API. Clearly I was mistaken. I'm not aware of anything that would make this a py3k-only feature. I'll add this to my TODO when working on future versions of APScheduler.

[Alex Grönholm]

25.08.2012 01:30, Brent Miller kirjoitti:

I just revisited the issue due to 3.0 development, and I concluded that the reason for this is because I'm submitting a bound method (Scheduler._run_job()) which is unpickleable on Python < 3.3 and as such, cannot be submitted to a ProcessPoolExecutor(). I think I can work around this, but it will require some refactoring.

[Christopher Cunningham]

Hi Alex. I didn't see this on the 3.0 list. Is this planned?

[Alex Grönholm]

20.05.2014 17:49, Christopher Cunningham kirjoitti:

Hi Alex. I didn't see this on the 3.0 list. Is this planned?

Yes, it already works on the development version.

On Saturday, July 27, 2013 5:06:47 PM UTC-4, Alex Grönholm wrote:

I just revisited the issue due to 3.0 development, and I concluded that the reason for this is because I'm submitting a bound method (Scheduler._run_job()) which is unpickleable on Python < 3.3 and as such, cannot be submitted to a ProcessPoolExecutor(). I think I can work around this, but it will require some refactoring.

 

--
You received this message because you are subscribed to the Google Groups "APScheduler" group.
To unsubscribe from this group and stop receiving emails from it, send an email to apscheduler...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.