Tornado's @run_on_executor not playing nicely with ProcessPoolExecutor?

[Hussein Elgridly]

Hi Tornadists,

I'm running Tornado 4.1 on Python 2.7, and having trouble with the @run_on_executor decorator. I've written a further decorator that automagically creates an object with the io_loop and executor attributes on it, and it works great when the executor I use is a ThreadPoolExecutor - but I run into pickling errors when I use a ProcessPoolExecutor.

I've cut back my code to a minimal example and posted it on Gist:

https://gist.github.com/helgridly/0b60c4768184a915e71b

If I do a GET on /thread, it works fine; if I hit /process, the request hangs and the Tornado process prints this traceback:

Traceback (most recent call last):

  File "/usr/lib/python2.7/multiprocessing/queues.py", line 266, in _feed

    send(obj)

PicklingError: Can't pickle <type 'thread.lock'>: attribute lookup thread.lock failed

If I set up Tornado to use HTTPS, the PicklingError is on ssl.SSLContext instead.

Is there anything I can do about this, or am I stuck using thread pools? Presumably there's something in the @gen.coroutine machinery that doesn't like being pickled here.

Thanks,

-Hussein

[Ben Darnell]

This is more a limitation of ProcessPoolExecutor: it requires that you be very careful in all the arguments to functions that you call on it to make sure you don't try to pass anything non-pickleable. Unfortunately run_on_executor makes this essentially impossible, since it requires that the function have a self argument which refers to an executor and an IOLoop. I think run_on_executor only works with thread pools (and DummyExecutor).

Even with thread pools, I now have doubts about whether run_on_executor is a good pattern. Its main virtue is that it works with both coroutine and callback styles, but if what you want is a coroutine I think it's probably better to just use the executor directly:

  class process(RequestHandler):

    def initialize(self):

      self.executor = executors['ps']

    @gen.coroutine

    def get(self):

      ret = yield self.executor.submit(self.sleep, 5)

      self.write(ret)

    # staticmethod avoids the self parameter and makes it safe for process pools.

    @staticmethod

    def sleep(secs):

      time.sleep(secs)

-Ben

        

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Hussein Elgridly]

I thought that might be the answer :( I liked the syntactic neatness of simply being able to decorate functions with the executor they should run on, but even if I ditch @run_on_executor pickle still doesn't handle decorators.

My investigations suggest that dill/pathos.multiprocessing might work as a drop-in replacement for pickle/multiprocessing, since dill serializes many more things than pickle, but I don't know how well dill and Tornado play together. I'll report back if I ever get the time to fiddle and find out, but don't hold your breath.

Thanks,

-H