[Errno 9] Bad file descriptor

[Stefan]

Hello all,

For some reasons, IOLoop.current().run_sync() starts raising next error: OSError: [Errno 9] Bad file descriptor

Some days ago it worked fine. I'm using Mac Os X.

Full trace:

Traceback (most recent call last):

  File "/Library/Python/2.7/site-packages/celery/task/trace.py", line 233, in trace_task

    R = retval = fun(*args, **kwargs)

  File "/Library/Python/2.7/site-packages/celery/task/trace.py", line 420, in __protected_call__

    return self.run(*args, **kwargs)

  File "/Users/Stefan/Projects/lamoda/textro-a/auth/deferred_tasks.py", line 25, in sendAdminInvites

    confirmation_id = IOLoop.current().run_sync(Confirmation.create)

  File "/Library/Python/2.7/site-packages/tornado/ioloop.py", line 365, in run_sync

    self.start()

  File "/Library/Python/2.7/site-packages/tornado/ioloop.py", line 646, in start

    event_pairs = self._impl.poll(poll_timeout)

  File "/Library/Python/2.7/site-packages/tornado/platform/kqueue.py", line 67, in poll

    kevents = self._kqueue.control(None, 1000, timeout)

None: [Errno 9] Bad file descriptor

Thank you.

[Ben Darnell]

You haven't given us much to go on, but way this can happen is if the IOLoop is created in a parent process and then inherited by a child process.  You should ensure that all your child processes are created before doing anything that touches IOLoop.instance() or IOLoop.current() (it's fine to use IOLoops other than the default one, though).

-Ben

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Stefan]

Hello Ben,

This question is regarding to this question: https://groups.google.com/forum/#!topic/python-tornado/bvwtZkLo7ik

Reusing async code in Celery.

The idea is to use code in another places (e.g. deferred task on celery). I have a class, object is creating by loading data from Redis. This creation is use async style: @gen.coroutine and async library to get data from Redis ... But at the same time i need to use this code on Celery. Celery is running in sync mode.

E.g.: I have some deferred task.

@celery.task

def some_deferred_task():

  obj = ILoop.instance().run_async(SomeClass.async_method)  # In SomeClass, async_method is using @gen.coroutine.

Celery is running in another process. To call this task, in Tornado process i use next: some_deferred_task.delay()

It was working previously, but yesterday i checked, it started raising Exception (((( Actually i don't after what changes it started raise the exception.

Then i write next code, which is working now:

@celery.task

def some_deferred_task():

  def _async_method(ioloop, callback):

     ioloop.current().run_sync(SomeClass.async_method)

    callback()

  ioloop = IOLoop()

  _async_method(ioloop=ioloop, callback=ioloop.stop)

  ioloop.start() 

I think i missing something, but can't get it. 

Thank you.

вторник, 27 августа 2013 г., 5:31:49 UTC+4 пользователь Ben Darnell написал:

[Ben Darnell]

On Mon, Aug 26, 2013 at 10:21 PM, Stefan <stefan....@gmail.com> wrote:

Then i write next code, which is working now:

@celery.task

def some_deferred_task():

  def _async_method(ioloop, callback):

     ioloop.current().run_sync(SomeClass.async_method)

    callback()

  ioloop = IOLoop()

  _async_method(ioloop=ioloop, callback=ioloop.stop)

  ioloop.start() 

If this is working now it's just by coincidence; this has the same problem as before.  The IOLoop you just created is not yet "current" when _async_method is run, so you're calling run_sync on the default IOLoop again.  (The newly-created IOLoop is started and then immediately stopped because you've already called its stop() method).

Do you have any code that might be calling IOLoop.close()?  That's another way this error could occur.

-Ben

[Stefan]

Thank you for reply, Ben.

And what is the best practice to use async code in sync method? How can i use run_sync in Celery deferred task method?

Should i start IOLoop when starting celery? E.g. rewrite Celery app class init and add ioloop instance there ... and start it there.

Then i will be able to call IOLoop.current().run_sync(async_method).

[Ben Darnell]

On Tue, Aug 27, 2013 at 12:39 AM, Stefan <stefan....@gmail.com> wrote:

Thank you for reply, Ben.

And what is the best practice to use async code in sync method? How can i use run_sync in Celery deferred task method?

run_sync is the right way to do this, it's just that something is messing up the shared IOLoop in your code.  For extra isolation, you can follow the pattern in the synchronous version of Tornado's HTTPClient and create a new IOLoop for each task:

https://github.com/facebook/tornado/blob/master/tornado/httpclient.py#L43

 

Should i start IOLoop when starting celery? E.g. rewrite Celery app class init and add ioloop instance there ... and start it there.

Then i will be able to call IOLoop.current().run_sync(async_method).

I wouldn't recommend this.  A fully-asynchronous Celery worker (so that you never have to use run_sync) would be useful, but if you're using a synchronous worker it's probably better to just create an IOLoop for each task instead of using a shared IOLoop that's not actually running all the time.

-Ben

[Stefan]

Thank you Ben

Good advise.

Yesterday, i tried next code: IOLoop.instance().add_callback()

This code will execute all async code in deferred mode: e.g. will load any data from Redis asynchronously. 

And then, if there is a need (e.g. email send) i can use Celery, but i will pass there already loaded data (e.g. dict())

I also can use ioLoop.PeriodicCallback for periodic tasks. 

Tornado is really good, everything you need already has it ))))

Thank you.

среда, 28 августа 2013 г., 4:02:00 UTC+4 пользователь Ben Darnell написал: