await process.wait() hangs forever on process created with a new_event_loop

[cheta...@gmail.com]

Hello,

I tried to run the following program with Python-3.5.0b4, and it hangs forever:

--------------------------------------------

import asyncio

async def sleepWithShell(loop):

    process = await asyncio.create_subprocess_shell("sleep 2", loop=loop)

    await process.wait()

    return True

async def sleepWithAsyncio(loop):

    await asyncio.sleep(2, loop=loop)

    return True

def main():

    loop = asyncio.new_event_loop()

    coros = [ sleepWithShell(loop) for i in range(5)]

    results = loop.run_until_complete(asyncio.gather(*coros, loop=loop))

    loop.close()

    print(results)

if __name__ == "__main__":

    main()

----------------------------------------------

In main, if you replace sleepWithShell with sleepWithAsyncio the program will run as expected. I looked through the code, and I now understand that process.wait doesn't work because the child process watcher is not associated with the loop that's passed.

In an ideal world, i'd like to be able to create a loop, use it and close it without changing any global state. It seems like that's possible with some coroutines, but not with create_subprocess_shell.

If passing a custom loop to create_subprocess_shell isn't always supported,  I was wondering if it's possible to check and raise an exception in subprocess.Process.wait instead of just hanging forever.

Thanks,

Chetan

[Victor Stinner]

Hi,

When you have a bug with asyncio, the first step is to enable the debug mode and the logger:
https://docs.python.org/dev/library/asyncio-dev.html#asyncio-debug-mode

What is your OS? For Windows, see:
https://docs.python.org/dev/library/asyncio-subprocess.html#windows-event-loop

Victor

[Guido van Rossum]

Wait, why isn't that just a bug that wait() doesn't know the associated loop?

It seems the child watcher is only associated with the loop on set_event_loop(), which in your example doesn't get called (and it shouldn't need to be called).

I've reproduced your bug; please file a bug per Victor's instructions.

--Guido

--

--Guido van Rossum (python.org/~guido)

[Victor Stinner]

Ah yes, you should call asyncio.get_event_loop() or asyncio.set_event_loop().

Victor

[cheta...@gmail.com]

Filed a bug as requested at http://bugs.python.org/issue24837. Victor, I am running linux (redhat 6).

If this is indeed a bug that will be fixed, i was wondering if the following will also be supported.

-------------------------------------------------------------------------------------

import asyncio

async def sleepWithShell(loop=None):

    process = await asyncio.create_subprocess_shell("sleep 2", loop=loop)

    await process.wait()

    return True

def sleepWithNewLoop():

    """This function uses asyncio in its implementation, but wants the fact                                                                          

    that it uses asyncio to be opaque to the user.                                                                                                   

    i.e, it needs to work both when the user isn't using asyncio at all,                                                                             

    and when the user already has his own loop and is calling this function from a                                                                   

    coroutine in his loop. Is that supported by  asyncio?                                                                                            

    """

    loop = asyncio.new_event_loop()

    result = loop.run_until_complete(sleepWithShell(loop))

    loop.close()

    return result

async def sleepWithMainLoopAndNewLoop():

    result = await sleepWithShell()

    print("sleep with shell in main loop ", result)

    result = sleepWithNewLoop()

    print("sleep with shell in new 'nested' loop ", result)

    return result

def main():

    asyncio.get_event_loop().run_until_complete(sleepWithMainLoopAndNewLoop())

    asyncio.get_event_loop().close()

if __name__ == "__main__":

    main()

-------------------------------------------------------------------------------------

sleepWithNewLoop is a poor example, but there's a lot of functions in a library that I maintain that could run faster if i could use asyncio in the implementation. I want the functions to work without having to require the user to manage an event loop, but I also want them to not affect the user's existing loop if he already has one.

Thanks for your time,

Chetan

[Victor Stinner]

To me it looks wrong to have two event loops per thread. You must get an error, at least in debug mode.

What do you want to do?

Victor

[cheta...@gmail.com]

Hi Victor,

If i use asyncio.sleep instead of create_subprocess_shell, it works find even though i have two event loops. There are no errors; the only warning i get is about the task taking too long (slow_callback_duration is 0.1 while my task takes 2 seconds so the warning is expected).

Here's the program (with two event loops) that works

----------------------------------------------------

import asyncio

import logging

async def sleepWithAsyncio(loop=None):

    await asyncio.sleep(2, loop=loop)

    return True

def sleepWithNewLoop():

    loop = asyncio.new_event_loop()

    result = loop.run_until_complete(sleepWithAsyncio(loop))

    loop.close()

    return result

  

async def sleepWithMainLoopAndNewLoop():

    result = await sleepWithAsyncio()

    print("sleep with asyncio in main loop ", result)

    result = sleepWithNewLoop()

    print("sleep with asyncio in new 'nested' loop ", result)

    return result

def main():

    logging.basicConfig(level=logging.DEBUG)

    asyncio.get_event_loop().run_until_complete(sleepWithMainLoopAndNewLoop())

    asyncio.get_event_loop().close()

  

if __name__ == "__main__":

    main()

-----------------------------------------------------

Thanks,

Chetan

[Guido van Rossum]

Victor, I agree with Chetan -- each event loop has its own state and this normally works fine -- it is a problem specific to the subprocess watcher architecture. Maybe we can get its original author interested again?