How to shutdown the event loop correctly?

[Florian Rüchel]

So I have already posted some mails to this list and here is one more. This time it is about cleanly exiting the event loop. The problem is pretty simple: I run my event loop using run_forever and it really does run forever, in an infinite loop until the user requests a stop. The stop is generally requested by hitting ^C, i.e. raising a KeyboardInterrupt. Contrary to a single-threaded application, an event loop that runs everything will always receive the KeyboardInterrupt exception (unless caught by a task/coroutine). Because of that I literally didn't include any handling inside the running loop for it. Instead I have code like this:

try:

    loop.run_forever()

except KeyboardInterrupt:

    pass

finally:

    loop.run_until_complete(shutdown())

Where shutdown would be a coroutine that handles cleaning up all still-running tasks by cancelling them (I fact there is a single task that is cancelled and cancels all tasks it started in turn). This way, here is the only place where I catch a KeyboardInterrupt since I assumed it would travel up the stack without anything catching it. And that certainly seems to be the case: Hitting ^C actually exits the loop and starts the shutdown coroutine. But here is where it gets weird: The shutdown fails because a KeyboardInterrupt is raised, even though I only sent one and already caught that! And even more weird: The stack trace actually looks like one loop was running inside the other:

Traceback (most recent call last):

  File "/home/javex/mypkg/__init__.py", line 261, in run

    event_loop.run_until_complete(shutdown())

  File "/home/javex/.virtualenvs/misc/lib/python3.3/site-packages/asyncio/base_events.py", line 203, in run_until_complete

    self.run_forever()

  ... 

  File "/home/javex/.virtualenvs/misc/lib/python3.3/site-packages/asyncio/futures.py", line 243, in result

    raise self._exception

  File "/home/javex/mypkg/__init__.py", line 245, in run

    asyncio.get_event_loop().run_forever()

  ...

  File "/home/javex/.virtualenvs/misc/lib/python3.3/site-packages/asyncio/unix_events.py", line 487, in _start

    universal_newlines=False, bufsize=bufsize, **kwargs)

  File "/usr/lib64/python3.3/subprocess.py", line 819, in __init__

    restore_signals, start_new_session)

  File "/usr/lib64/python3.3/subprocess.py", line 1409, in _execute_child

    part = _eintr_retry_call(os.read, errpipe_read, 50000)

  File "/usr/lib64/python3.3/subprocess.py", line 479, in _eintr_retry_call

    return func(*args)

KeyboardInterrupt

Full Trace

If you look at the stack trace, you can see that we are inside the run_until_complete part (above inside the finally block) and that it in turn winds up inside the run_forever part from above. Of course the bottom part of the stack trace always varies as it depends on where the function is currently working. However, the part above is always the same: It looks like one loop runs inside the other.

So here I am stuck. I don't know how to handle this cleanly. Previously I did try a solution similar to the one described in Detect exceptions not consumed where I forced myself to wrap any task in it. That worked but it was error prone and felt dirty. Instead, I now handle exception for each task individually which means I can react to them much better. However, additionally to handling normal exceptions, I handled a KeyboardInterrupt here, calling loop.stop(). However, that would only trigger when inside one of those tasks, not when running polling of the loop. So I had to handle this additionally similar to the above try-except-finally block which essentially meant having two different code paths handling the same thing while it feels like there should be a central point.

After reading the PEP; this list and searching for a solution, I could not find anything on it. So here it is: How do I exit a running loop cleanly by sending a KeyboardInterrupt? Note that I am aware of the Example: Set signal handlers for SIGINT and SIGTERM but that is Unix specific and my application should also run on Windows in the future. Otherwise this would be the perfect solution (btw: Why isn't Windows supported? From the signal documentation it seems that some signals (including SIGINT) can be caught on Windows as well.).

Thanks in advance for any suggestion on how to acheive the desired behavior.

Regards,

Florian

[Guido van Rossum]

I suspect that you're seeing a combination of the real traceback and a saved traceback due to the interrupted event handler. KeyboardInterrupt is a "BaseException" and those are generally ignored by the asyncio library (maybe we can do something better?).

I would recommend using the signal handler on UNIX and the other approach on Windows (the add_signal_handler() call raises ValueError on Windows). Signal handling on Windows is a horrible emulation in the C runtime and does not have the same semantics as on UNIX, that's why we couldn't implement add_signal_handler() there.

--
--Guido van Rossum (python.org/~guido)