FileNotFoundError when TemporaryDirectory tries to cleanup inside a coroutine

[Jack O'Connor]

I have the following test script:

import asyncio

import tempfile

@asyncio.coroutine

def error():

    raise KeyboardInterrupt()

@asyncio.coroutine

def main():

    with tempfile.TemporaryDirectory():

        yield from asyncio.gather(error())  # gather() is important here

loop = asyncio.get_event_loop()

loop.run_until_complete(main())

When I run that, I get several tracebacks in a row, and one of them is inexplicable to me:

Exception ignored in: <generator object main at 0x7fccf7a48ee8>

Traceback (most recent call last):

  File "test.py", line 11, in main

  File "/usr/lib/python3.4/tempfile.py", line 691, in __exit__

  File "/usr/lib/python3.4/tempfile.py", line 697, in cleanup

  File "/usr/lib/python3.4/shutil.py", line 454, in rmtree

  File "/usr/lib/python3.4/shutil.py", line 452, in rmtree

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpl2z0vwdy'

When the TemporaryDirectory context manager exits, it tries to clean up the directory it created. In this case, something appears to have already deleted that directory. There's no other rmdir() or anything like that in the script, so I'm confused. This part of the traceback goes away if I remove the gather() in there and instead yield directly from my error() coro. This also doesn't repro if I throw a RuntimeError instead of a KeyboardInterrupt.

Could gather() be removing a directory? Or maybe making my context manager exit twice?

[Guido van Rossum]

Wow. Good catch! I had four different theories about this, and none of them held up. Two things are suspect: the logic in TemporaryDirectory to ensure cleanup (there are two separate mechanisms), and the fact that KeyboardError is not a subclass of Exception (only of BaseException).

The latter means that many try/except clauses in asyncio don't get triggered, because (intentionally) it uses "except Exception:" instead of a bare except clause. This probably causes the main() generator to be abandoned by the asyncio scheduler, so the context manager's __exit__ isn't called, and instead the cleanup path via the weak reference removes the directory during some GC step. But then at a later GC step the main() generator is finalized, which throws GeneratorExit into the generator, which causes __exit__ to be called -- and it attempts to clean up a second time. I think.

I suspect that the real bug is that TemporaryDirectory assumes that the __exit__ clause is always reached before the finalizer cleans up, and apparently that is violated here, during end-of-program GC. You can put some print calls in to debug this further.

Please do file this as a bug on bugs.python.org!

--Guido

--
--Guido van Rossum (python.org/~guido)

[Jack O'Connor]

You were right about the TemporaryDirectory. There is also an interesting issue with BaseExceptions that's leading to an AttributeError. I filed two bugs:

http://bugs.python.org/issue22427

http://bugs.python.org/issue22428