[Django] #27890: runtests.py cleanup exception on Python 3.6
Django
---------------------------------------------+------------------------
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: master
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------------+------------------------
Per [https://github.com/django/django/pull/8127#issuecomment-283322945
Tim's suggestion], I've looked into a cleanup exception that happens when
running tests on Python 3.6:
{{{
$ ./tests/runtests.py basic
Testing against Django installed in '/home/tim/code/django/django' with up
to 3 processes
....
Destroying test database for alias 'other'...
Traceback (most recent call last):
File "/opt/python3.6.0/lib/python3.6/multiprocessing/util.py", line 254,
in _run_finalizers
finalizer()
File "/opt/python3.6.0/lib/python3.6/multiprocessing/util.py", line 186,
in __call__
res = self._callback(*self._args, **self._kwargs)
File "/home/tim/.virtualenvs/django36/lib/python3.6/shutil.py", line
465, in rmtree
onerror(os.lstat, path, sys.exc_info())
File "/home/tim/.virtualenvs/django36/lib/python3.6/shutil.py", line
463, in rmtree
orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory:
'/tmp/django_k0xziymh/pymp-i4s112bj'
}}}
What I found out:
- `multiprocessing` is involved. Indeed, running with `--parallel=1` does
not trigger this error.
- Jinja2 is involved. Running tests without Jinja2 installed does not
trigger this error.
What happens:
1. `multiprocessing` registers `atexit` handler.
1. `runtests.py` creates a temp dir (e.g. `/tmp/django_k0xziymh`) and
registers `atexit` handler to remove it.
1. `multiprocessing` creates a temp dir (e.g. `/tmp/django_k0xziymh/pymp-
i4s112bj`) which will be deleted in the handler it registered earlier.
1. Tests happen.
1. `runtests.py` exit handler deletes the temp dir.
1. `multiprocessing` exit handler tries to delete the inner temp dir but
it is already gone.
On earlier Python versions 1 and 2 are swapped and 5 and 6 are swapped, so
the error does not happen.
Jinja2 is imported by a chain of imports starting with `from django.test
import TestCase, TransactionTestCase` (full chain below).
On Python 3.6 Jinja2
[https://github.com/pallets/jinja/blob/master/jinja2/__init__.py#L74-L78
patches async support] which eventually imports `multiprocessing.util`
which registers the exit handler.
Possible solutions:
- Create the temp dir before importing `django` modules.
- Move some imports in `runtests.py` into functions, thereby delaying the
indirect import of Jinja2.
- Move some imports somewhere else into functions to break the import
chain.
- ...
The import chain leading up to the import of `multiprocessing.util` (with
uninteresting import machinery stack frames removed):
{{{
File "./runtests.py", line 18, in <module>
from django.test import TestCase, TransactionTestCase
File "/home/vytis/src/django/django/test/__init__.py", line 5, in
<module>
from django.test.client import Client, RequestFactory
File "/home/vytis/src/django/django/test/client.py", line 12, in
<module>
from django.core.handlers.base import BaseHandler
File "/home/vytis/src/django/django/core/handlers/base.py", line 7, in
<module>
from django.urls import get_resolver, set_urlconf
File "/home/vytis/src/django/django/urls/__init__.py", line 1, in
<module>
from .base import (
File "/home/vytis/src/django/django/urls/base.py", line 8, in <module>
from .exceptions import NoReverseMatch, Resolver404
File "/home/vytis/src/django/django/urls/exceptions.py", line 1, in
<module>
from django.http import Http404
File "/home/vytis/src/django/django/http/__init__.py", line 5, in
<module>
from django.http.response import (
File "/home/vytis/src/django/django/http/response.py", line 13, in
<module>
from django.core.serializers.json import DjangoJSONEncoder
File "/home/vytis/src/django/django/core/serializers/__init__.py", line
23, in <module>
from django.core.serializers.base import SerializerDoesNotExist
File "/home/vytis/src/django/django/core/serializers/base.py", line 6,
in <module>
from django.db import models
File "/home/vytis/src/django/django/db/models/__init__.py", line 3, in
<module>
from django.db.models.aggregates import * # NOQA
File "/home/vytis/src/django/django/db/models/aggregates.py", line 5, in
<module>
from django.db.models.expressions import Func, Star
File "/home/vytis/src/django/django/db/models/expressions.py", line 6,
in <module>
from django.db.models import fields
File "/home/vytis/src/django/django/db/models/fields/__init__.py", line
11, in <module>
from django import forms
File "/home/vytis/src/django/django/forms/__init__.py", line 6, in
<module>
from django.forms.boundfield import * # NOQA
File "/home/vytis/src/django/django/forms/boundfield.py", line 5, in
<module>
from django.forms.widgets import Textarea, TextInput
File "/home/vytis/src/django/django/forms/widgets.py", line 21, in
<module>
from .renderers import get_default_renderer
File "/home/vytis/src/django/django/forms/renderers.py", line 11, in
<module>
from django.template.backends.jinja2 import Jinja2
File "/home/vytis/src/django/django/template/backends/jinja2.py", line
1, in <module>
import jinja2
File "/home/vytis/src/env/django-py36/lib/python3.6/site-
packages/jinja2/__init__.py", line 81, in <module>
_patch_async()
File "/home/vytis/src/env/django-py36/lib/python3.6/site-
packages/jinja2/__init__.py", line 77, in _patch_async
from jinja2.asyncsupport import patch_all
File "/home/vytis/src/env/django-py36/lib/python3.6/site-
packages/jinja2/asyncsupport.py", line 13, in <module>
import asyncio
File "/opt/python/lib/python3.6/asyncio/__init__.py", line 21, in
<module>
from .base_events import *
File "/opt/python/lib/python3.6/asyncio/base_events.py", line 17, in
<module>
import concurrent.futures
File "/opt/python/lib/python3.6/concurrent/futures/__init__.py", line
17, in <module>
from concurrent.futures.process import ProcessPoolExecutor
File "/opt/python/lib/python3.6/concurrent/futures/process.py", line 55,
in <module>
from multiprocessing.connection import wait
File "/opt/python/lib/python3.6/multiprocessing/connection.py", line 23,
in <module>
from . import util
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27890>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* component: Testing framework => Core (Other)
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:1>
Django
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Claude Paroz):
Another option might be to register a custom function that wraps
`shutil.rmtree` and catches the exception.
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:2>
Django
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Vytis Banaitis):
Replying to [comment:2 Claude Paroz]:
> Another option might be to register a custom function that wraps
`shutil.rmtree` and catches the exception.
It's possible, but would require monkey-patching `multiprocessing`.
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:3>
Django
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Claude Paroz):
Replying to [comment:3 Vytis Banaitis]:
> Replying to [comment:2 Claude Paroz]:
> > Another option might be to register a custom function that wraps
`shutil.rmtree` and catches the exception.
> It's possible, but would require monkey-patching `multiprocessing`.
Really? My idea was something like:
{{{
def custom_delete(tmpdir):
try:
shutil.rmtree(tmpdir)
except FileNotFoundError:
pass
atexit.register(custom_delete, TMPDIR)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:4>
Django
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Vytis Banaitis):
Replying to [comment:4 Claude Paroz]:
> Really? My idea was something like:
>
> {{{
> def custom_delete(tmpdir):
> try:
> shutil.rmtree(tmpdir)
> except FileNotFoundError:
> pass
>
> atexit.register(custom_delete, TMPDIR)
> }}}
Removal of TMPDIR happens without error.
The error is raised by `multiprocessing` exit handler which tries to
delete a directory inside of TMPDIR.
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:5>
Django
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Claude Paroz):
Oh, now I see, sorry for the misunderstanding.
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:6>
Django
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* version: master => 1.11
* severity: Normal => Release blocker
Comment:
I looked a little and didn't see an obvious solution besides some import
rearranging as suggested in the ticket description.
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:7>
Django
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/8198 PR]. Not sure if it's ideal,
but good enough for now?
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:8>
Django
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham):
Rather than the rearranging imports approach of the first PR, an
alternative [https://github.com/django/django/pull/8210 PR] deletes
multiprocessing's temporary directory removal handler to avoid the error.
Going forward, patching cpython to ignore the error might be an option:
{{{ #!diff
diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py
index 1a2c0db..6843d09 100644
--- a/Lib/multiprocessing/util.py
+++ b/Lib/multiprocessing/util.py
@@ -113,7 +113,7 @@ def get_temp_dir():
import shutil, tempfile
tempdir = tempfile.mkdtemp(prefix='pymp-')
info('created temp directory %s', tempdir)
- Finalize(None, shutil.rmtree, args=[tempdir], exitpriority=-100)
+ Finalize(None, shutil.rmtree, args=[tempdir],
kwargs={'ignore_errors': True}, exitpriority=-100)
process.current_process()._config['tempdir'] = tempdir
return tempdir
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:9>
Django
Reporter: Vytis Banaitis | Owner: nobody
Component: Core (Other) | Version: 1.11
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham <timograham@…>):
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"0c6c859d4edf6e462d000da21b4fa5009cb2696f" 0c6c859]:
{{{
#!CommitTicketReference repository=""
revision="0c6c859d4edf6e462d000da21b4fa5009cb2696f"
Fixed #27890 -- Fixed FileNotFoundError cleanup exception in runtests.py
on Python 3.6+.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:10>
Django
Reporter: Vytis Banaitis | Owner: nobody
Component: Core (Other) | Version: 1.11
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"e0ddfa3af30e8e0328cef17fd424c4337efec58e" e0ddfa3a]:
{{{
#!CommitTicketReference repository=""
revision="e0ddfa3af30e8e0328cef17fd424c4337efec58e"
[1.11.x] Fixed #27890 -- Fixed FileNotFoundError cleanup exception in
runtests.py on Python 3.6+.
Backport of 0c6c859d4edf6e462d000da21b4fa5009cb2696f from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:11>