[Django] #27890: runtests.py cleanup exception on Python 3.6
Django
---------------------------------------------+------------------------
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: master
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------------+------------------------
Per [https://github.com/django/django/pull/8127#issuecomment-283322945
Tim's suggestion], I've looked into a cleanup exception that happens when
running tests on Python 3.6:
{{{
$ ./tests/runtests.py basic
Testing against Django installed in '/home/tim/code/django/django' with up
to 3 processes
....
Destroying test database for alias 'other'...
Traceback (most recent call last):
File "/opt/python3.6.0/lib/python3.6/multiprocessing/util.py", line 254,
in _run_finalizers
finalizer()
File "/opt/python3.6.0/lib/python3.6/multiprocessing/util.py", line 186,
in __call__
res = self._callback(*self._args, **self._kwargs)
File "/home/tim/.virtualenvs/django36/lib/python3.6/shutil.py", line
465, in rmtree
onerror(os.lstat, path, sys.exc_info())
File "/home/tim/.virtualenvs/django36/lib/python3.6/shutil.py", line
463, in rmtree
orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory:
'/tmp/django_k0xziymh/pymp-i4s112bj'
}}}
What I found out:
- `multiprocessing` is involved. Indeed, running with `--parallel=1` does
not trigger this error.
- Jinja2 is involved. Running tests without Jinja2 installed does not
trigger this error.
What happens:
1. `multiprocessing` registers `atexit` handler.
1. `runtests.py` creates a temp dir (e.g. `/tmp/django_k0xziymh`) and
registers `atexit` handler to remove it.
1. `multiprocessing` creates a temp dir (e.g. `/tmp/django_k0xziymh/pymp-
i4s112bj`) which will be deleted in the handler it registered earlier.
1. Tests happen.
1. `runtests.py` exit handler deletes the temp dir.
1. `multiprocessing` exit handler tries to delete the inner temp dir but
it is already gone.
On earlier Python versions 1 and 2 are swapped and 5 and 6 are swapped, so
the error does not happen.
Jinja2 is imported by a chain of imports starting with `from django.test
import TestCase, TransactionTestCase` (full chain below).
On Python 3.6 Jinja2
[https://github.com/pallets/jinja/blob/master/jinja2/__init__.py#L74-L78
patches async support] which eventually imports `multiprocessing.util`
which registers the exit handler.
Possible solutions:
- Create the temp dir before importing `django` modules.
- Move some imports in `runtests.py` into functions, thereby delaying the
indirect import of Jinja2.
- Move some imports somewhere else into functions to break the import
chain.
- ...
The import chain leading up to the import of `multiprocessing.util` (with
uninteresting import machinery stack frames removed):
{{{
File "./runtests.py", line 18, in <module>
from django.test import TestCase, TransactionTestCase
File "/home/vytis/src/django/django/test/__init__.py", line 5, in
<module>
from django.test.client import Client, RequestFactory
File "/home/vytis/src/django/django/test/client.py", line 12, in
<module>
from django.core.handlers.base import BaseHandler
File "/home/vytis/src/django/django/core/handlers/base.py", line 7, in
<module>
from django.urls import get_resolver, set_urlconf
File "/home/vytis/src/django/django/urls/__init__.py", line 1, in
<module>
from .base import (
File "/home/vytis/src/django/django/urls/base.py", line 8, in <module>
from .exceptions import NoReverseMatch, Resolver404
File "/home/vytis/src/django/django/urls/exceptions.py", line 1, in
<module>
from django.http import Http404
File "/home/vytis/src/django/django/http/__init__.py", line 5, in
<module>
from django.http.response import (
File "/home/vytis/src/django/django/http/response.py", line 13, in
<module>
from django.core.serializers.json import DjangoJSONEncoder
File "/home/vytis/src/django/django/core/serializers/__init__.py", line
23, in <module>
from django.core.serializers.base import SerializerDoesNotExist
File "/home/vytis/src/django/django/core/serializers/base.py", line 6,
in <module>
from django.db import models
File "/home/vytis/src/django/django/db/models/__init__.py", line 3, in
<module>
from django.db.models.aggregates import * # NOQA
File "/home/vytis/src/django/django/db/models/aggregates.py", line 5, in
<module>
from django.db.models.expressions import Func, Star
File "/home/vytis/src/django/django/db/models/expressions.py", line 6,
in <module>
from django.db.models import fields
File "/home/vytis/src/django/django/db/models/fields/__init__.py", line
11, in <module>
from django import forms
File "/home/vytis/src/django/django/forms/__init__.py", line 6, in
<module>
from django.forms.boundfield import * # NOQA
File "/home/vytis/src/django/django/forms/boundfield.py", line 5, in
<module>
from django.forms.widgets import Textarea, TextInput
File "/home/vytis/src/django/django/forms/widgets.py", line 21, in
<module>
from .renderers import get_default_renderer
File "/home/vytis/src/django/django/forms/renderers.py", line 11, in
<module>
from django.template.backends.jinja2 import Jinja2
File "/home/vytis/src/django/django/template/backends/jinja2.py", line
1, in <module>
import jinja2
File "/home/vytis/src/env/django-py36/lib/python3.6/site-
packages/jinja2/__init__.py", line 81, in <module>
_patch_async()
File "/home/vytis/src/env/django-py36/lib/python3.6/site-
packages/jinja2/__init__.py", line 77, in _patch_async
from jinja2.asyncsupport import patch_all
File "/home/vytis/src/env/django-py36/lib/python3.6/site-
packages/jinja2/asyncsupport.py", line 13, in <module>
import asyncio
File "/opt/python/lib/python3.6/asyncio/__init__.py", line 21, in
<module>
from .base_events import *
File "/opt/python/lib/python3.6/asyncio/base_events.py", line 17, in
<module>
import concurrent.futures
File "/opt/python/lib/python3.6/concurrent/futures/__init__.py", line
17, in <module>
from concurrent.futures.process import ProcessPoolExecutor
File "/opt/python/lib/python3.6/concurrent/futures/process.py", line 55,
in <module>
from multiprocessing.connection import wait
File "/opt/python/lib/python3.6/multiprocessing/connection.py", line 23,
in <module>
from . import util
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27890>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* component: Testing framework => Core (Other)
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:1>
Django
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Claude Paroz):
Another option might be to register a custom function that wraps
`shutil.rmtree` and catches the exception.
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:2>
Django
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Vytis Banaitis):
Replying to [comment:2 Claude Paroz]:
> Another option might be to register a custom function that wraps
`shutil.rmtree` and catches the exception.
It's possible, but would require monkey-patching `multiprocessing`.
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:3>
Django
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Claude Paroz):
Replying to [comment:3 Vytis Banaitis]:
> Replying to [comment:2 Claude Paroz]:
> > Another option might be to register a custom function that wraps
`shutil.rmtree` and catches the exception.
> It's possible, but would require monkey-patching `multiprocessing`.
Really? My idea was something like:
{{{
def custom_delete(tmpdir):
try:
shutil.rmtree(tmpdir)
except FileNotFoundError:
pass
atexit.register(custom_delete, TMPDIR)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:4>
Django
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Vytis Banaitis):
Replying to [comment:4 Claude Paroz]:
> Really? My idea was something like:
>
> {{{
> def custom_delete(tmpdir):
> try:
> shutil.rmtree(tmpdir)
> except FileNotFoundError:
> pass
>
> atexit.register(custom_delete, TMPDIR)
> }}}
Removal of TMPDIR happens without error.
The error is raised by `multiprocessing` exit handler which tries to
delete a directory inside of TMPDIR.
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:5>
Django
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Claude Paroz):
Oh, now I see, sorry for the misunderstanding.
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:6>
Django
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* version: master => 1.11
* severity: Normal => Release blocker
Comment:
I looked a little and didn't see an obvious solution besides some import
rearranging as suggested in the ticket description.
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:7>
Django
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/8198 PR]. Not sure if it's ideal,
but good enough for now?
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:8>
Django
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham):
Rather than the rearranging imports approach of the first PR, an
alternative [https://github.com/django/django/pull/8210 PR] deletes
multiprocessing's temporary directory removal handler to avoid the error.
Going forward, patching cpython to ignore the error might be an option:
{{{ #!diff
diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py
index 1a2c0db..6843d09 100644
--- a/Lib/multiprocessing/util.py
+++ b/Lib/multiprocessing/util.py
@@ -113,7 +113,7 @@ def get_temp_dir():
import shutil, tempfile
tempdir = tempfile.mkdtemp(prefix='pymp-')
info('created temp directory %s', tempdir)
- Finalize(None, shutil.rmtree, args=[tempdir], exitpriority=-100)
+ Finalize(None, shutil.rmtree, args=[tempdir],
kwargs={'ignore_errors': True}, exitpriority=-100)
process.current_process()._config['tempdir'] = tempdir
return tempdir
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:9>
Django
Reporter: Vytis Banaitis | Owner: nobody
Component: Core (Other) | Version: 1.11
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham <timograham@…>):
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"0c6c859d4edf6e462d000da21b4fa5009cb2696f" 0c6c859]:
{{{
#!CommitTicketReference repository=""
revision="0c6c859d4edf6e462d000da21b4fa5009cb2696f"
Fixed #27890 -- Fixed FileNotFoundError cleanup exception in runtests.py
on Python 3.6+.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:10>
Django
Reporter: Vytis Banaitis | Owner: nobody
Component: Core (Other) | Version: 1.11
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"e0ddfa3af30e8e0328cef17fd424c4337efec58e" e0ddfa3a]:
{{{
#!CommitTicketReference repository=""
revision="e0ddfa3af30e8e0328cef17fd424c4337efec58e"
[1.11.x] Fixed #27890 -- Fixed FileNotFoundError cleanup exception in
runtests.py on Python 3.6+.
Backport of 0c6c859d4edf6e462d000da21b4fa5009cb2696f from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:11>
Django
Component: Core (Other) | Version: 1.11
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Easy pickings: 0 | UI/UX: 0
Comment (by Jacob Walls <jacobtylerwalls@…>):
In [changeset:"cd6278c4c09e4af9b2988d585b372d9abeeb63ee" cd6278c]:
{{{#!CommitTicketReference repository=""
revision="cd6278c4c09e4af9b2988d585b372d9abeeb63ee"
Refs #27890 -- Avoided overwriting TMPDIR in runtests.py under forkserver
mode.
This variable should only be set once. Under forkserver, this module
was getting executed multiple times, causing nested temporary dirs
that didn't clean up properly, raising FileNotFoundError.
This similar to #27890 although a slightly different cause.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:12>
Django
Component: Core (Other) | Version: 1.11
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Easy pickings: 0 | UI/UX: 0
Comment (by Jacob Walls <jacobtylerwalls@…>):
{{{#!CommitTicketReference repository=""
revision="2b808d567de43d8ec591d9eb688ae66055e3aefc"
[6.0.x] Refs #27890 -- Avoided overwriting TMPDIR in runtests.py under
This variable should only be set once. Under forkserver, this module
was getting executed multiple times, causing nested temporary dirs
that didn't clean up properly, raising FileNotFoundError.
This similar to #27890 although a slightly different cause.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27890#comment:13>
Django
Component: Core (Other) | Version: 1.11
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Easy pickings: 0 | UI/UX: 0
Comment (by Jacob Walls <jacobtylerwalls@…>):
{{{#!CommitTicketReference repository=""
revision="9cc231e8243091519f5d627cd02ee40bbb853ced"
[5.2.x] Refs #27890 -- Avoided overwriting TMPDIR in runtests.py under
This variable should only be set once. Under forkserver, this module
was getting executed multiple times, causing nested temporary dirs
that didn't clean up properly, raising FileNotFoundError.
This similar to #27890 although a slightly different cause.
Backport of cd6278c4c09e4af9b2988d585b372d9abeeb63ee from main.
}}}
--
Django
Component: Core (Other) | Version: 1.11
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Easy pickings: 0 | UI/UX: 0
Comment (by Jacob Walls <jacobtylerwalls@…>):
{{{#!CommitTicketReference repository=""
revision="9b6751dba0c84073a2ee463293a2d7cfb3aa7184"
[4.2.x] Refs #27890 -- Avoided overwriting TMPDIR in runtests.py under
This variable should only be set once. Under forkserver, this module
was getting executed multiple times, causing nested temporary dirs
that didn't clean up properly, raising FileNotFoundError.
This similar to #27890 although a slightly different cause.
Backport of cd6278c4c09e4af9b2988d585b372d9abeeb63ee from main.
}}}
--