[Django] #27086: running servers.tests hangs in parallel mode
Django
-------------------------------+--------------------
Reporter: cjerdonek | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: master
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------
From master (97513269d73520d10722bbd10404be6ac4d48d07), running the
following command from the tests directory hangs for me after a couple
tests:
{{{
$ ./runtests.py servers.tests
}}}
This is with Python 3.5.2 on Mac OSX.
There is no error message. It just hangs. It works okay when passing
`--parallel 1`.
--
Ticket URL: <https://code.djangoproject.com/ticket/27086>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: cjerdonek | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: master
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
I can't reproduce on Ubuntu 14.04, Python 3.5.2. Just in case, how many
parallel process do you have? I have 4 here.
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:1>
Django
Reporter: cjerdonek | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: master
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by cjerdonek):
8, but it also happens when I pass `--parallel 2`. To narrow it down, it
happens when I run the following (I believe the test setup executes, but
it never gets to running the tests):
{{{
$ ./runtests.py servers.tests.LiveServerDatabase
servers.tests.LiveServerViews
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:2>
Django
Reporter: cjerdonek | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: master
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
You could check if the issue is present at the time that parallel test
running was added in Django and if not, bisect to find the commit that
introduced the problem.
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:3>
Django
Reporter: cjerdonek | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: master
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by bpeschier):
Just ran the tests at the mentioned commit on my macOS Sierra public beta
with a fresh 3.5.2 python environment. No problems there.
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:4>
Django
Reporter: cjerdonek | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: master
Severity: Normal | Resolution: worksforme
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: new => closed
* resolution: => worksforme
Comment:
Feel free to reopen if you find that Django is at fault.
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:5>
Django
Reporter: cjerdonek | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: master
Severity: Normal | Resolution: worksforme
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by cjerdonek):
I narrowed this down further to the following:
The hang happens when calling `urllib.request.urlopen()` inside
`LiveServerBase`. And digging further, inside CPython's request.py, the
hang happens when calling `_scproxy._get_proxies()`.
There seem to be known issues around `_scproxy` in CPython for Mac OS X
users. For example, see the following two issues in CPython's bug
tracker:
(1) Apple-supplied libsqlite3 on OS X is not fork safe; can cause
crashes:
http://bugs.python.org/issue27126
(See this comment, in particular:
http://bugs.python.org/issue27126#msg266405 )
(2) exception error in _scproxy.so when called after fork
http://bugs.python.org/issue13829
There is a chance that there are easy workarounds on the Django side that
can be made to accommodate Mac OS X users.
Also, for the record I'm using 10.11.6.
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:6>
Django
Reporter: cjerdonek | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: master
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by cjerdonek):
* status: closed => new
* resolution: worksforme =>
Comment:
I was able to get things working by causing Mac OS X's auto-detection of
proxy information to be skipped. This possibility is mentioned
[https://docs.python.org/3/library/urllib.request.html#urllib.request.ProxyHandler
here] in CPython's `urllib.request.ProxyHandler` documentation.
More detailed instructions appear in `urllib.request`'s module docstring.
To disable the auto-detection, I followed the instructions in the above-
mentioned module docstring and modified `LiveServerBase.urlopen()` to look
like the following (this is just a proof of concept):
{{{#!python
def urlopen(self, url):
import urllib.request
# Pass an empty dictionary to disable auto-detecting the proxy.
proxy_support = urllib.request.ProxyHandler({})
opener = urllib.request.build_opener(proxy_support)
urllib.request.install_opener(opener)
return urlopen(self.live_server_url + url)
}}}
After doing this, the following command worked fine:
{{{
$ ./runtests.py servers.tests
}}}
Since this seems like a fundamental bug / limitation on Mac OS X (in at
least some versions), it seems worth considering including Mac-specific
code to enable the parallel running of tests on this platform.
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:7>
Django
Reporter: cjerdonek | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Other) | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* component: Uncategorized => Core (Other)
* type: Uncategorized => Cleanup/optimization
* stage: Unreviewed => Accepted
Comment:
I wonder what's special about your system compared to other Mac OS
contributors who didn't encounter the problem.
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:8>
Django
--------------------------------------+------------------------------------
Reporter: cjerdonek | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Other) | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:9>
Django
--------------------------------------+------------------------------------
Type: Cleanup/optimization | Status: new
Component: Core (Other) | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by cjerdonek):
I've been unable to post my full comment because of the spam blocker, so
I'll try including only the link I wanted to mention (but broken apart to
evade the blocker).
The first part of the link is here: http://www.evanjones.ca/
The second part is here: fork-is-dangerous.html
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:10>
Django
--------------------------------------+------------------------------------
Type: Cleanup/optimization | Status: new
Component: Core (Other) | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by cjerdonek):
Also (to add more of what I wanted to say but couldn't above), I don't
know yet why I'm the first person to notice / report this. In case it
would help, I started working on a reduced way to reproduce this
(independent of the Django test suite).
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:11>
Django
--------------------------------------+------------------------------------
Type: Cleanup/optimization | Status: new
Component: Core (Other) | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by cjerdonek):
Oh, the person that wrote the blog post already created a "reduced test
case" that I started to look for: https://github.com/evanj/fork-thread-
bugs/blob/master/osx_python_crash_v1.py
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:12>
Django
--------------------------------------+------------------------------------
Type: Cleanup/optimization | Status: new
Component: Core (Other) | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Kees Hink):
* has_patch: 0 => 1
Comment:
Here is a documentation improvement showing a workaround:
https://github.com/django/django/pull/11203
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:13>
Django
--------------------------------------+------------------------------------
Reporter: Chris Jerdonek | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Other) | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Bell):
I have encountered the same problem (on MacOS 10.14.4 Mojave) and confirm
that the solution works for me. This would be a useful addition to the
documentation.
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:14>
Django
--------------------------------------+------------------------------------
Reporter: Chris Jerdonek | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Other) | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Bell):
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:15>
Django
--------------------------------------+------------------------------------
Reporter: Chris Jerdonek | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Other) | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Bell):
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:16>
Django
Reporter: Chris Jerdonek | Owner: nobody
Cleanup/optimization |
Component: Core (Other) | Version: master
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Bell):
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:17>
Django
-------------------------------------+-------------------------------------
Reporter: Chris Jerdonek | Owner: nobody
Cleanup/optimization |
Component: Core (Other) | Version: master
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* status: new => closed
* resolution: => wontfix
Comment:
Closing as per discussion on PR: lack of a decent reproduce, plenty of
Google results for the error, not our responsibility, and likely a problem
that's been resolved in recent releases (but c.f. lack of reproduce...)
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:18>
Django
-------------------------------------+-------------------------------------
Reporter: Chris Jerdonek | Owner: nobody
Cleanup/optimization |
Component: Core (Other) | Version: master
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* status: closed => new
* resolution: wontfix =>
Comment:
I'm at the sprints at DjangoCon. ''Many'' new contributors have hit this,
so I think the ''doesn't reproduce'' worry is passed. I'm going to re-open
this accordingly. We should doc so new contributors are able to get going.
Related is #30806 (which is targeting running `tox` specifically.)
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:19>
Django
-------------------------------------+-------------------------------------
Type: | Gibson
Cleanup/optimization | Status: assigned
Component: Core (Other) | Version: master
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* owner: nobody => Carlton Gibson
* status: new => assigned
* stage: Ready for checkin => Accepted
Comment:
I just want to adjust this, so moving to Accepted and assigning to myself.
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:20>
Django
-------------------------------------+-------------------------------------
Reporter: Chris Jerdonek | Owner: Carlton
Type: | Gibson
Component: Core (Other) | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson <carlton@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"5d9cf79baf07fc4aed7ad1b06990532a65378155" 5d9cf79b]:
{{{
#!CommitTicketReference repository=""
revision="5d9cf79baf07fc4aed7ad1b06990532a65378155"
Fixed #27086 -- Doc'd fix for test suite hang on macOS.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:21>
Django
-------------------------------------+-------------------------------------
Reporter: Chris Jerdonek | Owner: Carlton
Type: | Gibson
Cleanup/optimization | Status: closed
Component: Core (Other) | Version: master
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Carlton Gibson <carlton.gibson@…>):
In [changeset:"fffce1ac4c412ec49bde659b79dd943266c9434f" fffce1a]:
{{{
#!CommitTicketReference repository=""
revision="fffce1ac4c412ec49bde659b79dd943266c9434f"
[3.0.x] Fixed #27086 -- Doc'd fix for test suite hang on macOS.
Backport of 5d9cf79baf07fc4aed7ad1b06990532a65378155 from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:22>
Django
-------------------------------------+-------------------------------------
Reporter: Chris Jerdonek | Owner: Carlton
Type: | Gibson
Cleanup/optimization | Status: closed
Component: Core (Other) | Version: master
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Carlton Gibson <carlton.gibson@…>):
In [changeset:"4f7ba25e67681efbb67cc4810e919aa7369cbe34" 4f7ba25e]:
{{{
#!CommitTicketReference repository=""
revision="4f7ba25e67681efbb67cc4810e919aa7369cbe34"
[2.2.x] Fixed #27086 -- Doc'd fix for test suite hang on macOS.
Backport of 5d9cf79baf07fc4aed7ad1b06990532a65378155 from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27086#comment:23>