[Django] #20238: StoppableWSGIServer is not multithread support
Django
-----------------------------------+--------------------
Reporter: jwxiao@… | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 1.4
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------
Run a selenium test with LiveServerTestCase. Django starts
StoppableWSGIServer. The StoppableWSGIServer is not multithread support.
When the codes to be tested call urllib.urlopen to the same server (ie:
StoppableWSGIServer) the selenium test will hang and fail because of time-
out.
--
Ticket URL: <https://code.djangoproject.com/ticket/20238>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: jwxiao@… | Owner: nobody
Type: Bug | Status: closed
Component: Testing framework | Version: 1.4
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by julien):
* status: new => closed
* needs_docs: => 0
* resolution: => needsinfo
* needs_tests: => 0
* needs_better_patch: => 0
Comment:
Thank you for the report, however there is too little information here to
identify if or where there is an issue. Could you please provide a
detailed explanation of how to reproduce the issue you're facing, or even
better, could you provide some code or a test case? Thanks!
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:1>
Django
Reporter: jwxiao@… | Owner: nobody
Type: Bug | Status: closed
Component: Testing framework | Version: 1.4
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by julien):
Just a clarification: please feel free to reopen this ticket if you can
provide more detailed information.
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:2>
Django
Reporter: jwxiao@… | Owner: nobody
Type: Bug | Status: closed
Component: Testing framework | Version: 1.4
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by joeri):
Although this is not my ticket, I just ran into the same issue.
The problem occurs typically when, within a request/response cycle,
another request is done to the same `StoppableWSGIServer`. Now, the first
request can never finish because the second request is waiting for the
`StoppableWSGIServer` to respond (but it never will because it's waiting
for the first request...)
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:3>
Django
Reporter: jwxiao@… | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 1.4
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by bdrennan@…):
* status: closed => new
* resolution: needsinfo =>
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:4>
Django
Reporter: jwxiao@… | Owner: nobody
Type: Bug | Status: closed
Component: Testing framework | Version: 1.4
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by aaugustin):
* status: new => closed
* resolution: => wontfix
Comment:
Considering that:
- "the use case is somewhat fictional",
- the ticket was reopened without any additional information,
- more threading will certainly create more race conditions on shutdown,
especially when it comes to the database connections — it's taken months
to eliminate most from LiveServerTestCase, and I'm sure there are still
some left,
I don't think we should add threading without more compelling arguments.
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:5>
Django
Reporter: jwxiao@… | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 1.4
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Aymeric Augustin):
* status: closed => new
* resolution: wontfix =>
Comment:
I'm hitting this problem, which means I have a concrete use case to offer.
I have a view that generates a PDF document by converting a HTML document
served by the same server (with phantomjs). It works as follows:
- user makes a HTTP request for a PDF document
- pdf_view() starts a phantomjs subprocess
- phantomjs makes a HTTP request for the equivalent HTML document
- html_view() renders a HTML template
- phantomjs gets the HTML, converts it to PDF, and returns that
- pdf_view() returns the PDF
I cannot use that view with the LiveServerTestCase because it's single
threaded. The thread runs pdf_view(), waits for phantomjs, but there's no
thread available to run html_view(), so the test blocks there.
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:6>
Django
Reporter: jwxiao@… | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 1.4
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Aymeric Augustin):
The following monkey-patch does the job for me.
{{{
import socketserver
from django.test import testcases
class ThreadedWSGIServer(socketserver.ThreadingMixIn,
testcases.WSGIServer):
"""
Make Django's live server test case multi-threaded.
See https://code.djangoproject.com/ticket/20238#comment:6
"""
testcases.WSGIServer = ThreadedWSGIServer
}}}
I'm using the PostgreSQL database backend.
I don't think this proposal is sufficient for in-memory SQLite. A
WSGIHandler subclass that replaces the database connection for each new
thread by the database connection from the main thread is likely needed.
----
By the way, this piece of code in `django.core.servers.basehttp`:
{{{
if threading:
httpd_cls = type(str('WSGIServer'), (socketserver.ThreadingMixIn,
WSGIServer), {})
else:
httpd_cls = WSGIServer
}}}
could be replaced with:
{{{
httpd_cls = ThreadedWSGIServer if threading else WSGIServer
}}}
assuming the definition of `ThreadedWSGIServer` above.
There's no reason (that I can think of) to define a class dynamically
like that here. Saving a couple lines isn't worth the obfuscation.
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:7>
Django
Reporter: jwxiao@… | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 1.4
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Aymeric Augustin):
jwxiao, I'm sorry for not understanding your report originally.
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:8>
Django
-----------------------------------+------------------------------------
Reporter: jwxiao@… | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: 1.4
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* type: Bug => New feature
* stage: Unreviewed => Accepted
Comment:
This was also requested in #25970.
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:9>
Django
-----------------------------------+------------------------------------
Type: New feature | Status: assigned
Component: Testing framework | Version: 1.4
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Nadege):
* owner: nobody => Nadege
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:10>
Django
-----------------------------------+------------------------------------
Reporter: jwxiao@… | Owner: Nadege
Type: New feature | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham):
Support for multiple open connections is also needed if you want to write
a test that uses multiple browser instances as demonstrated in #27665
(closed as duplicate).
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:11>
Django
-----------------------------------+------------------------------------
Reporter: jwxiao@… | Owner: Nadege
Type: New feature | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Claude Paroz):
* has_patch: 0 => 1
* version: 1.4 => master
* needs_tests: 0 => 1
Comment:
I simply put Aymeric suggestion to a patch here:
https://github.com/django/django/pull/7768
If someone has an idea for a test, please help.
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:12>
Django
-----------------------------------+------------------------------------
Reporter: jwxiao@… | Owner: Nadege
Type: New feature | Status: assigned
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Aymeric Augustin):
Nadege also prepared a patch but hasn't submitted a PR yet. I was planning
to take a look at her work after the holidays.
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:13>
Django
-----------------------------------+------------------------------------
Reporter: jwxiao@… | Owner: Nadege
Type: New feature | Status: assigned
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Claude Paroz):
Oh, sorry, I didn't intend to duplicate her work.
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:14>
Django
-----------------------------------+------------------------------------
Reporter: jwxiao@… | Owner: Nadege
Type: New feature | Status: assigned
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Nadege):
* needs_tests: 1 => 0
Comment:
We actually made the sames changes, I added some tests as well.
They don't fail but hang whithout the patch.
I tried to do a test that could show potential database issue because
aymeric sugested that might be a problem
but things seems good.
Patch is here https://github.com/django/django/pull/7832
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:15>
Django
-----------------------------------+------------------------------------
Reporter: jwxiao@… | Owner: Nadege
Type: New feature | Status: assigned
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham):
Unless you're motivated to solve the Python 2 failures, we can target this
patch for Django 2.0 which drops support for Python 2. The 1.11 feature
freeze is Friday.
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:16>
Django
-----------------------------------+------------------------------------
Reporter: jwxiao@… | Owner: Nadege
Type: New feature | Status: assigned
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Nadege):
I'm good with targeting Django 2.0
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:17>
Django
-----------------------------------+------------------------------------
Reporter: jwxiao@… | Owner: Nadege
Type: New feature | Status: assigned
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Claude Paroz):
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:18>
Django
-----------------------------------+------------------------------------
Reporter: jwxiao@… | Owner: Nadege
Type: New feature | Status: assigned
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:19>
Django
-----------------------------------+------------------------------------
Reporter: jwxiao@… | Owner: Nadege
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham <timograham@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"bece837829eafbc22f2598dadf82c9a8364b085a" bece8378]:
{{{
#!CommitTicketReference repository=""
revision="bece837829eafbc22f2598dadf82c9a8364b085a"
Fixed #20238 -- Added threading support to LiveServerTestCase.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:20>
Django
-----------------------------------+------------------------------------
Reporter: jwxiao@… | Owner: Nadege
Type: New feature | Status: closed
Component: Testing framework | Version: master
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Chris Jerdonek):
FYI, I just filed #32416, which lists a possible regression caused by this
fix.
--
Ticket URL: <https://code.djangoproject.com/ticket/20238#comment:21>