Multi-Threaded Dev Server

[Chris]

I think http://code.djangoproject.com/ticket/3357 should be given
another look at enabling optional multi-threading on the dev server.

Jacob previously closed this ticket, noting the patch could introduce
threading bugs, and would provide functionality too similar to that of
a production environment.

This is my rationale to accept this ticket:
1. Ajax is very common place, and often requires a multi-threaded
server. The dev server is very convenient, but by not providing multi-
threading support, we're preventing it from being even more useful.
2. Since multi-threading would be an optional setting, the default
scenario would still be single-threaded and would not break anything.
Only those specifically desiring multi-threading would subject
themselves to any unforeseen bugs.
3. Fear of multi-threading bugs shouldn't be a reason to avoid multi-
threading, especially when it could be very useful. We don't even know
if there are multi-threading bugs. And even if there are, they can be
fixed.

Regards,
Chris

[Calvin Spealman]

Does it need to be multi-threaded? Can you get a multi-process dev
server instead/also?

--
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

[alex....@gmail.com]

If the patch that's currently on there works with no changes at all,
I'd be hesitantly in favor of it, however if there any bugs, or other
complications associated with it, my answer is to use a real server,
setting up something like CherryPy locally is almost no effort, as
seen here: http://www.oebfare.com/blog/2008/nov/03/writing-custom-management-command/

On Nov 16, 1:39 am, "Calvin Spealman" <ironfro...@gmail.com> wrote:
> Does it need to be multi-threaded? Can you get a multi-process dev
> server instead/also?
>
>
>

> On Sun, Nov 16, 2008 at 1:26 AM, Chris <chriss...@gmail.com> wrote:
>
> > I thinkhttp://code.djangoproject.com/ticket/3357should be given

> > another look at enabling optional multi-threading on the dev server.
>
> > Jacob previously closed this ticket, noting the patch could introduce
> > threading bugs, and would provide functionality too similar to that of
> > a production environment.
>
> > This is my rationale to accept this ticket:
> > 1. Ajax is very common place, and often requires a multi-threaded
> > server. The dev server is very convenient, but by not providing multi-
> > threading support, we're preventing it from being even more useful.
> > 2. Since multi-threading would be an optional setting, the default
> > scenario would still be single-threaded and would not break anything.
> > Only those specifically desiring multi-threading would subject
> > themselves to any unforeseen bugs.
> > 3. Fear of multi-threading bugs shouldn't be a reason to avoid multi-
> > threading, especially when it could be very useful. We don't even know
> > if there are multi-threading bugs. And even if there are, they can be
> > fixed.
>
> > Regards,
> > Chris
>
> --

> Read my blog! I depend on your acceptance of my opinion! I am interesting!http://techblog.ironfroggy.com/

[David Cramer]

I'm a bit confused how multi-threading has anything to do with AJAX?
Your requests will be slower, but they will still get processed (at
least I've never had any issues).

On Nov 16, 1:03 am, "alex.gay...@gmail.com" <alex.gay...@gmail.com>
wrote:

> If the patch that's currently on there works with no changes at all,
> I'd be hesitantly in favor of it, however if there any bugs, or other
> complications associated with it, my answer is to use a real server,
> setting up something like CherryPy locally is almost no effort, as

> seen here:http://www.oebfare.com/blog/2008/nov/03/writing-custom-management-com...

>
> On Nov 16, 1:39 am, "Calvin Spealman" <ironfro...@gmail.com> wrote:
>
> > Does it need to be multi-threaded? Can you get a multi-process dev
> > server instead/also?
>
> > On Sun, Nov 16, 2008 at 1:26 AM, Chris <chriss...@gmail.com> wrote:
>

> > > I thinkhttp://code.djangoproject.com/ticket/3357shouldbe given

[Vinay Sajip]

On Nov 16, 6:26 am, Chris <chriss...@gmail.com> wrote:
> I thinkhttp://code.djangoproject.com/ticket/3357should be given

I would second what David Cramer said. You don't need, in the general
case, need multi-threading on the server side for Ajax - it's really
about browser-side multi-threading. I'm not sure where your point 1
comes from: I use AJAX with the standard, single-threading development
server all the time, and I have not encountered any issues at all. I'd
say leave it as it is, unless of course you have encountered some
specific problem relating to the single-threaded nature of the server.
In that case, please post some details.

It's theoretically possible that with the server tied up with some
long-running request, the browser might time out before the server got
around to servicing an AJAX request which was next in line. However,
you should be able to configure client-side time-outs appropriately to
mitigate this.

Regards,

Vinay Sajip

[Julian]

Some months ago I tried a snippet from djangosnippets.org which shows a upload progress bar. I think some people want to use snippets like that and a multi-threaded server would help you developing such applications.

2008/11/16 Vinay Sajip <vinay...@yahoo.co.uk>

[Steve Holden]

Julian wrote:
> [...] I think some people want to use snippets like that [...]

Wouldn't you agree that's a pretty feeble use case for something as
potentially disruptive as multi-threaded serving? Particularly when the
CherryPy alternative is so readily available.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/

[Waylan Limberg]

On Sun, Nov 16, 2008 at 1:26 AM, Chris <chri...@gmail.com> wrote:
>
> I think http://code.djangoproject.com/ticket/3357 should be given
> another look at enabling optional multi-threading on the dev server.

> 3. Fear of multi-threading bugs shouldn't be a reason to avoid multi-

> threading, especially when it could be very useful. We don't even know
> if there are multi-threading bugs. And even if there are, they can be
> fixed.
>

Yet, the time spent identifying and fixing those buds means less time
developing real features that I need and can use on my production
sites. I'd say the developers time is better spent elsewhere.
Especially considering there are already working solutions out there.
I seem to recall at least one management command someone put together
that runs a multithreaded cherrypy server. Why reinvent the wheel?
Lets focus on real, useful features.

--
----
Waylan Limberg
way...@gmail.com

[Vinay Sajip]

On Nov 16, 5:21 pm, "Waylan Limberg" <way...@gmail.com> wrote:

> On Sun, Nov 16, 2008 at 1:26 AM, Chris <chriss...@gmail.com> wrote:
> Especially considering there are already working solutions out there.
> I seem to recall at least one management command someone put together
> that runs a multithreaded cherrypy server. Why reinvent the wheel?
> Lets focus on real, useful features.
>

Especially since Alex Gaynor helpfully posted a link to that specific
solution earlier in this thread. Here it is again:

http://www.oebfare.com/blog/2008/nov/03/writing-custom-management-command/

Regards,

Vinay Sajip

[David Cramer]

I think I may have to actually agree that multi-threading is somewhat
needed, since it does limit these kind of features from working. I,
however, would deem it less a priority than most thing. If you can
write a patch for it (or if one exists, I didn't look) though, I don't
see any reason to not extend runserver to use it.

[Calvin Spealman]

The last thing new people need when they come in and start playing
around with the dev server is to worry about all the issues that can
come up with threading. I would never recommend to anyone who doesn't
really know what they're doing and anyone who does can easily use any
one of the number of solutions other than the dev server to test those
setups. Keep it simple. Keep everyone sane. Even if it was added as
some optional flag, I can guarantee that a number of people without
enough of an understanding of threads are just going to enable it when
they read about it thinking "threads make it faster" and thinking no
more about it. I strongly urge this to not be added to what should be
a simple dev server.

[Ludvig Ericson]

On Nov 16, 2008, at 07:26, Chris wrote:

> 3. Fear of multi-threading bugs shouldn't be a reason to avoid multi-
> threading, especially when it could be very useful. We don't even know
> if there are multi-threading bugs. And even if there are, they can be
> fixed.

There are bugs. Django isn't thread-safe, and we know that.

The best solution I can come up with would be to use the type of
multitasking given by the processing library in py2.6.

http://pypi.python.org/pypi/processing

Ludvig Ericson

[matt westerburg]

Is there any documentation as to why Django is not threadsafe?  It just seems to me that with the Global Interpreter Lock and all, it would be threadsafe.  I don't intend to start a flame, I am just curious about learning why this might be.  I am favor of the mutlitprocess approach layed out by Ludvig. 

[Jacob Kaplan-Moss]

On Mon, Nov 17, 2008 at 10:54 AM, Ludvig Ericson
<ludvig....@gmail.com> wrote:
> There are bugs. Django isn't thread-safe, and we know that.

Um...

That's just not true. At one point (two years ago?) it wasn't, but
these days Django's deployed all over the place in mutli-threaded
situations. If it wasn't threadsafe we'd hear about it.

Jacob

[Leo Soto M.]

On Mon, Nov 17, 2008 at 4:14 PM, matt westerburg <west...@gmail.com> wrote:
> Is there any documentation as to why Django is not threadsafe? It just
> seems to me that with the Global Interpreter Lock and all, it would be
> threadsafe.

That wouldn't as nice as "really-really thread safe". You know, there
are GIL-less Python implementations out there ;-)

--
Leo Soto M.
http://blog.leosoto.com

[mrts]

On Nov 17, 6:54 pm, Ludvig Ericson <ludvig.eric...@gmail.com> wrote:
> On Nov 16, 2008, at 07:26, Chris wrote:
>
> > 3. Fear of multi-threading bugs shouldn't be a reason to avoid multi-
> > threading, especially when it could be very useful. We don't even know
> > if there are multi-threading bugs. And even if there are, they can be
> > fixed.
>
> There are bugs. Django isn't thread-safe, and we know that.

Which bugs in particular? By all means, if you find one, report it. I
have run Django with mod_wsgi threaded daemon mode with no ill effects
for a long time. See http://code.djangoproject.com/wiki/DjangoSpecifications/Core/Threading
, all bugs found during the review were fixed a long time ago (and
only one of them, #6950, was serious-ish).

[Vinay Sajip]

On Nov 17, 7:25 pm, "Jacob Kaplan-Moss" <jacob.kaplanm...@gmail.com>
wrote:

>
> That's just not true. At one point (two years ago?) it wasn't, but
> these days Django's deployed all over the place in mutli-threaded
> situations. If it wasn't threadsafe we'd hear about it.
>

As I understand it, there are some areas where you have to be careful.
For example, you can't be sure that caching templates won't lead to
problems, because there are some template tags where rendering state
is held in the nodes rather than the context. This post may give more
information:

http://lucumr.pocoo.org/cogitations/2008/09/16/why-jinja-is-not-django-and-why-django-should-have-a-look-at-it/#comment-12073

If you ensure that templates are always compiled from source for each
request, then this shouldn't bite you.

Best regards,

Vinay Sajip

[Matthew Russell]

Hi,

I'd like to point out that as you've stated, django is deployed in mulit-threaded environments successfully,

employing an app that makes use of geodjango in a multi-threaded environment is not.

Justin (Bronn) will testify to this I'm sure (as I and others have run into this issue whilst attempting to deploy mod_wsgi/mod_python/others in a mutli-threaded environment.  (This is due to the underlying libgeos c library not being thread safe itself)

I should (and will) create a ticket to make this clear in the django docs for the geodjango parts, so that others may avoid the pain we went throught a few months back.

Cheers,

Matt

2008/11/17 Jacob Kaplan-Moss <jacob.ka...@gmail.com>

--
Cheers,
Matt