Multithreaded development server
Almad
is there a way to run development server multithreaded, so it can
handle recurring requests?
Thanks,
Almad
Chris
that will enable multi-threading in the dev server.
Unfortunately, it's likely never to get adopted by Django, since the
lead devs don't want to deal with the maintenance overhead it would
entail.
Chris
John M
full server too. You only need to account for Admin files via some
well know workings.
J
Graham Dumpleton
On Jan 25, 11:08 am, Almad <b...@almad.net> wrote:
> Hello,
>
> is there a way to run development server multithreaded, so it can
> handle recurring requests?
http://blog.dscpl.com.au/2008/12/using-modwsgi-when-developing-django.html
Use daemon mode with that recipe for automatic restarting on changes.
Configure daemon processes to have multiple threads and if necessary
multiple processes. That way you can also test multi process
configuration properly.
Graham
Almad
makes django untestable even under simple conditions (like, using
urllib2 with digest authentication).
I'd love to use CherryPy as I'm using it for some applications anyway,
but I'm now trying to develop library that would allow us to test
Django normally and CP dependency is a bit weird...
Same goes for real server usage, as manage.py test should run without
too much environment configuration.
The patch is cool. I'll try to do some runtime patching...and I'll
probably fork it on github to ease maintainance.
Thanks,
Almad
Almad
from within: http://devel.almad.net/trac/django-sane-testing/changeset/d2c24247d7e4
However, it's still buggy for my case. If You want it too, You can use
CP WSGI as suggested by mikeal some time ago, I'm now using it too:
http://devel.almad.net/trac/django-sane-testing/changeset/00d317877c55
I still feel kinda weird that I must hack around Django so much to
make basic testing things working :-]]]
Almad
Russell Keith-Magee
Let's be clear here - the "basic testing" thing that is failing here
is a very specific, and somewhat esoteric edge case - the case of a
view that invokes, via HTTP, another view on the same server. Client
side AJAX self-calls and using urllib to call a URL on another server
won't lock up the Django development server.
The Django core team has (repeatedly) made a very deliberate decision
to keep the development server single threaded. There are many very
good web servers out there - Django isn't trying to get into the game
of competing with any of them. We view Django's job as providing a
good web stack for a real web server to use; the development server is
provided as a convenience, nothing more.
Having a singlethreaded server makes the implementation much simpler,
so we can concentrate our efforts on making a great web stack, rather
than wasting effort on writing, debugging, and maintaining a web
server that we don't really want people to use for serious work.
A multithreaded server would also act as a tacit encouragement to use
Django's development server for real deployments, which we _really_
don't want to encourage. If people start using Django's development
server in production, we would have to start treating it as a
production-ready server - doing security and performance audits and
the like - and again, this distracts us from what we consider to be
the "main game".
The _only_ type of view that is affected by this decision is the view
that uses urllib (or equivalent) to invoke another view on the same
server. We (the Django core team) have decided that this is a
compromise we can live with. The set of cases where this behavior is
required is very small, doesn't form a part of the requirements for
most (almost all) web development projects. For that small subset of
users that _do_ have a legitimate use for this, there are some
lightweight multithreaded options (like CherryPy) out there, and to
the best of my knowledge, none of these options _require_ the patching
of core in order to use them.
Yours,
Russ Magee %-)
birkin
On Jan 24, 7:08 pm, Almad <b...@almad.net> wrote:
> is there a way to run development server multithreaded, so it can
> handle recurring requests?
hang the single-threaded development server (when I'm also serving out
the webservice via the development server). A very simple solution in
this edge-case of mine is to simply start up an additional terminal
window and execute the runserver command again, specifying a selected
port-number that matches the url for the web-service in my application
settings file.
-Birkin
Almad
> On Jan 24, 7:08 pm,Almad<b...@almad.net> wrote:
>
> A very simple solution in
> this edge-case of mine is to simply start up an additional terminal
> window and execute the runserver command again
But thanks for the tip, would be useful...through it's not exactly
nice for automated suite.
> -Birkin
Almad
Almad
wrote:
> > I still feel kinda weird that I must hack around Django so much to
> > make basic testing things working :-]]]
>
> Let's be clear here - the "basic testing" thing that is failing here
> is a very specific, and somewhat esoteric edge case - the case of a
> view that invokes, via HTTP, another view on the same server.
times in a row - and that is enough to have a problem with dev server.
Which I insist on calling basic testing when we're talking about web
application.
> The Django core team has (repeatedly) made a very deliberate decision
> to keep the development server single threaded. There are many very
> good web servers out there - Django isn't trying to get into the game
> of competing with any of them. We view Django's job as providing a
> good web stack for a real web server to use; the development server is
> provided as a convenience, nothing more.
> Having a singlethreaded server makes the implementation much simpler,
> so we can concentrate our efforts on making a great web stack, rather
> than wasting effort on writing, debugging, and maintaining a web
> server that we don't really want people to use for serious work.
guess it's OK then.
(I'd just say that opitional multithreading contained in 10-line long
patch is not so complicated)
> A multithreaded server would also act as a tacit encouragement to use
> Django's development server for real deployments, which we _really_
> don't want to encourage. If people start using Django's development
> server in production, we would have to start treating it as a
> production-ready server - doing security and performance audits and
> the like - and again, this distracts us from what we consider to be
> the "main game".
production or you'll be banned from all our communication channels on
first complaint', I guess nothing can be done.
i guess PHP users are our target group.
> The _only_ type of view that is affected by this decision is the view
> that uses urllib (or equivalent) to invoke another view on the same
> server.
> We (the Django core team) have decided that this is a
> compromise we can live with. The set of cases where this behavior is
> required is very small, doesn't form a part of the requirements for
> most (almost all) web development projects. For that small subset of
> users that _do_ have a legitimate use for this, there are some
> lightweight multithreaded options (like CherryPy) out there, and to
> the best of my knowledge, none of these options _require_ the patching
> of core in order to use them.
acceptance testing (and thus require, like, Selenium) and discovered
that things looks like working after I lobotomized Django and replaced
it with CherryPy, I'm now satisfied.
> Yours,
> Russ Magee %-)
Almad
P.S.: Sorry for the sarcastic tone of my e-mail. I appreciate Your
work as well as Django framework (or at least some parts of it), but I
just don't get it: for each and every project I've written in Django,
I banged my head against the wall because of some issues that
prevented me from normal testing - and were rejected by devs because
of 'esoteric edge case'...and then reading 'Testing Django apps' on
first presentation slide, 'because it's most important'.
Almad
> On Jan 26, 1:19 am, Russell Keith-Magee <freakboy3...@gmail.com>
> wrote:
> i guess PHP users are our target group.
removed by my self-censoring filter.
Almad
Russell Keith-Magee
>
> P.S.: Sorry for the sarcastic tone of my e-mail. I appreciate Your
> work as well as Django framework (or at least some parts of it), but I
> just don't get it: for each and every project I've written in Django,
> I banged my head against the wall because of some issues that
> prevented me from normal testing - and were rejected by devs because
> of 'esoteric edge case'...and then reading 'Testing Django apps' on
> first presentation slide, 'because it's most important'.
Sorry, but my stress-induced fibritis can only take so much. As a
hint, if you ever find yourself in a position where you need to
apologize at the end of an email for the tone you've taken, it's
probably a good indicator that you need to go back and edit for tone.
As a further hint, using pejoratives in project naming - like, say,
"Sane Testing in Django" - isn't a good way to encourage others to
assist you.
I've explained the reasoning behind Django's decisions. I can only
assure you that I know many people that aren't affected by those
decisions, and are able to use Django's testing framework, as is,
without difficulty. Some of them are even able to use Selenium.
If you sit down and work through the problem, you will establish that
the consequences of those decisions aren't as widespread as you seem
to think. There is _exactly_ one limitation to Django's development
server - _concurrent_ web requests. To be sure, this does impose
limitations, but as long as you don't need concurrent server requests,
you won't have any problems with a single-threaded server. Most web
pages can be adequately served by a number of _sequential_ web
requests.
I will also remind you that just because _you_ hit a given problem
every single day, doesn't mean that _everyone_ hits that problem every
single day. Such is the nature of esoteric problems.
You will note that so far in this thread, you haven't actually
described your problem - just that a multithreaded server is
apparently the only way to solve it. If you ever feel like describing
your problem (sans rhetoric) and working constructively towards a
solution, let me know.
Yours,
Russ Magee %-)
Malcolm Tredinnick
> On Jan 26, 1:19 am, Russell Keith-Magee <freakboy3...@gmail.com>
> wrote:
> > On Sun, Jan 25, 2009 at 10:45 PM,Almad<b...@almad.net> wrote:
> > > I still feel kinda weird that I must hack around Django so much to
> > > make basic testing things working :-]]]
> >
> > Let's be clear here - the "basic testing" thing that is failing here
> > is a very specific, and somewhat esoteric edge case - the case of a
> > view that invokes, via HTTP, another view on the same server.
>
> No, case we're talking about is a test that calls one view multiple
> times in a row - and that is enough to have a problem with dev server.
Perhaps you could show some simplified example code? I have at least a
few dozen tests that do this and they don't have any problem. Serial
requests work fine with the development server. I know of plenty of
other people doing similar things.
[...]
> OK, if it's not meant for serious development and testing work, I
> guess it's OK then.
Depends on your definition of "serious". Given the thousands of Django
applications that have, at least, started out using the development
server, your claim seems a little vacuous.
> (I'd just say that opitional multithreading contained in 10-line long
> patch is not so complicated)
>
> > A multithreaded server would also act as a tacit encouragement to use
> > Django's development server for real deployments, which we _really_
> > don't want to encourage. If people start using Django's development
> > server in production, we would have to start treating it as a
> > production-ready server - doing security and performance audits and
> > the like - and again, this distracts us from what we consider to be
> > the "main game".
>
> OK, if community is expected to ignore 'do not use this option for
> production or you'll be banned from all our communication channels on
> first complaint', I guess nothing can be done.
And if somebody had said that, you *might* even be have a semi-valid
complaint. What colour is the sky in your world? The "community" is not
expected to ignore the 'do not use in production'. They're expected to
read it and take it to heart. Nothing has ever been said about banning.
Russell pointed out something that has been mentioned multiple times in
the past: the development server has a well-defined role to play and the
step-up from there to using other servers when circumstances require is
incredibly minimal. Graham's blog post, given earlier in the thread,
about how to use mod_wsgi is just one example of that. Once we expand
the scope of the development server, it *will* be used in further
unintended ways and we *will* be obliged to maintain it, including
issuing security fixes and the like.
You might well have some reduced outlook on the time commitment involved
there, which is fine, since you aren't one of the people on the hook for
things like that in Django. The group of core maintainers have decided
to devote time elsewhere.
>
> i guess PHP users are our target group.
>
> > The _only_ type of view that is affected by this decision is the view
> > that uses urllib (or equivalent) to invoke another view on the same
> > server.
>
> Yes, that's the only type of view affected. And some tests...
>
> > We (the Django core team) have decided that this is a
> > compromise we can live with. The set of cases where this behavior is
> > required is very small, doesn't form a part of the requirements for
> > most (almost all) web development projects. For that small subset of
> > users that _do_ have a legitimate use for this, there are some
> > lightweight multithreaded options (like CherryPy) out there, and to
> > the best of my knowledge, none of these options _require_ the patching
> > of core in order to use them.
>
> Well, after I fall into small group of users that do some automatized
> acceptance testing (and thus require, like, Selenium) and discovered
> that things looks like working after I lobotomized Django and replaced
> it with CherryPy, I'm now satisfied.
When you could have just used mod_wsgi, or CherryPy's WSGI server or any
number of other things. Nice over-reaction there.
Note, also, that there's a ticket open in Trac to add some extra bits
and pieces for allowing Selenium integration with Django's test
framework. That would also give you some idea of the direction things
could go.
This was a very disappointing mail to read. Whilst I realise you might
well be frustrated, the tone is simply insulting to the amount of work
that has gone into developing and supporting Django and tries to inflate
whatever problem it is that you're having (which hasn't been explained
in detail) way out of proportion.
Regards,
Malcolm
Almad
> Sorry, but my stress-induced fibritis can only take so much. As a
> hint, if you ever find yourself in a position where you need to
> apologize at the end of an email for the tone you've taken, it's
> probably a good indicator that you need to go back and edit for tone.
> As a further hint, using pejoratives in project naming - like, say,
> "Sane Testing in Django" - isn't a good way to encourage others to
> assist you.
that. However, I'll be more glad if I can wipe the whole project by
finding proper Django ways (or perhaps few Django patches) and do some
integration.
> I've explained the reasoning behind Django's decisions. I can only
> assure you that I know many people that aren't affected by those
> decisions, and are able to use Django's testing framework, as is,
> without difficulty. Some of them are even able to use Selenium.
cannot use the xUnit infrastructure and setup database in pre-defined
state? Or is there a simple way to do it, are there any hints around?
> If you sit down and work through the problem, you will establish that
> the consequences of those decisions aren't as widespread as you seem
> to think. There is _exactly_ one limitation to Django'sdevelopment
> server - _concurrent_ web requests. To be sure, this does impose
> limitations, but as long as you don't need concurrent server requests,
> you won't have any problems with a single-threaded server. Most web
> pages can be adequately served by a number of _sequential_ web
> requests.
urllib2 do some concurrent requests without my knowledge.
See #10117 (http://code.djangoproject.com/ticket/10117) for sample
code.
> I will also remind you that just because _you_ hit a given problem
> every single day, doesn't mean that _everyone_ hits that problem every
> single day. Such is the nature of esoteric problems.
> You will note that so far in this thread, you haven't actually
> your problem (sans rhetoric) and working constructively towards a
> solution, let me know.
widespread) is regression testing of digest-protected REST APIs. Test
client is way around (plus it does not support digest, but as Django
do not support it as well, I can't blame it), and I'd like to use some
other software tools that cannot do whitebox, and urllib cannot be
used due to bug #10117.
I must say, however, that now with some integration things works for
me and I'm not working on integration (as this relies on #2879 anyway,
which was having patch shortly after creation, got ignored and now
laying silently for two years as even mikeal got up for runtime
patching).
Almad
On 28 Led, 06:01, Malcolm Tredinnick <malc...@pointy-stick.com> wrote:
> On Tue, 2009-01-27 at 06:21 -0800, Almad wrote:
> > On Jan 26, 1:19 am, Russell Keith-Magee <freakboy3...@gmail.com>
> > wrote:
> Perhaps you could show some simplified example code? I have at least a
> few dozen tests that do this and they don't have any problem. Serial
impractical, but minimalist example.
> Depends on your definition of "serious". Given the thousands of Django
> applications that have, at least, started out using thedevelopment
> server, your claim seems a little vacuous.
Django is used heavily...and I see a lot of problems caused by lack of
tests. And tests are lacking because of some problems with test
creation in Django I'm trying to point on (in bad manner, right, but
being a bit depressed by running into same issues as two years ago
when I first tried to develop with Django).
> [...]
> there, which is fine, since you aren't one of the people on the hook for
> things like that in Django. The group of core maintainers have decided
> to devote time elsewhere.
BTW I discovered DjangoCerise project (http://xhtml.net/scripts/Django-
CherryPy-server-DjangoCerise); main CP WSGI is 44k; wouldn't be worth
trying to integrate it in django, as it's well-tested and reviewed
enough? I'd do it, if there is any chance for having it in mainline.
> > Well, after I fall into small group of users that do some automatized
> > acceptance testing (and thus require, like, Selenium) and discovered
> > that things looks like working after I lobotomized Django and replaced
> > it with CherryPy, I'm now satisfied.
>
> When you could have just used mod_wsgi, or CherryPy's WSGI server or any
> number of other things. Nice over-reaction there.
setup and as simple as possible, or dev's will not use them (and I see
that around me every day).
> Note, also, that there's a ticket open in Trac to add some extra bits
> and pieces for allowing Selenium integration with Django's test
> framework. That would also give you some idea of the direction things
> could go.
and for requirements #3357 denied by core dev and #2879 being
practically dead
> This was a very disappointing mail to read. Whilst I realise you might
> well be frustrated, the tone is simply insulting to the amount of work
> that has gone into developing and supporting Django
much good work as You and core devs.
> and tries to inflate
> whatever problem it is that you're having (which hasn't been explained
> in detail) way out of proportion.
I should have. I hope I have now better explained my problems and
motivations and why I consider them crutial, coming from TDD
environment.
> Regards,
> Malcolm
Almad
Ned Batchelder
The core devs are not going to change their mind about multi-threading the development server, so please stop asking them to. The disagreement on the table now is whether you have a common problem here, or an esoteric one. When I look at the code in 10117, there doesn't seem to be anything multi-threaded about it. That's good news for you, because it may bring your problem into the category of "will be fixed" from "won't be fixed". If you could help us understand your precise problem more, and by that I mean give us more information about where you think the multi-threading is coming from, or otherwise explain the connection reset, then we could all get beyond the mud-slinging.
--Ned.
http://nedbatchelder.com
-- Ned Batchelder, http://nedbatchelder.com
Almad
> Almad, you've pointed to ticket 10117 a few times now, but I don't see
> how this has anything to do with a single-threaded server. I think if
> you could clarify that point, maybe we'd be able to make some progress
> on this.
but I'll do some more tracing later.
> The core devs are not going to change their mind about multi-threading
> the development server, so please stop asking them to.
> The disagreement
> on the table now is whether you have a common problem here, or an
> esoteric one. When I look at the code in 10117, there doesn't seem to
> be anything multi-threaded about it. That's good news for you, because
> it may bring your problem into the category of "will be fixed" from
> "won't be fixed". If you could help us understand your precise problem
> more, and by that I mean give us more information about where you think
> the multi-threading is coming from, or otherwise explain the connection
> reset, then we could all get beyond the mud-slinging.
, it looks like it might be a wsgi bug. What puzzles me is, that if I
recall my debugging correctly, this bug was fixed for me even without
using CP WSGI, just multithreading Django's one.
> --Ned.http://nedbatchelder.com
Almad