Proposal: First-class WSGI support in Django 1.3 / twod.wsgi
Gustavo Narea
I'd like Django's WSGI support to be as good as in the other WSGI frameworks (or better).
With this in mind, I have developed a project called twod.wsgi [1] which brings full WSGI capabilities to Django, whether running in development or deployment. This project is fully tested, 100% backwards compatible and has now been in use for over 5 months in a production environment.
Ideally I think full WSGI support is something that should be available out-of-the-box in Django and therefore I want to contribute it to Django. (And I'm reopening this discussion, as suggested when I tried to push it to Django 1.2 [2].)
Among many other things, one of the components alone provides solutions to some enterprise requirements for Django <http://groups.google.com/group/django-developers/browse_thread/thread/c89e028a536514d3>:
* It�s now possible to run code at startup time, in a straight-forward yet extremely flexible fashion, which will also work on development servers if you want it to � not only when deployed on a production server.
* You can now stop using a Python module to store your application settings, in order to use an intuitive and widely used
mechanism that scales up and scales down. It will just work and you won�t have to update any other file in your application.
* By providing full WSGI support in development mode, we are able to work around the differences in the process environment between django development server and django hosted using mod_wsgi <http://blog.dscpl.com.au/2010/03/improved-wsgi-script-for-use-with.html>
* It�s finally possible to run WSGI middleware in development servers, the same way you may do it on production servers.
And this is just the tip of the iceberg. By improving Django�s interoperability, we would gain the ability to rapidly integrate many pieces of third party software with Django, or simply use a component which outperforms Django�s current implementation for your requirements.
If you want to read more about how we�ve done this, you can look here: http://packages.python.org/twod.wsgi/manual/index.html#introduction
We're willing to work closely with the core development team to adapt twod.wsgi if necessary and integrate it in Django 1.3.
What do you think?
- Gustavo Narea.
[1] http://packages.python.org/twod.wsgi/
[2] http://groups.google.com/group/django-developers/browse_thread/thread/08c7ffeee7b9343c
Ivan Sagalaev
> Among many other things, one of the components alone provides solutions to some enterprise requirements for Django<http://groups.google.com/group/django-developers/browse_thread/thread/c89e028a536514d3>:
>
> * It�s now possible to run code at startup time, in a straight-forward yet extremely flexible fashion, which will also work on development servers if you want it to � not only when deployed on a production server.
> * You can now stop using a Python module to store your application settings, in order to use an intuitive and widely used
> mechanism that scales up and scales down. It will just work and you won�t have to update any other file in your application.
> * By providing full WSGI support in development mode, we are able to work around the differences in the process environment between django development server and django hosted using mod_wsgi<http://blog.dscpl.com.au/2010/03/improved-wsgi-script-for-use-with.html>
> * It�s finally possible to run WSGI middleware in development servers, the same way you may do it on production servers.
>
> And this is just the tip of the iceberg.
Gustavo,
Could you please give a concise technical overview, in high-level terms,
on what twod.wsgi actually does to Django code?
From the part that I quoted above I see that there are things that
either don't have anything to do with WSGI (I'm fine with settings.py
BTW) or don't explain anything (I didn't have problems with running
Django under Paste along other WSGI apps).
P.S. I won't say for others but for me, as a developer, the phrase "just
the tip of the iceberg" sounds scary, not attractive :-).
Gustavo Narea
On May 26, 4:52 pm, Ivan Sagalaev <man...@softwaremaniacs.org> wrote:
> Could you please give a concise technical overview, in high-level terms,
> on what twod.wsgi actually does to Django code?
Sure. There are different components, so I'll elaborate on them
individually:
Paste Deploy application factory instead of `manage runserver'
--------------------------------------------------------------
The problem with running the application in development mode is that
there's no way to add WSGI middleware, it's like Django assumes that
you'd only need WSGI middleware on deployment.
Therefore I think we should either adopt Paste Script's development
server (which makes your app behave like it'd do under mod_wsgi, for
example; with no Django-specific magic) or implement a development
server that serves the WSGI application you give it (which could be
wrapped around WSGI middleware, or it could not even be a Django
application).
I'd prefer sticking to Paste Script's server because it's multi-
threaded and we could *optionally* use Paste Deploy configuration
files, instead of Python files for configuration:
http://packages.python.org/twod.wsgi/manual/paste-factory.html
(And one of the advantages of using INI files for configuration
instead of Python code is that it's a lot easier to extend:
http://packages.python.org/twod.wsgi/manual/paste-factory.html#multiple-configuration-files)
Better request objects
----------------------
There are two problems with Django's request objects from a WSGI point
of view:
- It copies the WSGI environment variables. That makes
interoperability with WSGI libraries harder or not possible at all,
because the request can be changed but the WSGI environ wouldn't be
modified, and vice versa, if the WSGI environ is modified the request
wouldn't be updated.
- It doesn't expose an API to handle most of the properties of a
request -- only the most common ones.
WebOb's request object is a better proxy of the WSGI environ from my
point of view, which is why I think Django's request object should
extend it. It doesn't have the problems above and it has more
features:
http://packages.python.org/twod.wsgi/manual/request-objects.html
I've managed to sub-class both WebOb.Request and Django's HttpRequest
without breaking backwards compatibility.
Embedded WSGI applications
--------------------------
At present there's no built-in mechanism to run WSGI applications from
Django. Doing so could be extremely powerful and useful in some
situations, because it gives you the ability to filter the request
another application receives, and the response it returns -- Using
request and response objects, respectively.
And by "WSGI" application, I mean pretty much *any* Web application.
Java applications, PHP applications, a Web site like www.google.com.
Anything, thanks to 3rd party WSGI applications that can run CGI
scripts and proxies, for example:
http://pythonpaste.org/modules/cgiapp.html#paste.cgiapp.CGIApplication
http://pythonpaste.org/modules/proxy.html#paste.proxy.Proxy
So, if you have, say, a Trac instance running on your Web site, you
can make it use Django's authentication data for the current user, and
authenticate users with your own Django-powered login form, and log
users out from Django, and many other things. Check this sample Django
application that implements a Single Sign-On mechanism with Trac:
http://bitbucket.org/Gustavo/weesgo/
URL routing arguments support
-----------------------------
From the manual <http://packages.python.org/twod.wsgi/manual/routing-
args.html>:
"routing_args is an extension to the WSGI standard which normalises
the place to put the arguments found in the URL. This is particularly
useful for 3rd party WSGI libraries and the dispatching components in
Web frameworks are expected to set it, but Django does not: Therefore
we created the RoutingArgsMiddleware Django middleware.
If you requested the path /blog/posts/hello-world/comments/3, then the
arguments hello-world and 3 will be available in the request object.
Depending on how you defined your URL patterns, they’ll be a
dictionary (if you used named groups) or a tuple (if you didn’t set
names for the matching groups). The former are referred to as “named
arguments” and the later as “positional arguments” in routing_args
specification.
RoutingArgsMiddleware simply puts the arguments found by the Django
URL resolver in the request object. It’s such a simple thing, but it’s
key for Django-independent libraries, which may not be run in the
context of a Django middleware nor a Django view."
Serving static files (aka "media")
----------------------------------
Letting a WSGI application serve static files on development servers
is better because it's faster (given that Django won't get run) and
more importantly, Django doesn't serve the media on deployment after
all.
See: http://packages.python.org/twod.wsgi/manual/media-apps.html
That's it, AFAIR. Please let me know what you think.
Cheers,
PS: Trust me, that's just the tip of the iceberg ;-)
Gustavo Narea
http://packages.python.org/twod.wsgi/manual/embedded-apps.html
On May 27, 10:08 am, Gustavo Narea <gna...@tech.2degreesnetwork.com>
wrote:
> Hello,
>
> On May 26, 4:52 pm, Ivan Sagalaev <man...@softwaremaniacs.org> wrote:
>
> > Could you please give a concise technical overview, in high-level terms,
> > on what twod.wsgi actually does to Django code?
>
> Sure. There are different components, so I'll elaborate on them
> individually:
>
> Paste Deploy application factory instead of `manage runserver'
> --------------------------------------------------------------
>
> The problem with running the application in development mode is that
> there's no way to add WSGI middleware, it's like Django assumes that
> you'd only need WSGI middleware on deployment.
>
> Therefore I think we should either adopt Paste Script's development
> server (which makes your app behave like it'd do under mod_wsgi, for
> example; with no Django-specific magic) or implement a development
> server that serves the WSGI application you give it (which could be
> wrapped around WSGI middleware, or it could not even be a Django
> application).
>
> I'd prefer sticking to Paste Script's server because it's multi-
> threaded and we could *optionally* use Paste Deploy configuration
> files, instead of Python files for configuration:http://packages.python.org/twod.wsgi/manual/paste-factory.html
>
> (And one of the advantages of using INI files for configuration
> instead of Python code is that it's a lot easier to extend:http://packages.python.org/twod.wsgi/manual/paste-factory.html#multip...)
> scripts and proxies, for example:http://pythonpaste.org/modules/cgiapp.html#paste.cgiapp.CGIApplicationhttp://pythonpaste.org/modules/proxy.html#paste.proxy.Proxy
Russell Keith-Magee
<gna...@tech.2degreesnetwork.com> wrote:
> Hello,
>
> On May 26, 4:52 pm, Ivan Sagalaev <man...@softwaremaniacs.org> wrote:
>> Could you please give a concise technical overview, in high-level terms,
>> on what twod.wsgi actually does to Django code?
>
> Sure. There are different components, so I'll elaborate on them
> individually:
Hi Gustavo,
This is all very helpful information; thanks for breaking it down like this.
I've talked this over with a few people at the sprints, and we've
pretty much ended up at the same point -- we're deeply confused.
Comments inline below.
I also want to be very clear -- this feedback is *not* us saying "go
away", it's us saying "we're not clear on what you're proposing and
why you're proposing it". Please take this criticism in the sprit it
is intended -- to work out exactly what it is that we *aren't* doing,
and more importantly, *why* we should be doing it another way.
> Paste Deploy application factory instead of `manage runserver'
> --------------------------------------------------------------
>
> The problem with running the application in development mode is that
> there's no way to add WSGI middleware, it's like Django assumes that
> you'd only need WSGI middleware on deployment.
>
> Therefore I think we should either adopt Paste Script's development
> server (which makes your app behave like it'd do under mod_wsgi, for
> example; with no Django-specific magic) or implement a development
> server that serves the WSGI application you give it (which could be
> wrapped around WSGI middleware, or it could not even be a Django
> application).
>
> I'd prefer sticking to Paste Script's server because it's multi-
> threaded and we could *optionally* use Paste Deploy configuration
> files, instead of Python files for configuration:
> http://packages.python.org/twod.wsgi/manual/paste-factory.html
So - what you seem to be proposing here is that we add Paste as a
dependency of Django in order to support:
1) A format of configuration that is different to Django's
2) Support for WSGI middlewares in the development server
Despite your protestations, (1) is a matter of opinion, and it's not
an opinion I happen to share. Personally, I really like the fact that
Django's configuration files are Python files and not Yet Another
Configuration Format; I also contest the assertion that INI files are
more configurable.
(2) isn't a matter of opinion; however, it's not a use case that I've
seen a particularly compelling use case for, either. I'm certainly
willing to be convinced otherwise, though -- now is your opportunity
to convince us.
Regardless, as you've demonstrated with twod.wsgi -- support for a
"rich" development server is also something that can be provided as an
external resource. Given that the overhead of this change is adding a
dependency to a project that has made a history of not having
dependencies, I'm more inclined to modify the docs to say "If you
want/need to use WSGI middleware as part of your development
environment, you'll need to test your deployments under a full WSGI
container -- either a full web server, or a lightweight test server
such as the one provided by twod.wsgi".
> Better request objects
> ----------------------
>
> There are two problems with Django's request objects from a WSGI point
> of view:
>
> - It copies the WSGI environment variables. That makes
> interoperability with WSGI libraries harder or not possible at all,
> because the request can be changed but the WSGI environ wouldn't be
> modified, and vice versa, if the WSGI environ is modified the request
> wouldn't be updated.
> - It doesn't expose an API to handle most of the properties of a
> request -- only the most common ones.
>
> WebOb's request object is a better proxy of the WSGI environ from my
> point of view, which is why I think Django's request object should
> extend it. It doesn't have the problems above and it has more
> features:
> http://packages.python.org/twod.wsgi/manual/request-objects.html
>
> I've managed to sub-class both WebOb.Request and Django's HttpRequest
> without breaking backwards compatibility.
Why do we need to subclass WebOb here? Again, you're proposing the
introduction of a major dependency to Django; if there are
bugs/inconsistencies in Django's WSGI implementation, those bugs
should be fixed. I'm not sure I see why introducing WebOb is the right
approach here.
> Embedded WSGI applications
> --------------------------
>
> At present there's no built-in mechanism to run WSGI applications from
> Django. Doing so could be extremely powerful and useful in some
> situations, because it gives you the ability to filter the request
> another application receives, and the response it returns -- Using
> request and response objects, respectively.
>
> And by "WSGI" application, I mean pretty much *any* Web application.
> Java applications, PHP applications, a Web site like www.google.com.
> Anything, thanks to 3rd party WSGI applications that can run CGI
> scripts and proxies, for example:
> http://pythonpaste.org/modules/cgiapp.html#paste.cgiapp.CGIApplication
> http://pythonpaste.org/modules/proxy.html#paste.proxy.Proxy
>
> So, if you have, say, a Trac instance running on your Web site, you
> can make it use Django's authentication data for the current user, and
> authenticate users with your own Django-powered login form, and log
> users out from Django, and many other things. Check this sample Django
> application that implements a Single Sign-On mechanism with Trac:
> http://bitbucket.org/Gustavo/weesgo/
Isn't this the first point again? If you're using mod_wsgi (or any
other WSGI container) you can deploy whatever you want; the problem is
purely the development server.
> URL routing arguments support
> -----------------------------
>
> From the manual <http://packages.python.org/twod.wsgi/manual/routing-
> args.html>:
>
> "routing_args is an extension to the WSGI standard which normalises
> the place to put the arguments found in the URL. This is particularly
> useful for 3rd party WSGI libraries and the dispatching components in
> Web frameworks are expected to set it, but Django does not: Therefore
> we created the RoutingArgsMiddleware Django middleware.
>
> If you requested the path /blog/posts/hello-world/comments/3, then the
> arguments hello-world and 3 will be available in the request object.
> Depending on how you defined your URL patterns, they’ll be a
> dictionary (if you used named groups) or a tuple (if you didn’t set
> names for the matching groups). The former are referred to as “named
> arguments” and the later as “positional arguments” in routing_args
> specification.
>
> RoutingArgsMiddleware simply puts the arguments found by the Django
> URL resolver in the request object. It’s such a simple thing, but it’s
> key for Django-independent libraries, which may not be run in the
> context of a Django middleware nor a Django view."
Ok; so here you're proposing that Django support an extension to the
WSGI specification for which Django provides an alternate
implementation. Until routing args becomes part of the WSGI spec, this
strikes me as something that *should* be housed in an external
project.
> Serving static files (aka "media")
> ----------------------------------
>
> Letting a WSGI application serve static files on development servers
> is better because it's faster (given that Django won't get run) and
> more importantly, Django doesn't serve the media on deployment after
> all.
Again - why should we care about this? We don't need to massively
optimize the devserver; it's a nasty hack to get a test site working
for development purposes. If you have a real deployment, you shouldn't
be using the Django stack to serve media -- you should be using a
lightweight web server that has high throughput for handles files.
This is something that is clearly documented in Django's own docs.
We're aware that we haven't been the most responsive citizens when it
comes to working with the rest of the Python web community, and we
certainly intend to get more active with the WSGI specification
process -- especially as it relates to the Python 3 transition.
However, that doesn't mean we're about to completely change the way
Django works in order to support aspects of the WSGI spec that the
Django community has been able to live without, or for which the
Django community has alternate approaches. We're entirely happy to
accept changes that make Django more WSGI compliant; we're not going
to be so responsive to changes that try to make Django into a
different web framework.
Yours,
Russ Magee %-)
Gustavo Narea
On May 28, 6:13 pm, Russell Keith-Magee <russ...@keith-magee.com>
wrote:
>
> I've talked this over with a few people at the sprints, and we've
> pretty much ended up at the same point -- we're deeply confused.
> Comments inline below.
Sorry, I think I didn't explain some components well and that led to
the confusion. I'll try to do it better this time...
> I also want to be very clear -- this feedback is *not* us saying "go
> away", it's us saying "we're not clear on what you're proposing and
> why you're proposing it". Please take this criticism in the sprit it
> is intended -- to work out exactly what it is that we *aren't* doing,
> and more importantly, *why* we should be doing it another way.
> > Paste Deploy application factory instead of `manage runserver'
> > --------------------------------------------------------------
>
> > The problem with running the application in development mode is that
> > there's no way to add WSGI middleware, it's like Django assumes that
> > you'd only need WSGI middleware on deployment.
>
> > Therefore I think we should either adopt Paste Script's development
> > server (which makes your app behave like it'd do under mod_wsgi, for
> > example; with no Django-specific magic) or implement a development
> > server that serves the WSGI application you give it (which could be
> > wrapped around WSGI middleware, or it could not even be a Django
> > application).
>
> > I'd prefer sticking to Paste Script's server because it's multi-
> > threaded and we could *optionally* use Paste Deploy configuration
> > files, instead of Python files for configuration:
> >http://packages.python.org/twod.wsgi/manual/paste-factory.html
>
> So - what you seem to be proposing here is that we add Paste as a
> dependency of Django in order to support:
>
> 1) A format of configuration that is different to Django's
> 2) Support for WSGI middlewares in the development server
dependency, but support WSGI middleware in a transparent way; for
example, the way Paste Deploy does.
The reason why I proposed Paste is because it's widely used and it'd
prevent us from implementing something that is already implemented.
> Despite your protestations, (1) is a matter of opinion, and it's not
> an opinion I happen to share. Personally, I really like the fact that
> Django's configuration files are Python files and not Yet Another
> Configuration Format; I also contest the assertion that INI files are
> more configurable.
secondary. I'm not that interested in changing the configuration
system, at least not within this WSGI-related proposal: I just wanted
to point out another advantage of using Paste, which you could use if
you want to (it wouldn't be mandatory).
> (2) isn't a matter of opinion; however, it's not a use case that I've
> seen a particularly compelling use case for, either. I'm certainly
> willing to be convinced otherwise, though -- now is your opportunity
> to convince us.
must be present both on development and deployment, you have to get
rid of `manage runserver' and use a development server like the one
from Paste or CherryPy.
There are lots of WSGI middleware out there that you may want/need to
use, and there are even projects like Paste or Repoze whose only goal
is to develop framework-independent WSGI libraries (many if not most
of them are middleware). In the following links you can find some of
the WSGI middleware available and find that most of them are
applicable to both development and deployment:
http://pythonpaste.org/modindex.html (some items aren't middleware,
but WSGI applications)
Gustavo Narea
trimmed by Google Groups.
Let me take a deep breath and write it again... :'(
> but WSGI applications)
Gustavo Narea
On May 28, 6:13 pm, Russell Keith-Magee <russ...@keith-magee.com>
wrote:
> This is all very helpful information; thanks for breaking it down like this.
>
> I've talked this over with a few people at the sprints, and we've
> pretty much ended up at the same point -- we're deeply confused.
> Comments inline below.
Thank you very much for taking time to talk about it. :)
Sorry, I think I didn't explain some components well and that led to
the confusion. I'll try to do it better this time...
> I also want to be very clear -- this feedback is *not* us saying "go
> away", it's us saying "we're not clear on what you're proposing and
> why you're proposing it". Please take this criticism in the sprit it
> is intended -- to work out exactly what it is that we *aren't* doing,
> and more importantly, *why* we should be doing it another way.
Sure, I understand.
but WSGI applications)
http://repoze.org/repoze_components.html#middleware
http://wsgi.org/wsgi/Middleware_and_Utilities
Some of them are alternatives to components offered by Django itself,
which some people might prefer to use. Others are just things that
cannot/shouldn't be done at the framework level.
I'll give you one real-world example:
We don't put all the static files under MEDIA_ROOT for the limitations
described on <http://groups.google.com/group/django-developers/
browse_thread/thread/b333c14f40acd22a>. We have four kind of static
files and we want them to be in different directories:
- Front-end files, like JS or CSS.
- Files generated by our application, like thumbnails.
- Uploaded files.
- Uploaded files whose downloads are restricted.
The first three are served by three WSGI applications on development.
They're served by the Web server on deployment.
The fourth is served by a WSGI application which tells the Web server
whether the requested file can be served or not (i.e., an X-Sendfile
implementation in WSGI). This application is present both on
development and deployment.
Then we have the Django-powered WSGI application. That makes 5 WSGI
applications on development and 2 on deployment.
What makes things work is a WSGI middleware which dispatches the
requests to the right WSGI application. And thanks to Paste, this is a
piece of cake:
"""
# The code is roughly like this:
from django.core.handlers.wsgi import WSGIHandler
from paste.urlmap import URLMap
from paste.urlparser import StaticURLParser
application = URLMap()
application['/'] = WSGIHandler()
application['/files/restricted'] = ProtectedXSendfileApp("/path/to/
protected/files")
if development:
# The following directories are served by the Web server on
deployment:
application['/files/style'] = StaticURLParser("/var/branch/style")
application['/files/generated'] = StaticURLParser("/var/data/
generated-files")
application['/files/uploads'] = StaticURLParser("/var/data/media")
"""
Where URLMap is the WSGI middleware that dispatches each request to
WSGIHandler, ProtectedXSendfileApp or StaticURLParser.
> Regardless, as you've demonstrated with twod.wsgi -- support for a
> "rich" development server is also something that can be provided as an
> external resource. Given that the overhead of this change is adding a
> dependency to a project that has made a history of not having
> dependencies, I'm more inclined to modify the docs to say "If you
> want/need to use WSGI middleware as part of your development
> environment, you'll need to test your deployments under a full WSGI
> container -- either a full web server, or a lightweight test server
> such as the one provided by twod.wsgi".
is that you're limiting it to do simple things only. Neither mod_wsgi
or Paste development servers are more WSGI compliant, AFAIK. I could
run a TurboGears application with your development server if I wanted,
for example.
Unlocking the development server so that it could serve a WSGI
application other than django.core.handlers.wsgi:WSGIHandler would be
a huge step forward, and doing it is a trivial task: That server could
take the WSGI application defined in the setting WSGI_APPLICATION.
When that setting is not set, django.core.handlers.wsgi:WSGIHandler
would be used.
So I could have the following in settings.py:
# ...
WSGI_APPLICATION = "coolapp.wsgi.application"
# ...
And the contents of coolapp.wsgi would be:
from django.core.handlers.wsgi import WSGIHandler
from django.conf import settings
from paste.urlmap import URLMap
from paste.urlparser import StaticURLParser
application = URLMap()
application['/'] = WSGIHandler()
application['/files/restricted'] = ProtectedXSendfileApp("/path/to/
protected/files")
if settings.DEBUG:
# The following directories are served by the Web server on
deployment:
application['/files/style'] = StaticURLParser("/var/branch/
style")
application['/files/generated'] = StaticURLParser("/var/data/
generated-files")
application[settings.MEDIA_URL] =
StaticURLParser(settings.MEDIA_ROOT)
Or, that setting could represent a callable which returns the WSGI
application object.
Paste's server has many more useful features for WSGI applications and
middleware, and more importantly, it doesn't have any Django-specific
magic which can lead to problems on deployment <http://
blog.dscpl.com.au/2010/03/improved-wsgi-script-for-use-with.html>,
which is why I'd suggest to adopt it. But at least having this basic
feature would be much better than what we have now.
> > Better request objects
> > ----------------------
>
> > There are two problems with Django's request objects from a WSGI point
> > of view:
>
> > - It copies the WSGI environment variables. That makes
> > interoperability with WSGI libraries harder or not possible at all,
> > because the request can be changed but the WSGI environ wouldn't be
> > modified, and vice versa, if the WSGI environ is modified the request
> > wouldn't be updated.
> > - It doesn't expose an API to handle most of the properties of a
> > request -- only the most common ones.
>
> > WebOb's request object is a better proxy of the WSGI environ from my
> > point of view, which is why I think Django's request object should
> > extend it. It doesn't have the problems above and it has more
> > features:
> >http://packages.python.org/twod.wsgi/manual/request-objects.html
>
> > I've managed to sub-class both WebOb.Request and Django's HttpRequest
> > without breaking backwards compatibility.
>
> Why do we need to subclass WebOb here? Again, you're proposing the
> introduction of a major dependency to Django; if there are
> bugs/inconsistencies in Django's WSGI implementation, those bugs
> should be fixed. I'm not sure I see why introducing WebOb is the right
> approach here.
limitation of not introducing a new dependency, I would end up copying
code from the WebOb project.
WebOb offers "the" framework-independent Pythonic wrappers for WSGI
requests and responses, so it's widely used and known to work well.
I'd prefer to extend it instead of copying it, taking advantage of any
bugfix in the future.
I believe reusing WebOb would be the best for you the core development
team (less code to write/maintain) and the users, who would get these
new backwards-compatible features thanks to code that has been around
for a while (instead of a less mature implementation specific to
Django).
> > Embedded WSGI applications
> > --------------------------
>
> > At present there's no built-in mechanism to run WSGI applications from
> > Django. Doing so could be extremely powerful and useful in some
> > situations, because it gives you the ability to filter the request
> > another application receives, and the response it returns -- Using
> > request and response objects, respectively.
>
> > And by "WSGI" application, I mean pretty much *any* Web application.
> > Java applications, PHP applications, a Web site likewww.google.com.
> > Anything, thanks to 3rd party WSGI applications that can run CGI
> > scripts and proxies, for example:
> >http://pythonpaste.org/modules/cgiapp.html#paste.cgiapp.CGIApplication
> >http://pythonpaste.org/modules/proxy.html#paste.proxy.Proxy
>
> > So, if you have, say, a Trac instance running on your Web site, you
> > can make it use Django's authentication data for the current user, and
> > authenticate users with your own Django-powered login form, and log
> > users out from Django, and many other things. Check this sample Django
> > application that implements a Single Sign-On mechanism with Trac:
> >http://bitbucket.org/Gustavo/weesgo/
>
> Isn't this the first point again? If you're using mod_wsgi (or any
> other WSGI container) you can deploy whatever you want; the problem is
> purely the development server.
A WSGI application can be almost anything, as I said above. It could
be a Python function that serves the static files under a root
directory. Or a fully-featured, standalone, database-driven, Python-
based application. It could be a CGI/FastCGI script running PHP under-
the-hood. It could be a proxy to a web site running on a local/remote
server. Or it could be something completely different.
And you may need to embed WSGI applications in your Django projects,
so that you can run them from *Django views* in order to filter the
requests they receive and/or filter the responses they return --
Transparently, using Django request and response objects respectively.
TurboGears, Pylons, CherryPy, Werkzeug, repoze.bfg, etc... They all
have built-in mechanisms to embed (aka "mount") WSGI applications, and
I'll give you two examples where doing this is necessary:
Example 1: Single Sign-On
-------------------------
Your Django-powered registration and login systems are already in
place, with nice looking forms for registration and login
respectively. Then, you need to run a Java Web application on the same
Web site, which would be used by the same users of your main Django-
powered Web site.
Wouldn't it be great if Django could handle authentication for that
Java application? If you could filter the requests received by that
application, you could let it know when the request comes from a
logged in user or display your login form when that application tries
to authenticate anonymous users, among many other things.
That would be a piece of cake with an embed application in a Django
view. For example, here is a 7 LOC Single Sign-On integration between
Trac and Django:
http://bitbucket.org/Gustavo/weesgo/src/tip/weesgo/views.py#cl-21
Example 2: Serve applications on-the-fly
----------------------------------------
Say you run a Bitbucket-like Web site, which offers Bazaar and
Mercurial hosting. You may want to serve the branches under the same
URL pattern regardless of the VCS, and enforce the same authorization
controls, again, regardless of the VCS.
Because Bazaar and Mercurial can be run as WSGI applications out-of-
the-box, doing so would be easy:
# Untested code!
from bzr.blah import BzrWSGIApp
from hg.blah import HgWSGIApp
from twod.wsgi import call_wsgi_app
# The URL paths look like: /projects/{username}/{branch_name}/
{branch_path}
def serve_branch(request, username, branch_name, branch_path):
branch = get_object_or_404(Branch, username=username,
branch_name=branch_name)
if not is_authorized(request, branch):
raise Something
if branch.vcs == "bzr":
vcs_wsgi_app = BzrWSGIApp
else:
vcs_wsgi_app = HgWSGIApp
return call_wsgi_app(vcs_wsgi_app, request, branch_path)
It's not pretty common but there are lots of cases for this, specially
for developers working on complex applications which usually iteract
with other applications. And Django is the only mainstream Python
framework that doesn't support this. I believe this should change.
maintains the WSGI specification, and Django doesn't even provide
something similar AFAIK.
Put simply, the routing_args specification requires dispatchers to put
the arguments found in the URL in the WSGI environment dictionary.
Django doesn't put this information in the WSGI environ (or in the
request object, AFAIR), but it's something absolutely trivial to
solve.
This is an implementation in the form of Django middleware, but I
believe an appropriate solution would be to do it in the WSGI handler:
http://bitbucket.org/2degrees/twod.wsgi/src/tip/twod/wsgi/middleware.py
> > Serving static files (aka "media")
> > ----------------------------------
>
> > Letting a WSGI application serve static files on development servers
> > is better because it's faster (given that Django won't get run) and
> > more importantly, Django doesn't serve the media on deployment after
> > all.
>
> Again - why should we care about this? We don't need to massively
> optimize the devserver; it's a nasty hack to get a test site working
> for development purposes. If you have a real deployment, you shouldn't
> be using the Django stack to serve media -- you should be using a
> lightweight web server that has high throughput for handles files.
> This is something that is clearly documented in Django's own docs.
> We're aware that we haven't been the most responsive citizens when it
> comes to working with the rest of the Python web community, and we
> certainly intend to get more active with the WSGI specification
> process -- especially as it relates to the Python 3 transition.
> However, that doesn't mean we're about to completely change the way
> Django works in order to support aspects of the WSGI spec that the
> Django community has been able to live without, or for which the
> Django community has alternate approaches. We're entirely happy to
> accept changes that make Django more WSGI compliant; we're not going
> to be so responsive to changes that try to make Django into a
> different web framework.
require a major rewrite of the framework. And more importantly, people
who don't need these advanced features won't notice any difference*.
So, I don't see any of the things I have proposed as a risk of turning
Django into a different framework.
To sum up, this proposal is all about improving WSGI support in
Django. Adding Paste as a dependency is the approach I personally
recommend for technical reasons, and also because I'd see it as one
step toward a better relationship with the rest of the Python Web
community, which would be beneficial for all of us.
And to be honest, I don't think making history of not having
dependencies is necesarily a good thing. If a 3rd party library would
improve considerably a part of Django, I think it should be added
instead of duplicating efforts to come up with a not-so-good solution
tied to Django. And in this case, I think Paste is a better solution;
the result of the effort put by the wider WSGI community, which Django
could use without breaking backwards compatibility.
Cheers,
- Gustavo Narea.
* Even if you decide to replace the built-in development server with
Paste's, you could run it from "manage runserver"... And so people
will continue to run the same command.
Reinout van Rees
>
> Basically, when you need to integrate a piece of WSGI middleware that
> must be present both on development and deployment, you have to get
> rid of `manage runserver' and use a development server like the one
> from Paste or CherryPy.
Middleware that *must* be present: it reminded me of an old blog post
that warned about mandatory middleware. I think I've read it in more
than one place, though:
http://dirtsimple.org/2007/02/wsgi-middleware-considered-harmful.html
The core idea of that post is that if something is mandatory, it isn't
middleware anymore. It should be part of your application. I don't
really have an opinion about this myself (yet).
Do you know what the current way of thinking on this is?
Reinout
--
Reinout van Rees - rei...@vanrees.org - http://reinout.vanrees.org
Programmer at http://www.nelen-schuurmans.nl
"Military engineers build missiles. Civil engineers build targets"
Gustavo Narea
> On 05/29/2010 01:51 AM, Gustavo Narea wrote:
> > Basically, when you need to integrate a piece of WSGI middleware that
> > must be present both on development and deployment, you have to get
> > rid of `manage runserver' and use a development server like the one
> > from Paste or CherryPy.
>
> Middleware that *must* be present: it reminded me of an old blog post
> that warned about mandatory middleware. I think I've read it in more
> than one place, though:
>
> http://dirtsimple.org/2007/02/wsgi-middleware-considered-harmful.html
>
> The core idea of that post is that if something is mandatory, it isn't
> middleware anymore. It should be part of your application. I don't
> really have an opinion about this myself (yet).
there are lots of middleware out there that expose an API that is used
inside the WSGI application, and doing so is so common that frameworks
like Pylons set up new projects like that. And Django developers may
need/want to use that kind of middleware.
> Do you know what the current way of thinking on this is?
like it, presumably because of the reasons described in that post from
2007.
- Gustavo.
Russell Keith-Magee
> Hello,
>
> On May 28, 6:13 pm, Russell Keith-Magee <russ...@keith-magee.com>
> wrote:
>> This is all very helpful information; thanks for breaking it down like this.
>>
>> I've talked this over with a few people at the sprints, and we've
>> pretty much ended up at the same point -- we're deeply confused.
>> Comments inline below.
>
> Thank you very much for taking time to talk about it. :)
>
> Sorry, I think I didn't explain some components well and that led to
> the confusion. I'll try to do it better this time...
>
>> I also want to be very clear -- this feedback is *not* us saying "go
>> away", it's us saying "we're not clear on what you're proposing and
>> why you're proposing it". Please take this criticism in the sprit it
>> is intended -- to work out exactly what it is that we *aren't* doing,
>> and more importantly, *why* we should be doing it another way.
>
> Sure, I understand.
>
>> So - what you seem to be proposing here is that we add Paste as a
>> dependency of Django in order to support:
>>
>> 1) A format of configuration that is different to Django's
>> 2) Support for WSGI middlewares in the development server
>
> That's right, although the proposal is not mainly to add Paste as a
> dependency, but support WSGI middleware in a transparent way; for
> example, the way Paste Deploy does.
The issue I'm unclear on is what you mean by "transparent". Django
provides an interface that implements the WSGI spec. There may well be
errors in that implementation or violations of the spec, and if there
are, I'm happy to address them. But ultimately, we either implement
the spec or we don't.
If Django's implementation of the WSGI spec is compliant, but you
can't use it with out WSGI components, then to my reading either the
other components aren't implementing the spec, or the WSGI spec is
missing something big.
> The reason why I proposed Paste is because it's widely used and it'd
> prevent us from implementing something that is already implemented.
Your argument also presupposes that Paste is an inherently better
implementation. Well, I don't wan't to brag, but Django also has an
implemented WSGI implementation, and it's just as battle hardened as
Paste. :-)
I won't argue that duplication of code and effort is a virtue. If
there is a benefit to be gained in leveraging someone else's work, and
we can do so without compromising our own project, then we should.
However, this raises the much larger issue of how to manage Django's
relationship with external libraries in a way that maintains our
'beginner friendly' history. The 'minimal dependency' philosophy is,
in my opinion, one of the reasons that Django has been able to get the
traction that it has gained.
There have been some initial discussions about this general problem --
specifically, in relation to introducing support for unittest2.
However, it's a discussion that is much larger than improving WSGI
support, and it's a discussion that we need to have as a community.
>> (2) isn't a matter of opinion; however, it's not a use case that I've
>> seen a particularly compelling use case for, either. I'm certainly
>> willing to be convinced otherwise, though -- now is your opportunity
>> to convince us.
>
> Basically, when you need to integrate a piece of WSGI middleware that
> must be present both on development and deployment, you have to get
> rid of `manage runserver' and use a development server like the one
> from Paste or CherryPy.
Sure - if you have a complex deployment (by which I mean "anything
other than a single Django project and some static files"), you need
to have a complex deployment scheme. I don't see anything unusual in
that. Django ships a development server as a "getting started fast"
hack; it's not intended to be anything remotely approaching a useful
web server, and we're going to resist any attempts to turn it into
one. Django isn't a web server, and the development server isn't
intended for production use.
> There are lots of WSGI middleware out there that you may want/need to
> use, and there are even projects like Paste or Repoze whose only goal
> is to develop framework-independent WSGI libraries (many if not most
> of them are middleware). In the following links you can find some of
> the WSGI middleware available and find that most of them are
> applicable to both development and deployment:
> http://pythonpaste.org/modindex.html (some items aren't middleware,
> but WSGI applications)
> http://repoze.org/repoze_components.html#middleware
> http://wsgi.org/wsgi/Middleware_and_Utilities
>
> Some of them are alternatives to components offered by Django itself,
> which some people might prefer to use. Others are just things that
> cannot/shouldn't be done at the framework level.
>
> I'll give you one real-world example:
>
> We don't put all the static files under MEDIA_ROOT for the limitations
> described on <http://groups.google.com/group/django-developers/
> browse_thread/thread/b333c14f40acd22a>. We have four kind of static
> files and we want them to be in different directories:
>
> - Front-end files, like JS or CSS.
> - Files generated by our application, like thumbnails.
> - Uploaded files.
> - Uploaded files whose downloads are restricted.
This is a great example of a use case where you shouldn't be using
Django's development server at all. You have a bunch of serving rules
for static content. A proper web server can handle those. But as far
as developing and testing your Django application is concerned, it
shouldn't matter how the static files are served - a static file
resource is either available or it isn't.
When you move to production, you're going to have to test the
integration of the various components inside your production web
server; I'm not sure I see hiding the need for this integration
testing as a virtue.
>> Regardless, as you've demonstrated with twod.wsgi -- support for a
>> "rich" development server is also something that can be provided as an
>> external resource. Given that the overhead of this change is adding a
>> dependency to a project that has made a history of not having
>> dependencies, I'm more inclined to modify the docs to say "If you
>> want/need to use WSGI middleware as part of your development
>> environment, you'll need to test your deployments under a full WSGI
>> container -- either a full web server, or a lightweight test server
>> such as the one provided by twod.wsgi".
>
> Django's development server *is* a WSGI compliant gateway, the problem
> is that you're limiting it to do simple things only. Neither mod_wsgi
> or Paste development servers are more WSGI compliant, AFAIK. I could
> run a TurboGears application with your development server if I wanted,
> for example.
Yes - but the limitation is (at least partially) by design. We don't
want anyone to be under the impression that Django's development
server is anywhere close to being suitable for production. The more we
make the development server behave like a production server, the
harder it is to maintain that distinction.
> Unlocking the development server so that it could serve a WSGI
> application other than django.core.handlers.wsgi:WSGIHandler would be
> a huge step forward, and doing it is a trivial task: That server could
> take the WSGI application defined in the setting WSGI_APPLICATION.
> When that setting is not set, django.core.handlers.wsgi:WSGIHandler
> would be used.
...
Ok - I'm still not completely sold on the need for this, but I'm at
least interested in seeing some code to see how big a change it would
be. If it doesn't require a whole lot of complexity, it may be worth
making this change.
WebOb is not *the* set of Pythonic wrappers - it's *a" set of Pythonic
wrappers. Yes, it's the implementation that is used by many web
frameworks that aren't Django, but that doesn't automatically make it
better. WebOb may be used by more frameworks, but I'd argue Django is
used on more actual websites.
As for 'less mature' -- Django has been open sourced for 5 years, and
was in closed-source development for a couple of years before that.
It's used by many high traffic sites. Claiming Django's implementation
is immature borders on being troll bait. You'll get a lot more
traction for your arguments if you stay away from the epithets.
I'm a lot more interested in seeing the *specific* changes that are
required in order to make Django's request/response WSGI compliant.
...
> It's not pretty common but there are lots of cases for this, specially
> for developers working on complex applications which usually iteract
> with other applications. And Django is the only mainstream Python
> framework that doesn't support this. I believe this should change.
Ok - I misunderstood your argument. This sounds like a reasonable
suggestion; again -- what specifically is the flaw/bug in Django that
makes this impossible at present? What changes are required?
Ok; I wasn't aware this was an official spec. I'd need to see the spec
before I could pass comment on this. Can you provide a reference?
I'd also point out that just because a specification exists, doesn't
necessarily mean that we have to implement it. Django has a way to
parse URL arguments, and it's fundamentally different to the "routing"
approach advocated by Rails and Pylons; if there is a way for the two
to coexist, I'm happy to look at it, but again, I'm not especially
interested in making Django "more like Pylons" just so we can satisfy
a spec. For example, CGI is a perfectly functioning spec - that
doesn't mean we have to support it.
>> We're aware that we haven't been the most responsive citizens when it
>> comes to working with the rest of the Python web community, and we
>> certainly intend to get more active with the WSGI specification
>> process -- especially as it relates to the Python 3 transition.
>> However, that doesn't mean we're about to completely change the way
>> Django works in order to support aspects of the WSGI spec that the
>> Django community has been able to live without, or for which the
>> Django community has alternate approaches. We're entirely happy to
>> accept changes that make Django more WSGI compliant; we're not going
>> to be so responsive to changes that try to make Django into a
>> different web framework.
>
> The changes I am proposing are all backwards compatible. None of them
> require a major rewrite of the framework. And more importantly, people
> who don't need these advanced features won't notice any difference*.
> So, I don't see any of the things I have proposed as a risk of turning
> Django into a different framework.
Well, you say these changes don't require a major rewrite, but then
you propose introducing two major external dependencies. When you
factor in the social cost of introducing dependencies, it's not a
trivial set of changes that you're proposing.
In summary, here's what I'm hearing:
You have four proposals:
1) Modifying the development server to provide a configurable WSGI interface
2) Cleaning up request/response objects
3) Allowing embedded WSGI applications
4) Support URL routing arguments.
I'm not sold on the need for (1), but if it can be implemented in a
simple patch, it's possibly worth doing.
(2) and (3) sounds like a legitimate bugs/incompatibilities that are
worth fixing.
I don't know enough about (4) to be able to judge it.
I'd be interested to see patches; Is your django-wsgi branch [1] still
a useful resource to look at? From a quick inspection of the twod.wsgi
code, it appears to be implemented in the 'use Paste and WebOb' style;
do you have any version of the code that is implemented as patches
against trunk Django?
[1] http://bitbucket.org/Gustavo/django-wsgi/
I'm also interested how this relates to the work that was done on
Django's WSGI interface during last year's GSoC. Are you at all
familiar with the work on this branch? If so, what is the extent of
the overlap between your code and that branch?
Yours
Russ Magee %-)
Gustavo Narea
I am not saying that Django's WSGI implementation doesn't comply with
the specification. In fact, I've been talking about improving "WSGI
support" not "WSGI compliance".
It does comply with the specification, but just internally without
exposing all the WSGI-related functionality that could be consumed by
3rd party libraries. In other words, Django plays well with the
gateway using WSGI, but makes it really hard for a third component to
come into play.
I should've been more precise and have said "Open up WSGI support for
3rd party libraries", instead of simply "improve WSGI support".
On Jun 2, 4:54 pm, Russell Keith-Magee <russ...@keith-magee.com>
wrote:
> > dependency, but support WSGI middleware in a transparent way; for
> > example, the way Paste Deploy does.
>
> The issue I'm unclear on is what you mean by "transparent". Django
> provides an interface that implements the WSGI spec. There may well be
> errors in that implementation or violations of the spec, and if there
> are, I'm happy to address them. But ultimately, we either implement
> the spec or we don't.
>
> If Django's implementation of the WSGI spec is compliant, but you
> can't use it with out WSGI components, then to my reading either the
> other components aren't implementing the spec, or the WSGI spec is
> missing something big.
and the resulting callable would be the final WSGI application that
should be taken by the development and deployment gateway (for
example, `manage runserver' and mod_wsgi, respectively).
This callable could be the same in both scenarios, unless the
developer explicitly changes it.
> > The reason why I proposed Paste is because it's widely used and it'd
> > prevent us from implementing something that is already implemented.
>
> Your argument also presupposes that Paste is an inherently better
> implementation. Well, I don't wan't to brag, but Django also has an
> implemented WSGI implementation, and it's just as battle hardened as
> Paste. :-)
>
> I won't argue that duplication of code and effort is a virtue. If
> there is a benefit to be gained in leveraging someone else's work, and
> we can do so without compromising our own project, then we should.
> However, this raises the much larger issue of how to manage Django's
> relationship with external libraries in a way that maintains our
> 'beginner friendly' history. The 'minimal dependency' philosophy is,
> in my opinion, one of the reasons that Django has been able to get the
> traction that it has gained.
backwards compatibility, but also all the documentation written so far
will still be relevant.
We wouldn't be changing existing behavior, but adding new
functionality for "advanced" users. Things would only change for those
of us who write framework-independent WSGI libraries and Django users
who want to integrate such libraries.
> There have been some initial discussions about this general problem --
> specifically, in relation to introducing support for unittest2.
> However, it's a discussion that is much larger than improving WSGI
> support, and it's a discussion that we need to have as a community.
> > There are lots of WSGI middleware out there that you may want/need to
> > use, and there are even projects like Paste or Repoze whose only goal
> > is to develop framework-independent WSGI libraries (many if not most
> > of them are middleware). In the following links you can find some of
> > the WSGI middleware available and find that most of them are
> > applicable to both development and deployment:
is that you may need/want to use WSGI middleware on development too:
Ignoring the static files that could be served by Django or a
production server in the example above, there are two WSGI
applications that should be served on development and deployment, and
having a WSGI middleware that dispatches requests to the right
application is one solution.
> >> Regardless, as you've demonstrated with twod.wsgi -- support for a
> >> "rich" development server is also something that can be provided as an
> >> external resource. Given that the overhead of this change is adding a
> >> dependency to a project that has made a history of not having
> >> dependencies, I'm more inclined to modify the docs to say "If you
> >> want/need to use WSGI middleware as part of your development
> >> environment, you'll need to test your deployments under a full WSGI
> >> container -- either a full web server, or a lightweight test server
> >> such as the one provided by twod.wsgi".
>
> > Django's development server *is* a WSGI compliant gateway, the problem
> > is that you're limiting it to do simple things only. Neither mod_wsgi
> > or Paste development servers are more WSGI compliant, AFAIK. I could
> > run a TurboGears application with your development server if I wanted,
> > for example.
>
> Yes - but the limitation is (at least partially) by design. We don't
> want anyone to be under the impression that Django's development
> server is anywhere close to being suitable for production. The more we
> make the development server behave like a production server, the
> harder it is to maintain that distinction.
hard to add WSGI middleware to a Django project during development.
> > Unlocking the development server so that it could serve a WSGI
> > application other than django.core.handlers.wsgi:WSGIHandler would be
> > a huge step forward, and doing it is a trivial task: That server could
> > take the WSGI application defined in the setting WSGI_APPLICATION.
> > When that setting is not set, django.core.handlers.wsgi:WSGIHandler
> > would be used.
>
> ...
>
> Ok - I'm still not completely sold on the need for this, but I'm at
> least interested in seeing some code to see how big a change it would
> be. If it doesn't require a whole lot of complexity, it may be worth
> making this change.
that server internally (i.e., the AdminMediaHandler).
In runserver.py, you could replace:
try:
handler = AdminMediaHandler(WSGIHandler(), admin_media_path)
with:
wsgi_app_str = getattr(settings, "WSGI_APPLICATION",
"django.core.handlers.wsgi.WSGIHandler")
wsgi_app = import_object(wsgi_app_str)
try:
handler = AdminMediaHandler(wsgi_app, admin_media_path)
import_object() is a function I just made up; I'm not sure if there's
something like that in Django already. We could either implement it on
top of importlib or do something different.
new features: "these new backwards-compatible features thanks to code
implementation specific to Django)".
mature". I called "less mature" any potential *change* *in* *Django*
in order to solve problems WebOb already solves.
> I'm a lot more interested in seeing the *specific* changes that are
> required in order to make Django's request/response WSGI compliant.
interoperability: Some items from the WSGI environment are copied,
then the request is modified but the changes are never applied to the
original WSGI environment. This has two consequences:
1.- WSGI libraries end up working on an outdated WSGI environ.
2.- Any change in the WSGI environment is not propagated to the
Django request object.
As an example, I can mention authentication: Django authenticates the
user and places the object on request.user, but according to the WSGI
environment, the user is anonymous (because REMOTE_USER is not set).
This would be solved by implementing getters and setters in
HttpRequest, instead of copying values from the WSGI environ. Here's a
list of the things that could be proxied, for example:
http://pythonpaste.org/webob/class-webob.Request.html
> > A WSGI application can be almost anything, as I said above. It could
> > be a Python function that serves the static files under a root
> > directory. Or a fully-featured, standalone, database-driven, Python-
> > based application. It could be a CGI/FastCGI script running PHP under-
> > the-hood. It could be a proxy to a web site running on a local/remote
> > server. Or it could be something completely different.
> ...
> > It's not pretty common but there are lots of cases for this, specially
> > for developers working on complex applications which usually iteract
> > with other applications. And Django is the only mainstream Python
> > framework that doesn't support this. I believe this should change.
>
> Ok - I misunderstood your argument. This sounds like a reasonable
> suggestion; again -- what specifically is the flaw/bug in Django that
> makes this impossible at present? What changes are required?
code instead of updating existing code.
> I'd also point out that just because a specification exists, doesn't
> necessarily mean that we have to implement it. Django has a way to
> parse URL arguments, and it's fundamentally different to the "routing"
> approach advocated by Rails and Pylons; if there is a way for the two
> to coexist, I'm happy to look at it, but again, I'm not especially
> interested in making Django "more like Pylons" just so we can satisfy
> a spec. For example, CGI is a perfectly functioning spec - that
> doesn't mean we have to support it.
to put the arguments you found in the URL, in the WSGI environ.
> > The changes I am proposing are all backwards compatible. None of them
> > require a major rewrite of the framework. And more importantly, people
> > who don't need these advanced features won't notice any difference*.
> > So, I don't see any of the things I have proposed as a risk of turning
> > Django into a different framework.
>
> Well, you say these changes don't require a major rewrite, but then
> you propose introducing two major external dependencies. When you
> factor in the social cost of introducing dependencies, it's not a
> trivial set of changes that you're proposing.
make Django into a different framework (in reply to your previous
email).
> In summary, here's what I'm hearing:
>
> You have four proposals:
> 1) Modifying the development server to provide a configurable WSGI interface
> 2) Cleaning up request/response objects
> 3) Allowing embedded WSGI applications
> 4) Support URL routing arguments.
> I'm not sold on the need for (1), but if it can be implemented in a
> simple patch, it's possibly worth doing.
>
> (2) and (3) sounds like a legitimate bugs/incompatibilities that are
> worth fixing.
>
> I don't know enough about (4) to be able to judge it.
information we need, it just has to put it in the WSGI environ.
> I'd be interested to see patches; Is your django-wsgi branch [1] still
> a useful resource to look at? From a quick inspection of the twod.wsgi
> code, it appears to be implemented in the 'use Paste and WebOb' style;
> do you have any version of the code that is implemented as patches
> against trunk Django?
>
> [1]http://bitbucket.org/Gustavo/django-wsgi/
requests that proxy the environ, I decided to create twod.wsgi in
order to extend WebOb -- So it doesn't have a fix for (2).
The implementation of (3) in twod.wsgi is slightly better (and a lot
simpler because of WebOb).
I don't have a patch for (1), yet.
> I'm also interested how this relates to the work that was done on
> Django's WSGI interface during last year's GSoC. Are you at all
> familiar with the work on this branch? If so, what is the extent of
> the overlap between your code and that branch?
Cheers,
- Gustavo.
Gustavo Narea
Not that I want to rush this, but have you had time to read my last
email or made a decision? :-)
Cheers,
- Gustavo.
Russell Keith-Magee
> Hello, Russell et al.
Apologies for the delay in replying -- life has been a little hectic of late.
>> > Unlocking the development server so that it could serve a WSGI
>> > application other than django.core.handlers.wsgi:WSGIHandler would be
>> > a huge step forward, and doing it is a trivial task: That server could
>> > take the WSGI application defined in the setting WSGI_APPLICATION.
>> > When that setting is not set, django.core.handlers.wsgi:WSGIHandler
>> > would be used.
>>
>> ...
>>
>> Ok - I'm still not completely sold on the need for this, but I'm at
>> least interested in seeing some code to see how big a change it would
>> be. If it doesn't require a whole lot of complexity, it may be worth
>> making this change.
>
> It should be easy, specially since you already use WSGI middleware in
> that server internally (i.e., the AdminMediaHandler).
>
> In runserver.py, you could replace:
> try:
> handler = AdminMediaHandler(WSGIHandler(), admin_media_path)
>
> with:
> wsgi_app_str = getattr(settings, "WSGI_APPLICATION",
> "django.core.handlers.wsgi.WSGIHandler")
> wsgi_app = import_object(wsgi_app_str)
>
> try:
> handler = AdminMediaHandler(wsgi_app, admin_media_path)
If this is all we need to do, then why didn't you say so? My
understanding of your original proposal was that we needed to
integrate Paste for some reason -- but if we just need to open up a
configuration point, then I don't have any particular objections.
Write up a patch, docs and tests and lets get it into trunk.
My only caveat on this is that I want to make sure we do it right.
Graham Dumpleton recently(ish) blogged about Django configuration
within mod_wsgi [1]. His comments largely concern inconsistencies
between the development server and deployment. I still need to fully
digest what he has written; if there's any action required or
appropriate on our part, I want to make sure we've integrated those
changes before we start encouraging widespread use of new WSGI
configuration points.
[1] http://blog.dscpl.com.au/2010/03/improved-wsgi-script-for-use-with.html
> import_object() is a function I just made up; I'm not sure if there's
> something like that in Django already. We could either implement it on
> top of importlib or do something different.
Django has django.utils.importlib.load_module, which mirrors the
importlib tools available in Python 2.7. This method is used
extensively in Django as a way of implementing extensions and backends
for various features.
I maintain your statement is at best ambiguous; however, I'll take you
at your word that you weren't trolling.
> I did *not* say Django or the current request objects were "less
> mature". I called "less mature" any potential *change* *in* *Django*
> in order to solve problems WebOb already solves.
We have two choices here:
* A small set of (possibly complex) changes to fix bugs in Django's
request layer.
* A large set of changes to replace Django's request layer with WebOb.
I simply don't accept that (1) is necessarily more complex than (2).
Replacing core infrastructure is *never* a trivial activity, and in
this case, replacing a piece of core infrastructure means introducing
a major dependency into Django as a framework. Simply stating "WebOb
is better" won't convince me here -- you'll need to demonstrate why.
>> In summary, here's what I'm hearing:
>>
>> You have four proposals:
>> 1) Modifying the development server to provide a configurable WSGI interface
>> 2) Cleaning up request/response objects
>> 3) Allowing embedded WSGI applications
>> 4) Support URL routing arguments.
>
> That's right.
...
>> I'd be interested to see patches; Is your django-wsgi branch [1] still
>> a useful resource to look at? From a quick inspection of the twod.wsgi
>> code, it appears to be implemented in the 'use Paste and WebOb' style;
>> do you have any version of the code that is implemented as patches
>> against trunk Django?
>>
>> [1]http://bitbucket.org/Gustavo/django-wsgi/
>
> That branch has fixes for (3) and (4). When I had the need for
> requests that proxy the environ, I decided to create twod.wsgi in
> order to extend WebOb -- So it doesn't have a fix for (2).
>
> The implementation of (3) in twod.wsgi is slightly better (and a lot
> simpler because of WebOb).
>
> I don't have a patch for (1), yet.
Ok - at this point, I'm broadly happy with your proposals (subject to
the caveats I've given along the way). The next step is to show us
actual code. This won't get applied to trunk as a single monolithic
"fix WSGI" patch - it needs to be (at least) 4 patches, one for each
feature that is being added, and each patch will need to be tested
(and if appropriate, documented). Each of these features should also
be logged as a ticket (if they aren't already).
If you're looking to maximize the likelihood that this will land in
trunk, the best approach will be to concentrate on one of the four
issues (preferably starting with a small one), and see it through to
completion; then move onto the next issue, and so on.
>> I'm also interested how this relates to the work that was done on
>> Django's WSGI interface during last year's GSoC. Are you at all
>> familiar with the work on this branch? If so, what is the extent of
>> the overlap between your code and that branch?
>
> AFAIR, that branch would've solved (3) only.
It might only solve (3) from your list, but I'm pretty sure it solves
a bunch of other issues as well. The final diff for the branch
contains a lot of code, and theoretically represents 12 weeks of
full-time work; I may have to try poking Chris Cahoon off-list for a
description of what is contained in his branch. If there is other ore
in that branch (and I suspect there is), we should be aiming to mine
it.
Yours,
Russ Magee %-)
Graham Dumpleton
On Jun 14, 10:39 pm, Russell Keith-Magee <russ...@keith-magee.com>
wrote:
From what I remember of this thread, haven't been following it too
closely though, I would think the idea of relying on WebOb and Paste
Deploy is a bad idea.
There are bigger issues around WSGI deployment that need to be solved
and using Paste Deploy as it is now would only make those larger
deployment issues even harder to solve in a clean way.
Graham
Russell Keith-Magee
Sounds like a plan. I'll do some reading on the flight over to make
sure I'm up to speed.
Yours,
Russ Magee %-)
Gustavo Narea
Russell said:
> > It should be easy, specially since you already use WSGI middleware in
> > that server internally (i.e., the AdminMediaHandler).
> >
> > In runserver.py, you could replace:
> > try:
> > handler = AdminMediaHandler(WSGIHandler(), admin_media_path)
> >
> > with:
> > wsgi_app_str = getattr(settings, "WSGI_APPLICATION",
> > "django.core.handlers.wsgi.WSGIHandler")
> > wsgi_app = import_object(wsgi_app_str)
> >
> > try:
> > handler = AdminMediaHandler(wsgi_app, admin_media_path)
>
> If this is all we need to do, then why didn't you say so? My
> understanding of your original proposal was that we needed to
> integrate Paste for some reason -- but if we just need to open up a
> configuration point, then I don't have any particular objections.
> Write up a patch, docs and tests and lets get it into trunk.
That change simply makes using WSGI middleware possible in the development
server.
> My only caveat on this is that I want to make sure we do it right.
> Graham Dumpleton recently(ish) blogged about Django configuration
> within mod_wsgi [1]. His comments largely concern inconsistencies
> between the development server and deployment. I still need to fully
> digest what he has written; if there's any action required or
> appropriate on our part, I want to make sure we've integrated those
> changes before we start encouraging widespread use of new WSGI
> configuration points.
>
> [1] http://blog.dscpl.com.au/2010/03/improved-wsgi-script-for-use-with.html
Inconsistencies between the development server and production environments
wouldn't be solved with that patch.
Any Django-agnostic development server, like Paste's, would solve it because
developers will get the same behavior across environments. This is one of the
reasons why I recommended Paste Script.
I think Graham is trying to fix a wider problem related to deployment. The
inconsistency issue in particular could be avoided if the development server
didn't perform Django-specific routines at startup which no other WSGI gateway
would do: Not importing any module at startup time; do it lazily instead.
When called, the "manage" script imports the modules in INSTALLED_APPS to
generate the list of management commands which triggers other actions which
would not happen under other servers. Put simply, running the development
server within the "manage" script is the cause of the inconsistencies.
The other solution I can think of would be doing exactly the opposite: Make
the WSGIHandler trigger the actions caused by the "manage" script (e.g.,
importing all the INSTALLED_APPS at startup time, validate models).
> > import_object() is a function I just made up; I'm not sure if there's
> > something like that in Django already. We could either implement it on
> > top of importlib or do something different.
>
> Django has django.utils.importlib.load_module, which mirrors the
> importlib tools available in Python 2.7. This method is used
> extensively in Django as a way of implementing extensions and backends
> for various features.
Good, I'll keep that in mind.
> > I did *not* say Django or the current request objects were "less
> > mature". I called "less mature" any potential *change* *in* *Django*
> > in order to solve problems WebOb already solves.
>
> We have two choices here:
> * A small set of (possibly complex) changes to fix bugs in Django's
> request layer.
> * A large set of changes to replace Django's request layer with WebOb.
>
> I simply don't accept that (1) is necessarily more complex than (2).
> Replacing core infrastructure is *never* a trivial activity, and in
> this case, replacing a piece of core infrastructure means introducing
> a major dependency into Django as a framework. Simply stating "WebOb
> is better" won't convince me here -- you'll need to demonstrate why.
From a WSGI/interoperability side of things, it's better because it's
stateless. Every change you make is propagated/applied to the WSGI environ,
not kept to itself.
As for Django-specific things, I think the HttpRequest object should set
environ['REMOTE_USER'] to request.user.username when request.user is set to an
authenticated user... for example.
From a broader non-WSGI-specific perspective, WebOb also provides getters and
setters that proxy the environ to many things not covered by
HttpRequest/WSGIRequest in Django: In Django requests, the headers are
available under request.META["HTTP_*"] and their values are *always*
*strings*. WebOb lets you handle then using appropriate Python types; e.g.,
the content length is an integer, the Date header is a datetime object.
Here's the complete list of getters/setters:
http://pythonpaste.org/webob/class-webob.Request.html
and, just in case, here's the source code:
http://bitbucket.org/ianb/webob/src/tip/webob/request.py#cl-88
If these reasons are not enough to add WebOb as a dependency, I'm still
willing to provide a patch to at least make WSGIRequest propagate the relevant
changes to the WSGI environ, which is my #1 concern.
> Ok - at this point, I'm broadly happy with your proposals (subject to
> the caveats I've given along the way). The next step is to show us
> actual code. This won't get applied to trunk as a single monolithic
> "fix WSGI" patch - it needs to be (at least) 4 patches, one for each
> feature that is being added, and each patch will need to be tested
> (and if appropriate, documented). Each of these features should also
> be logged as a ticket (if they aren't already).
>
> If you're looking to maximize the likelihood that this will land in
> trunk, the best approach will be to concentrate on one of the four
> issues (preferably starting with a small one), and see it through to
> completion; then move onto the next issue, and so on.
That sounds good to me. I'll get to work on the patch to make WSGI middleware
possible in the development server.
> >> I'm also interested how this relates to the work that was done on
> >> Django's WSGI interface during last year's GSoC. Are you at all
> >> familiar with the work on this branch? If so, what is the extent of
> >> the overlap between your code and that branch?
> >
> > AFAIR, that branch would've solved (3) only.
>
> It might only solve (3) from your list, but I'm pretty sure it solves
> a bunch of other issues as well. The final diff for the branch
> contains a lot of code, and theoretically represents 12 weeks of
> full-time work; I may have to try poking Chris Cahoon off-list for a
> description of what is contained in his branch. If there is other ore
> in that branch (and I suspect there is), we should be aiming to mine
> it.
Yes, it's likely it also does more things. I checked that branch ~7 months
ago.
Cheers.
--
Gustavo Narea <xri://=Gustavo>.
| Tech blog: =Gustavo/(+blog)/tech ~ About me: =Gustavo/about |
Gustavo Narea
Graham said:
> From what I remember of this thread, haven't been following it too
> closely though, I would think the idea of relying on WebOb and Paste
> Deploy is a bad idea.
>
> There are bigger issues around WSGI deployment that need to be solved
> and using Paste Deploy as it is now would only make those larger
> deployment issues even harder to solve in a clean way.
Initially, I suggested Paste Script as a replacement for the development
server because it doesn't do any Django-specific thing at startup time, which
then may lead to inconsistencies on deployment.
I did suggest Paste Deploy, but not to solve said inconsistencies. It was
because of the INI configuration which also supports adding WSGI middleware
declaratively.
But both of them were ruled out in this thread.
Could you please elaborate on why relying on WebOb would be a bad idea?
Graham Dumpleton
On Jun 16, 8:11 am, Gustavo Narea <m...@gustavonarea.net> wrote:
> Hello, Graham.
>
> Graham said:
>
> > From what I remember of this thread, haven't been following it too
> > closely though, I would think the idea of relying on WebOb and Paste
> > Deploy is a bad idea.
>
> > There are bigger issues around WSGI deployment that need to be solved
> > and using Paste Deploy as it is now would only make those larger
> > deployment issues even harder to solve in a clean way.
>
> Initially, I suggested Paste Script as a replacement for the development
> server because it doesn't do any Django-specific thing at startup time, which
> then may lead to inconsistencies on deployment.
>
> I did suggest Paste Deploy, but not to solve said inconsistencies. It was
> because of the INI configuration which also supports adding WSGI middleware
> declaratively.
>
> But both of them were ruled out in this thread.
>
> Could you please elaborate on why relying on WebOb would be a bad idea?
possible. As much as that may cause some people problem as far as
flexibility to do some stuff, in the main it is probably better for
the majority that it is like that.
Also, WebOb isn't the only request/response object package around,
there is also Werkzeug and others. Who is to judge which is really the
best even if one was contemplated for Django as a dependency. Overall,
given Django's goal of being as self contained as possible, why would
it be wrong for Django to modify their existing request object to be
more compatible with a WSGI architecture. I certainly see no middle
ground where all the existing request/response object packages authors
are going to cooperate together and come up with one best practice
implementation that becomes a standard, they cant even agree what to
do about WSGI on Python 3.
End result is that if you choose one third party package over another
you are always going to piss off some people, it is politically a much
easier path for Django to do this sort of stuff itself to achieve
better WSGI interoperability around ability to reuse WSGI components.
By doing so you also avoid the whole problem of release management
where dependent on third party packages and Django authors can stay in
control and dictate their own destiny.
I also look at TurboGears as a an extreme example of where what you
want to do can go and have to shake my head sometimes. The large
number of dependencies that can result in a TurboGears application
sees a huge amount of third party modules being imported, often with
many not actually being used or only minimally used. All this does is
to bloat out the process and causes significant load times for the
applications when first started up causing restrictions on how it can
be hosted. This large dependency graph also causes problems at time
with ensuring you have the exact required versions of all modules for
things to work properly which goes back to the release management
problems.
So, as much as you may see the change as minimal, it is the thin end
of the wedge and Django is doing well with its current approach, so I
really don't see the need to change if the current way of doing things
is working for what Django is trying to provide.
Graham
Graham Dumpleton
environment setup that isn't done by other WSGI hosting mechanisms.
Specifically, it initialises Python logging in a way suited to Paste
based applications. Like with Django development server, you can
develop a Paste based application on Paste server which will then not
work on other WSGI hosting mechanisms.
> This is one of the
> reasons why I recommended Paste Script.
>
> I think Graham is trying to fix a wider problem related to deployment. The
> inconsistency issue in particular could be avoided if the development server
> didn't perform Django-specific routines at startup which no other WSGI gateway
> would do: Not importing any module at startup time; do it lazily instead.
pointing out the issues with Django deployment, but I haven't even
mentioned them in these posts and don't intend to.
What you are trying to do with using Paste Script/Deploy is tinkering
at the edges of a larger issue you probably don't even know exists or
don't appreciate. Paste Script/Deploy will not solve that larger issue
as there are lots of things it doesn't do. Adopting it will only make
it harder to solve the real problems later on.
Trying to address the bigger issue is going to need some cooperation
between all frameworks and hosting solution authors to come up with a
consistent way of doing certain stuff. Right now though I don't have
time to lay out properly what the issues are or implement a proof of
concept which illustrates the direction one could take to solve the
problem. It may be the case that Paste Script/Deploy could be adapted
to meet that requirement at some point, but by itself, because the
focus of Paste Deploy is more about constructing WSGI components
together to create an application, as opposed to solving the problem
of deployment on a specific hosting mechanism, I feel right now it is
a poor choice.
If time allows I hope to talk to Russell about some of this stuff when
he is at PyCon in Sydney, but it will still need a lot more work after
that.
Ultimately if something doesn't get done to address the bigger issues,
the WSGI deployment (distinct from composing together WSGI components
into an application), will continue to be a dogs breakfast.
Graham
Gustavo Narea
On 16/06/10 01:33, Graham Dumpleton wrote:
> Paste has its own problems. Paste server itself does process
> environment setup that isn't done by other WSGI hosting mechanisms.
> Specifically, it initialises Python logging in a way suited to Paste
> based applications. Like with Django development server, you can
> develop a Paste based application on Paste server which will then not
> work on other WSGI hosting mechanisms.
>
Paste Script will set up logging its way if and only if you explicitly
tell it to; i.e., defining logging configuration in the Paste Deploy
file or passing an argument to `paster'. I cannot think of another thing
that it does at startup time which may affect deployment, but I may be
wrong.
I think there's always the risk of things working on a development
server but not on the production environment, but the thing is how that
risk can be minimized.
I have no idea what those problems are, so I never meant to solve them.
I just found two issues and Paste Deploy/Script seemed like a solution.
I'm curious about them though. I'll keep an eye on your blog and Web-SIG.
--
Gustavo Narea.
Software Developer.
2degrees, Ltd. <http://www.2degreesnetwork.com/>.
Graham Dumpleton
On Jun 16, 7:32 pm, Gustavo Narea <gustavona...@2degreesnetwork.com>
wrote:
>
> On 16/06/10 01:33, Graham Dumpleton wrote:
>
> > Paste has its own problems. Paste server itself does process
> > environment setup that isn't done by other WSGI hosting mechanisms.
> > Specifically, it initialises Python logging in a way suited to Paste
> > based applications. Like with Django development server, you can
> > develop a Paste based application on Paste server which will then not
> > work on other WSGI hosting mechanisms.
>
> Paste Script will set up logging its way if and only if you explicitly
> tell it to; i.e., defining logging configuration in the Paste Deploy
> file or passing an argument to `paster'. I cannot think of another thing
> that it does at startup time which may affect deployment, but I may be
> wrong.
Thus you end up with a fiddle as documented in:
http://wiki.pylonshq.com/display/pylonscookbook/Logging+under+mod_wsgi
The point I am making is that these hosting system specific fiddles
should never need to be done by a user.
Graham
Gustavo Narea
> But that will not happen if you use a different hosting mechanism.
> Thus you end up with a fiddle as documented in:
>
> http://wiki.pylonshq.com/display/pylonscookbook/Logging+under+mod_wsgi
>
> The point I am making is that these hosting system specific fiddles
> should never need to be done by a user.
>
Right, I see your point.
Gustavo Narea
On Jun 14, 1:39 pm, Russell Keith-Magee <russ...@keith-magee.com>
wrote:
> the caveats I've given along the way). The next step is to show us
> actual code. This won't get applied to trunk as a single monolithic
> "fix WSGI" patch - it needs to be (at least) 4 patches, one for each
> feature that is being added, and each patch will need to be tested
> (and if appropriate, documented). Each of these features should also
> be logged as a ticket (if they aren't already).
>
> If you're looking to maximize the likelihood that this will land in
> trunk, the best approach will be to concentrate on one of the four
> issues (preferably starting with a small one), and see it through to
> completion; then move onto the next issue, and so on.
know where to put the tests.
The first patch (wsgiorg.routing_args implementation; #12075) would
involve adding a couple of lines to
django.core.handlers.base:BaseHandler.get_response() (to put the
arguments in the request) and also adding a couple of methods in
HttpRequest to access pythonically the so-called positional and named
arguments in the request, so we'd need tests for these two components
(i.e., the request and the handler).
I found an existing test suite for request objects (regressiontests/
requests/tests.py), but it has only ~6 doctests and I'm not sure if
you really want me to introduce these new tests as doctests, given
that the rest of the test suite seems to use regular unit tests.
As for tests for the handlers, I wasn't able to find any; /
regressiontests/servers/tests.py is the closest thing I found, but I'm
not sure that handler-related tests fit there. If I had to add a new
test suite, I thought it would be best to create it for WSGI-related
tests in tests/regressiontests/wsgi/ because it could be reused by my
future patches.
What do you think I should do?
Thanks in advance.
Russell Keith-Magee
You're correct that the handlers aren't extensively tested at the
moment (except indirectly through the test client). If one of the
upshots of your work is to improve this, then I'd call that a big win.
Excluding indirect testing effects, the tests for the WSGI interface
are in three places:
* regressiontests/servers
* regressiontests/builtin_server
* regressiontests/request
None of these are especially rigorous tests; in fact, the first two
should probably be merged into a single directory testing the
operation of the builtin server.
So - I'd say your suggestion of a new test directory is the right
approach. I'd be inclined to call that directory
"regressiontests/handlers", and put WSGI tests as a submodule of that
directory (after all, WSGI is just one of the server interfaces we
support).
Regarding doctests - one of the current GSoC projects is trying to
replace all the doctests with unittests. If you're adding new tests,
stick to unittests.
Yours,
Russ Magee %-)