Environment variables in WSGIScriptAlias? (ie, dynamic Django virtual hosts)

[Fotinakis]

I am using mod_wsgi in embedded mode and attempting to make an Apache
configuration that will not require any VirtualHost changes to deploy
new Django sites (similar to the concept of dynamically configured
mass virtual hosting http://httpd.apache.org/docs/2.2/vhosts/mass.html
). All that would be required is to create a new folder, such as /usr/
local/django/_SERVER_NAME_/ ... and then it will Just Work. At least,
that is the goal.

I've tried many variations of the following (kind of ugly)
configuration options:

# Causes the server name to be obtained from the client's Host
headers, instead of the ServerName directive
UseCanonicalName Off

RewriteEngine On
# ServerName derived from a Host: header may be in any case
RewriteMap lowercase int:tolower
RewriteRule . - [E=wsgiservername:${lowercase:%{SERVER_NAME}}]

WSGIScriptAlias / /usr/local/django/%{ENV:wsgiservername}/apache/
django.wsgi
<Directory /usr/local/django/%{ENV:wsgiservername}/apache>
Order deny,allow
Allow from all
</Directory>

The error log always says something like:
Target WSGI script not found or unable to stat: /usr/local/django/%
{ENV:wsgiservername}

I'm assuming then that WSGIScriptAlias doesn't parse environment
variables? Though, I'm probably terribly misunderstanding the order
that Apache follows when interpreting directives.

Is there a way to do this with a WSGIScriptAliasMatch? The problem I
see is that the requested path doesn't contain the server name, so a
regex can't pull it out. Perhaps a RewriteRule can mangle the path and
put the ServerName in? :/

WSGIScriptAliasMatch ^( **can the ServerName ever exist here?**)/ /usr/
local/django/$1/apache/django.wsgi

That seems to be the cleanest way to do it, if something like that
would work (is there potential here for a new directive, such as
WSGIVirtualScriptAlias?). Any ideas would be much appreciated ...
thank you!

[Graham Dumpleton]

Which of the methods in that document are you trying to use,
VirtualDocumentRoot from mod_vhost_alias or a mod_rewrite approach?

Can you post example for the actual virtual host directory
configuration you are using for document root/static files so can see
what it needs to work in conjunction with, or have you not even looked
at that aspect yet.

Graham

2009/1/5 Fotinakis <fotin...@gmail.com>:

[Graham Dumpleton]

I can probably provide an answer even though you haven't replied.

RewriteEngine On
RewriteMap tolower int:tolower

RewriteRule ^/(.*)
/usr/local/django/${tolower:%{SERVER_NAME}}/apache/django.wsgi/$1
RewriteRule . - [E=APPLICATION_GROUP:${tolower:%{SERVER_NAME}}]

WSGIProcessGroup %{GLOBAL}
WSGIApplicationGroup %{ENV:APPLICATION_GROUP}

<DirectoryMatch /usr/local/django/*/apache>
Order allow,deny
Allow from all
Options ExecCGI
AddHandler wsgi-script .wsgi
</DirectoryMatch>

Bit more work required if you want static media served by same server,
more so if you want separate static media directories for each Django
site. To handle the static media directory you could probably use
VirtualDocumentRoot to specify the location of document roots for all
sites. Then add an appropriate RewriteCond to above so RewriteRule
isn't done for URLs which map to static media directories.

Another RewriteCond you might want is to check for existence of the
WSGI script file for a particular host value and if not present force
a forbidden response. This way avoids problems with getting strange
errors if request arrives with host for which there isn't yet a
configured site.

So, play with that and post back what you come up with that works, or
if you have other issues.

Graham

2009/1/5 Graham Dumpleton <graham.d...@gmail.com>:

[Fotinakis]

Definitely on the right track, that's exactly what I'm looking for ...
one error though in the Apache logs:

File does not exist: /usr/local/django/project/apache/django.wsgi/

(including the trailing slash) I've confirmed that the file does
definitely exist. It seems to be trying to stat the full file and
thinking that it is a directory. For example:
$ stat /usr/local/django/project/apache/django.wsgi/
stat: cannot stat `/usr/local/django/asusu/apache/django.wsgi/': Not a
directory

A stat without the trailing slash works as expected. Bug? Or do we
just need a different syntax in the RewriteRule around /$1 ?

Thanks for the help and your dedication to your community. Every other
post I've seen recently has had your name on it, thanks for supporting
all of us so much. :)


On Jan 5, 3:18 am, "Graham Dumpleton" <graham.dumple...@gmail.com>
wrote:

> 2009/1/5 Graham Dumpleton <graham.dumple...@gmail.com>:

>
> > Which of the methods in that document are you trying to use,
> > VirtualDocumentRoot from mod_vhost_alias or a mod_rewrite approach?
>
> > Can you post example for the actual virtual host directory
> > configuration you are using for document root/static files so can see
> > what it needs to work in conjunction with, or have you not even looked
> > at that aspect yet.
>
> > Graham
>

> > 2009/1/5 Fotinakis <fotinaki...@gmail.com>:

>
> >> I am using mod_wsgi in embedded mode and attempting to make an Apache
> >> configuration that will not require any VirtualHost changes to deploy
> >> new Django sites (similar to the concept of dynamically configured

> >> mass virtual hostinghttp://httpd.apache.org/docs/2.2/vhosts/mass.html

[Fotinakis]

Got it! I think your regex was just a little off ... changed the * to
(.+) and everything seems to work with the other syntax as it is:

<DirectoryMatch /usr/local/django/(.+)/apache>

Order allow,deny
Allow from all
Options ExecCGI
AddHandler wsgi-script .wsgi
</DirectoryMatch>

Also works under an SSL configuration. I'm going to work through the
static file serving problem sometime, I'll post here when I figure out
a good way to do that. Thanks!

[Graham Dumpleton]

2009/1/6 Fotinakis <fotin...@gmail.com>:

>
> Got it! I think your regex was just a little off ... changed the * to
> (.+) and everything seems to work with the other syntax as it is:
>
> <DirectoryMatch /usr/local/django/(.+)/apache>
> Order allow,deny
> Allow from all
> Options ExecCGI
> AddHandler wsgi-script .wsgi
> </DirectoryMatch>

Yeah, for '*', should have been just Directory and not DirectoryMatch.

<Directory /usr/local/django/*/apache>

Order allow,deny
Allow from all
Options ExecCGI
AddHandler wsgi-script .wsgi
</Directory>

Graham

[Graham Dumpleton]

Bouncing this back onto list for general interest.

2009/3/15 ben.d...@googlemail.com <ben.d...@googlemail.com>:
>
> Been meaning to do this for ages now. Thanks Graham for all your help
> in other posts about this. Anyhow, following on from the thread as
> mentioned above, script below includes media handling, using
> VirtualDocumentRoot as mentioned above. The hard work was really
> already done above and this one has been modified to handle
> subdomains, using the subdomain name to identify the relevant
> folders.
>
> I only just got it work and have posted it in a slightly overexcited
> way so probably needs tidying and strengthening. But any obvservations
> most welcome (yup favicon not handled), thanks:
>
> RewriteEngine on
> RewriteMap tolower int:tolower
>
> VirtualDocumentRoot /home/sites/static/%1/
>
> <Directory /home/sites/static/* >

> Order deny,allow
> Allow from all
> </Directory>
>

> RewriteCond %{HTTP_HOST} ^([^.]+)\.domain\.co\.uk$ [NC]
> RewriteRule . - [E=subdomain:%1]
>
> RewriteCond %{REQUEST_URI} !^/media/*
> RewriteCond %{REQUEST_URL} !^favicon.ico
> RewriteRule ^/(.*) /home/sites/init/modwsgi/${tolower:%
> {ENV:subdomain}}/django.wsgi/$1

> RewriteRule . - [E=APPLICATION_GROUP:${tolower:%{SERVER_NAME}}]
>
> WSGIProcessGroup %{GLOBAL}

> WSGIApplicationGroup %{ENV:subdomain}
>
> <Directory /home/sites/init/modwsgi/* >

[ben.d...@googlemail.com]

Hi Graham,
One final question. The scripts above required a wsgi file per site.
I've rewritten it to direct all the sites through a single .wsgi file,
then used thin wrapper round the application to send it on its way to
the correct (django) settings based on environs settings. This seems
to work pretty well, even though it removes the 'touch' reloading
method.

But I'm concerned whether this script might be setting itself up for a
fall due to the way modwsgi is handling apache processes in embedded
mode since a new application group is being defined for each site
(which it needs to be since each site may be operating in different
virtualenv)? Isn't this what the daemon process mode is for? I'm
thinking of this line from the Configuration Guidelines:

'Where multiple applications, potentially owned by different users,
need to be run, 'daemon' mode of mod_wsgi should instead be used."

If that's true can the daemon process name be defined dynamically? Or
does this method necessarily require embedded mode?

Once again, many thanks for your advice on this.
Ben


On Mar 15, 12:20 am, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:

> Bouncing this back onto list for general interest.
>

> 2009/3/15 ben.dja...@googlemail.com <ben.dja...@googlemail.com>:

[Graham Dumpleton]

2009/3/23 ben.d...@googlemail.com <ben.d...@googlemail.com>:

>
> Hi Graham,
> One final question. The scripts above required a wsgi file per site.
> I've rewritten it to direct all the sites through a single .wsgi file,
> then used thin wrapper round the application to send it on its way to
> the correct (django) settings based on environs settings. This seems
> to work pretty well, even though it removes the 'touch' reloading
> method.
>
> But I'm concerned whether this script might be setting itself up for a
> fall due to the way modwsgi is handling apache processes in embedded
> mode since a new application group is being defined for each site
> (which it needs to be since each site may be operating in different
> virtualenv)? Isn't this what the daemon process mode is for? I'm
> thinking of this line from the Configuration Guidelines:
>
> 'Where multiple applications, potentially owned by different users,
> need to be run, 'daemon' mode of mod_wsgi should instead be used."

Using daemon mode would certainly be preferable, but as you already
know, one can do it with different application groups (sub
interpreters) within same process.

>  If that's true can the daemon process name be defined dynamically? Or
> does this method necessarily require embedded mode?

You can switch it around.

WSGIProcessGroup %{ENV:subdomain}
WSGIApplicationGroup %{GLOBAL}

The only problem with this is that you then need to statically define
a WSGIDaemonProcess directive for each sub domain, which is contrary
to what your original aims were from memory. That is, to not have to
change the Apache configuration to add a new site.

One of the main aims in mod_wsgi 4.0, see:

http://blog.dscpl.com.au/2009/03/future-roadmap-for-modwsgi.html

is to provide support for dynamic creation of daemon process groups
through defining only a single parameterisable template.

So, you will not get exactly what you wish until then.

The closest thing you will get for now is to define in advance
WSGIDaemonProcess directives for a pool of daemon process groups. This
would be given generic names. You would then use a RewriteMap to
dynamically assign stuff for a sub domain to one of the daemon process
groups in the pool of daemon process groups.

WSGIDaemonProcess site-1
WSGIDaemonProcess site-2
WSGIDaemonProcess site-3
WSGIDaemonProcess site-4
WSGIDaemonProcess site-5
WSGIDaemonProcess site-6
WSGIDaemonProcess site-7

RewriteEngine On
RewriteMap wsgiprocmap txt:/etc/httpd/wsgiprocmap.txt
RewriteRule . - [E=PROCESS_GROUP:${wsgiprocmap:%{ENV:subdomain}}]

WSGIProcessGroup %{ENV:PROCESS_GROUP}

Then when you add a new site, as well as creating directories in file
system, add an entry in the rewrite map, such as:

foobar.com site-1

Does mean you have to do some manual stuff in Apache configuration,
but you can at least preconfigure a whole bunch and then map then with
the write map file without having to restart Apache, as Apache will
automatically reread the rewrite map file when it changes.

Have I captured the intent of what you are trying to do?

Graham

[ben.d...@googlemail.com]

Hi Graham,
Thanks for your uber-fast response. I see exactly what you mean with
the pre-set processes and rewrite map - and it's a great work around
till the v4.0 release. Many thanks.

But are you saying daemon mode is only preferable due to the tendency
for apache to be misconfigured? A correctly configured apache set up
would be able to handle the embedded dynamic set up (and be better
performing too)?

Ben




On Mar 23, 10:49 am, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:
> 2009/3/23 ben.dja...@googlemail.com <ben.dja...@googlemail.com>:

[Graham Dumpleton]

2009/3/23 ben.d...@googlemail.com <ben.d...@googlemail.com>:

>
> Hi Graham,
> Thanks for your uber-fast response. I see exactly what you mean with
> the pre-set processes and rewrite map - and it's a great work around
> till the v4.0 release. Many thanks.
>
> But are you saying daemon mode is only preferable due to the tendency
> for apache to be misconfigured?

I am presuming you are talking about issues described in:

http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html

> A correctly configured apache set up
> would be able to handle the embedded dynamic set up (and be better
> performing too)?

The performance difference for a large Python web application is
probably not going to be that significantly different for the two
modes as the performance of Apache/mod_wsgi isn't going to be the
bottleneck.

The determining factor may there be amount of memory used in your
case. For embedded mode, where using separate application group for
each site, you have a copy of every site in every process. For daemon
mode, provided application is multithread safe, can probably get away
with one multithreaded daemon process. Thus your memory requirements
are much much less as only one copy of each site instead of a copy in
every child process.

My advice these days would be to use daemon mode unless you have a
very good reason not to and can show through testing of your specific
application that using embedded mode is warranted. If using embedded
mode, at same time, you would need to go to effort of configuring
Apache MPM settings appropriately, as well as perhaps setting up a
separate media server.

With daemon mode, you can at least leave Apache MPM settings set up
for static media and use same server. This isn't necessarily a good if
using embedded mode as static file serving and dynamic web application
have different requirements as to what MPM settings should be.

Graham

[ben.d...@googlemail.com]

Thanks a lot for your patience going through the ABC with this. I
can't give enough thanks for your constant and inspirational open-
source work and the help you give to stragglers like myself. I hope in
the not too distant future i can at least come back with enough beer
tokens to last a while, not just the words. Cheers and have a good
day, Ben




On Mar 23, 11:23 am, Graham Dumpleton <graham.dumple...@gmail.com>

wrote:
> 2009/3/23 ben.dja...@googlemail.com <ben.dja...@googlemail.com>:
>
>
>
> > Hi Graham,

> > Thanks for your uber-fast response. I see exactly what you mean with
> > the pre-set processes and rewrite map - and it's a great work around
> > till the v4.0 release. Many thanks.
>
> > But are you saying daemon mode is only preferable due to the tendency
> > for apache to be misconfigured?
>
> I am presuming you are talking about issues described in:
>

> http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usa...