WSGIPythonHome/WSGIPythonPath and Python initialisation.
Graham Dumpleton
>
>
> > If I compile mod_wsgi --with-python /home/asterisk/python2.5/bin/
> > python all works ok.
> > If I set WSGIPythonHome to non-existent directory I get same error "No
> > module named os".
> > So it seems that mod_wsgi loads python binary from path found at
> > compilation time. So in my case it picks up python2.4 and when set
> > WSGIPythonHome it tries to load os module from 2.5!
>
> You _have_ to build mod_wsgi using the python version you're using for
> your applications. Using a different version and setting the home
> directory to some other version can't work.
As mentioned in 'Multiple Python Versions' of:
http://code.google.com/p/modwsgi/wiki/InstallationIssues
The are also comments in:
http://modwsgi.googlecode.com/svn/tags/mod_wsgi-1.2/README
about --with-python, PATH and WSGIPythonExecutable.
I accept that these haven't been updated yet to correspond to
WSGIPythonExecutable being removed.
In short though, mod_wsgi can only be used with the major/minor
version of Python that it was compiled for. Slight differences in
patch level revision are okay provided your Python installation
provided a shared library and mod_wsgi is using it.
That said, the WSGIPythonHome directive can be used for two purposes.
The first is where you have compiled mod_wsgi against the correct
version of Python, but at run time that version isn't what is in the
PATH of Apache when it is run.
Thus, if you had:
/usr/bin/python2.4
and:
/usr/local/bin/python2.5
and did use:
--with-python=/usr/local/bin/python2.5
it is still necessary that /usr/local/bin be first in PATH of the user
that Apache is started as.
This is because the default strategy of Python internals when it is
being initialised is to look in PATH for 'python' executable. It will
then use this as a landmark to work out where the Python lib directory
is.
Thus, if the 'python' executable in the PATH is /usr/bin/python, it
will then look two directories up and expect the Python lib directory
to be found under that directory as /usr/lib/python2.5.
Obviously this isn't where you have installed Python 2.5, so it is
necessary to override where the Python internals initialisation code
looks.
In mod_wsgi 1.0, provided you were using a UNIX (not MacOS X) system,
you could override the Python internal initialisation so that instead
of looking in PATH for 'python', it would use the location specified
using WSGIPythonExecutable as the landmark and go from there. Thus:
WSGIPythonExecutable /usr/local/bin/python2.5
would have made things work.
Because this doesn't work on MacOS X or Windows though, thus have
preference for not providing it.
Instead, use an alternate mechanism in Python internal initialisation
code for overriding where to find the Python lib directory. If one was
using command line Python executable, this would be done by setting
the PYTHONHOME environment variable. In mod_wsgi, doesn't use
environment variable though but use a directive which hooks into
Python internals and calls Py_SetPythonHome() function instead.
Anyway, the end result is that it will not even go looking in PATH for
'python' executable and will instead blindly accept the supplied
location as the directory prefix for where Python is installed. This
location should equate to the directory two above where the 'python'
executable was. Thus:
WSGIPythonHome /usr/local
It will then as a result expect the Python lib directory to be
/usr/local/lib/python2.5 and everything should work, including on
MacOS X and Windows (I hope).
All the above has to be done outside of VirtualHost at global scope as
Python is only initialised once for the whole of Apache and so it
can't be different for different VirtualHost containers. See below for
solution to this though.
If you now have additional Python module directories you want
searched, if using embedded mode of mod_wsgi, you would use:
WSGIPythonPath /home/user/lib/python
In mod_wsgi 1.X, this would also have be inherited by daemon mode
processes as well. This though has changed in mod_wsgi 2.0c2 and if
you want to specify additional module directories for daemon
processes, you must use 'python-path' option to WSGIDaemonProcess
directive.
WSGIDaemonProcess example python-path=/home/user/lib/python
This change was made so that different paths could be specified for
different daemon process groups and also the embedded processes
themselves. if you were using daemon mode, this may be why your
modules still couldn't be found.
Now, even if the 'python' executable found in PATH is the correct
version, one may still want to override this. This would be done when
one is using one of the Python virtual environment mechanisms such a
virtualenv.
In the simplest case, this may just be a matter of pointing at the
virtual environment. So, if you used:
python virtualenv.py /usr/local/pythonenv/ENV1
you would use:
WSGIPythonHome /usr/local/pythonenv/ENV1
This would have the result that mod_wsgi in either embedded or daemon
mode would use that distinct Python virtual environment.
If you have more complicated arrangements, you might also want to use
WSGIPythonPath directive in combination with this.
What you can do here is to use WSGIPythonHome as a base Python
installation. In this case you would use virtualenv.py to create a
virtual environment which ignores the original Python installations
site-packages directory.
python virtualenv.py --no-site-packages /usr/local/pythonenv/VIRGIN
In other words it will be like you hadn't installed anything at all on
top of a fresh Python installation.
WSGIPythonHome /usr/local/pythonenv/VIRGIN
If now you want all applications running in embedded mode to use
installed modules from a particular virtual environment, you would
use:
WSGIPythonPath /usr/local/pythonenv/ENV1/lib/python2.5/site-packages
If applications running in a specific daemon process group needed a
different set of installed modules, use:
WSGIDaemonProcess django
python-path=/usr/local/pythonenv/ENV2/lib/python2.5/site-packages
WSGIDaemonProcess turbogears
python-path=/usr/local/pythonenv/ENV3/lib/python2.5/site-packages
Note that here am only adding the 'site-packages' directory from the
subsequent virtual environments. This is because the base Python
installed is picked up from the VIRGIN Python installation. The
subsequent virtual environments would also have been created with
--no-site-packages as well and only populated with the packages you
actually wanted.
Both WSGIPythonPath and 'python-path' option can list multiple
directories, just use ':' separator or UNIX and ';' on Windows.
Now, a big difference in mod_wsgi 2.0c2 that allows this to work, is
that in 2.0c2 setting WSGIPythonPath or 'python-path' isn't just the
same as setting PYTHONPATH environment variable. Instead, it is
equivalent to calling site.addsitedir() from within a Python script.
Doing this has the affect of not only adding the directory to
sys.path, but also looks in the directory and interprets all the .pth
files and adds to sys.path and directories specified in those as well.
As a result, any Python egg stuff will work.
End result now is that it is relatively easy to run up
Apache/Python/mod_wsgi so that it doesn't use the site wide
site-packages and embedded mode applications or daemon mode
applications can all use their own sets of packages and any risk of
site wide packages being incorrectly used or interfering with a
specific applications versions is eliminated.
This all would be especially important in a shared hosting environment
where each user was given their own daemon process to use for their
applications as they could totally control themselves what versions of
packages they want to use.
Note that at no time is the Python path specific to a VirtualHost. One
can only set it for all embedded mode applications, or all
applications in specific daemon processes. This is actually more
flexible than it being tied to VirtualHost as applications under the
same VirtualHost can use different sets of packages. A lot of the time
a VirtualHost would be delegated as a whole to a daemon process group,
so setting 'python-path' for that WSGIDaemonProcess has the same
effect of doing it for the VirtualHost.
Also note that you can still set sys.path from inside the WSGI
application script, and so it can be further customised for
application per interpreter. In this case you do have to worry about
issues with conflicts between C extension module versions if running
multiple applications in the same process as already described in
mod_wsgi wiki documentation, as well as some of my older blog entries.
Anyway, hope that explains things a bit better and it probably even
goes into a bit more detail than what I was going to blog about. :-)
If things still don't work as you expect, then post the Apache
configuration you are using for mod_wsgi so I can see how you are
using daemon processes or not.
Graham