wsgi not using correct python executable

[Scott Schaefer]

Hello,

First, system information:
Ubuntu 13.10
mod_wsgi 3.4 compiled against Python 3.3 (no problems)
Apache 2.4

Seems when I am running my application, mod_wsgi isn't using the correct interpreter. In my app.wsgi file, I'm outputting the value of sys.executable to a temp file. When mod_wsgi loads my app file, the path to the executable should be written out to the file so I can read it. The path being written out is /usr/bin/python3 instead of /home/webuser/projmercury/merc_dev/bin/python3

I'm setting both the python-home and python-path flags on my DaemonProcess. The following is from my site configuration in Apache:

WSGILazyInitialization On
WSGIRestrictEmbedded On

Listen 443
<VirtualHost *:443>
        ServerName localhost
        ServerAlias mercury
        SSLEngine on
        SSLCertificateFile /etc/ssl/mercury/mercury.crt
        SSLCertificateKeyFile /etc/ssl/mercury/mercury.key

        WSGIDaemonProcess mercury user=webuser group=webuser \
                python-home=/home/webuser/projmercury/merc_dev \
                python-path=/home/webuser/projmercury/merc_dev:/home/webuser/projmercury/merc_dev/lib/python3.3/site-packages \
                display-name=%{GROUP}
        WSGIProcessGroup mercury
        WSGIScriptAlias / /var/www/mercury.wsgi

...
</VirtualHost>


From the Apache error.log, I'm seeing the following log entries for mod_wsgi

[:debug] [pid 14496:tid 139645645408128] mod_wsgi.c(10488): mod_wsgi (pid=14496): Socket for 'mercury' is '/var/run/apache2/wsgi.14496.0.1.sock'.
[:info] [pid 14500:tid 139645645408128] mod_wsgi (pid=14500): Starting process 'mercury' with uid=1000, gid=1000 and threads=15.
[:info] [pid 14500:tid 139645645408128] mod_wsgi (pid=14500): Python home /home/webuser/projmercury/merc_dev.
[:info] [pid 14500:tid 139645645408128] mod_wsgi (pid=14500): Initializing Python.
[mpm_worker:notice] [pid 14496:tid 139645645408128] AH00292: Apache/2.4.6 (Ubuntu) mod_wsgi/3.4 Python/3.3.2+ OpenSSL/1.0.1e configured -- resuming normal operations
[:info] [pid 14500:tid 139645645408128] mod_wsgi (pid=14500): Attach interpreter ''.
[:info] [pid 14500:tid 139645645408128] mod_wsgi (pid=14500): Adding '(null)' to path.
[:info] [pid 14500:tid 139645645408128] mod_wsgi (pid=14500): Adding '/home/webuser/projmercury/merc_dev/lib/python3.3/site-packages' to path.
[:info] [pid 14500:tid 139645489436416] mod_wsgi (pid=14500): Create interpreter 'localhost|'.
[:info] [pid 14500:tid 139645489436416] mod_wsgi (pid=14500): Adding '(null)' to path.
[:info] [pid 14500:tid 139645489436416] mod_wsgi (pid=14500): Adding '/home/webuser/projmercury/merc_dev/lib/python3.3/site-packages' to path.
[:info] [pid 14500:tid 139645489436416] [client 10.113.229.150:2761] mod_wsgi (pid=14500, process='mercury', application='localhost|'): Loading WSGI script '/var/www/mercury.wsgi'.
[:info] [pid 14496:tid 139645645408128] mod_wsgi (pid=14500): Process 'mercury' has died, restarting.
[:info] [pid 14579:tid 139645645408128] mod_wsgi (pid=14579): Starting process 'mercury' with uid=1000, gid=1000 and threads=15.
[:info] [pid 14579:tid 139645645408128] mod_wsgi (pid=14579): Python home /home/webuser/projmercury/merc_dev.
[:info] [pid 14579:tid 139645645408128] mod_wsgi (pid=14579): Initializing Python.
[:info] [pid 14579:tid 139645645408128] mod_wsgi (pid=14579): Attach interpreter ''.
[:info] [pid 14579:tid 139645645408128] mod_wsgi (pid=14579): Adding '(null)' to path.
[:info] [pid 14579:tid 139645645408128] mod_wsgi (pid=14579): Adding '/home/webuser/projmercury/merc_dev/lib/python3.3/site-packages' to path.

As you can see, the Python home log entries reflect the correct location, but the file output shows the wrong interpreter.

My application is running fine but I'm not sure why the wrong interpreter is being used. I've tried to upgrade to Ubuntu 14.04 and it breaks my app because python3.4 becomes the default python3 installation (thankfully I can rollback the VM snapshot to 13.10). Pointing to the interpreter in the virtual environment via python-home should fix this issue with Ubuntu 14.04 if it would work correctly. Is there something I'm forgetting to configure?

In case you wonder, here are the lines from my mercury.wsgi file:
import sys
with open('/tmp/merc_interpreter','w') as f:
    f.write(sys.executable+'\n\n')

Thank you for any assistance.

[Graham Dumpleton]

From memory, under mod_wsgi sys.executable may not mean much and may not reflect what Python installation is used. When using mod_wsgi the actual Python executable is never used/executed because it embeds Python directly into the Apache processes by linking the Python library.

The more important indication of what Python installation is used is what Python shared library is being found and what sys.prefix is set to.

Work through the various checks in:

http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation

and see if they all check out.

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.

[Graham Dumpleton]

On 11/05/2014, at 10:15 AM, Scott Schaefer <sjsch...@gmail.com> wrote:

        WSGIDaemonProcess mercury user=webuser group=webuser \
                python-home=/home/webuser/projmercury/merc_dev \
                python-path=/home/webuser/projmercury/merc_dev:/home/webuser/projmercury/merc_dev/lib/python3.3/site-packages \
                display-name=%{GROUP}

And you don't need to list /home/webuser/projmercury/merc_dev/lib/python3.3/site-packages in python-path. If all is okay you having set python-home to the value of the sys.prefix for that Python installation/virtual environment should be enough for the site-packages from that installation to be used.

Graham

[Scott Schaefer]

Thank you for the explanation. I'll go ahead and try those other few things to verify the configuration.

In the mean time, suppose I upgrade to 14.04, which replaces Python 3.3 with Python 3.4. The shared library that mod_wsgi was linked against may then be gone (Python 3.3 lib). Will this cause it to break even though I'm pointing it to a 3.3 virtualenv?

[Graham Dumpleton]

Removing the Python installation which the Python virtual environment and mod_wsgi were dependent on would generally break both.

[Scott Schaefer]

True, but in my case, the virtual environment is completely stand-alone. I had virtualenv copy the python files instead of symlink to them. The distro upgrade can overwrite python 3.3 and my virtualenv will remain intact and functional. So, that being the case, I assume since mod_wsgi was linked against the shared library, it would then cease to function. I'd have to link it somewhere else which, from reading the docs, is bad practice.

[Graham Dumpleton]

I am not aware of the option to make virtualenv make copies of files rather than symlink then, but I would doubt whether it would copy the Python shared library because it needs to be in a location where the operating system can find it. Even if the virtualenv created a physical copy of 'python', then I can't see how removing the system Python version it was dependent on wouldn't break the need of the 'python' executable to find the shared library.

Either way, mod_wsgi itself is compiled for one specific major/minor version of Python. You cannot remove the version it is dependent upon and try and make it use a different version without recompiling mod_wsgi.

Graham