Proper Use of the WSGIPythonHome Directive?

[Dylan Combs]

So, I see that issues similar to this have been raised in the past, but the issues resolved there do not appear to be the same as the issue as I am seeing, so hopefully someone can let me know what I'm doing wrong here.  I have developed a workaround using what appears to be undocumented behavior from the WSGI Module, but I am interested in understanding why seems to be necessary.

System Configuration

• RHEL 6.8
• httpd 2.2
• mod_wsgi 4.5.6 compiled with:
• anaconda 4.1.1 (Python 3.5)

I’m using Red Hat Enterprise Linux 6.8 (fully up to date) with the standard httpd package delivered and supported for that distribution (so httpd 2.2).  I compiled mod_wsgi from source, so I’m using the latest and greatest (4.5.6) and I performed that compilation using Python 3.5 through Continuum.io’s Anaconda software as follows:

 $ wget https://github.com/GrahamDumpleton/mod_wsgi/releases/tag/4.5.6
 $ tar xf 4.5.6.tar && cd 4.5.6
 $ ./configure --with-python=/opt/anaconda3/bin/python3.5
 $ LD_RUN_PATH=/opt/anaconda3/lib make
 $ sudo make install

The compilation succeeds and the module appears to be properly linked to the appropriate Python library:

# ldd /etc/httpd/modules/mod_wsgi.so
        linux-vdso.so.1 =>  (0x00007fffff9f7000)
        libpython3.5m.so.1.0 => /opt/anaconda3/lib/libpython3.5m.so.1.0 (0x00007f6b55b00000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6b558d7000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f6b556d3000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007f6b554d0000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f6b552c7000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f6b55043000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f6b54caf000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f6b56228000)

Initial Trouble

When I tried to start the httpd daemon using the init script and Upstart service command provided by RHEL, the attempt failed and I found in /var/log/httpd/error_log (after increasing too l

evel verbosity):

Could not find platform independent libraries <prefix> 
Could not find platform dependent libraries <exec_prefix> 
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>] 
[info] mod_wsgi (pid=3786): Python home /opt/anaconda3. 
[info] mod_wsgi (pid=3786): Initializing Python. 
[info] mod_wsgi (pid=3787): Starting process 'sampleapp' with uid=48, gid=48 and threads=15. 
[info] mod_wsgi (pid=3787): Python home /opt/anaconda3. 
[info] mod_wsgi (pid=3787): Initializing Python. 
Fatal Python error: Py_Initialize: Unable to get the locale encoding 
ImportError: No module named 'encodings'

And so it begins…

My Quest in Directing Apache to the Anaconda Python Resources

Though it appears from the log and from the documentation on mod_wsgi that I was correctly establishing the configuration directive necessary to point httpd to the correct location for Python resources, the error seems pretty clearly indicative of a failure in that respect.  After using every damn WSGI directive (WSGIPythonHome, WSGIPythonPath, WSGIDaemonProcess with the python-home or python-path options…) which seemed plausibly related, all to no avail, I changed tactics.

I ruled out permission and SELinux issues through a variety of standard means.  I tried running Apache as root without SELinux enabled just to make sure this couldn’t possibly be the problem.  Just to be 100% sure and confirm for myself my own rudimentary assessment capabilities, I even `chmod -R 777`’d /opt/anaconda3 (not before taking a backup of it which I then restored after the test, of course).  It was clear that httpd could access the files and do whatever it pleased with them; for some reason, despite seeming to acknowledge clear direction, even, it just refused to find them.

So I decided I’d just have to stack trace httpd and see what, exactly, is going on.  In doing so, I noticed that the httpd daemon would seem to successfully locate the /opt/anaconda3 directory (as expected).  In performing the initial module load, it would successfully open the necessary Python libraries.

open("/opt/anaconda3/lib/libpython3.5m.so.1.0", O_RDONLY) = 5

w00t.  No issues so far.

However, despite the fact that mod_wsgi logs an info event indicating that it has properly recognized the WSGIPythonHome directive I am providing (which is accurate; it is equivalent with the sys.prefix variable, as the documentation indicates it should be), the startup process invariably devolves into searching what appears to be a default series of locations for the Python resources (/usr, for example).

From the stack trace (my user name converted to “myusername”):

stat("/sbin/python3", 0x7fffa0fba600)   = -1 ENOENT (No such file or directory) 
stat("/bin/python3", 0x7fffa0fba600)    = -1 ENOENT (No such file or directory) 
stat("/usr/sbin/python3", 0x7fffa0fba600) = -1 ENOENT (No such file or directory) 
stat("/usr/bin/python3", 0x7fffa0fba600) = -1 ENOENT (No such file or directory) 
readlink("", 0x7fffa0fa5520, 4096)      = -1 ENOENT (No such file or directory) 
open("pyvenv.cfg", O_RDONLY)            = -1 ENOENT (No such file or directory) 
open("pyvenv.cfg", O_RDONLY)            = -1 ENOENT (No such file or directory) 
stat("Modules/Setup", 0x7fffa0fa64a0)   = -1 ENOENT (No such file or directory) 
getcwd("/home/myusername", 4096)         = 16 
stat("/home/myusername/lib/python3.5/os.py", 0x7fffa0fa64a0) = -1 ENOENT (No such file or directory) 
stat("/home/myusername/lib/python3.5/os.pyc", 0x7fffa0fa63d0) = -1 ENOENT (No such file or directory) 
stat("/home/myusername/lib/python3.5/os.py", 0x7fffa0fa64a0) = -1 ENOENT (No such file or directory) 
stat("/home/myusername/lib/python3.5/os.pyc", 0x7fffa0fa63d0) = -1 ENOENT (No such file or directory) 
stat("/home/lib/python3.5/os.py", 0x7fffa0fa64a0) = -1 ENOENT (No such file or directory) 
stat("/home/lib/python3.5/os.pyc", 0x7fffa0fa63d0) = -1 ENOENT (No such file or directory) 
stat("/home/ilan/minonda/envs/_build/lib/python3.5/os.py", 0x7fffa0fa63d0) = -1 ENOENT (No such file or directory) 
stat("/home/ilan/minonda/envs/_build/lib/python3.5/os.pyc", 0x7fffa0fa6320) = -1 ENOENT (No such file or directory) 
write(2, "Could not find platform independ"..., 55) = 55 
stat("pybuilddir.txt", 0x7fffa0fa1440)  = -1 ENOENT (No such file or directory) 
getcwd("/home/myusername", 4096)         = 16 
stat("/home/myusername/lib/python3.5/lib-dynload", 0x7fffa0fa5520) = -1 ENOENT (No such file or directory) 
stat("/home/myusername/lib/python3.5/lib-dynload", 0x7fffa0fa5520) = -1 ENOENT (No such file or directory) 
stat("/home/lib/python3.5/lib-dynload", 0x7fffa0fa5520) = -1 ENOENT (No such file or directory) 
stat("/home/ilan/minonda/envs/_build/lib/python3.5/lib-dynload", 0x7fffa0fa5520) = -1 ENOENT (No such file or directory) 
write(2, "Could not find platform dependen"..., 58) = 58 
write(2, "Consider setting $PYTHONHOME to "..., 57) = 57

Despite that info event in /var/log/httpd/error_log showing me that mod WSGI seems to properly recognize the information I provided with the WSGIPythonHome directive, I never see any attempt from the software to look to /opt/anaconda3 for the Python resources.  If you’ll note, the search becomes so desperate, in fact, that it includes some weird searches to subdirectories and resources in the non-existant /home/ilan directory, which looks like it must be somehow inadvertently included information on behalf of the designer of Anaconda, as I found a forum post signed “ilan” from him, it seems.  Eventually, as you can see, the search process fails and httpd proclaims its inability to locate the resources.

I noticed, however, that the search pattern being run by httpd/mod_wsgi looks like it includes an attempt to locate Python resources using the current working directory.  So… I changed my current working directory to /opt/anaconda3 and executed httpd manually from there and, voila, if I start the httpd process from within the Python Home location, it works! GLORY! IT WORKS!

But… I can’t exactly change the httpd default working directory to solve this problem; that’s crazy.  However, you also might have noticed in my stack trace that one of the steps Apache was conducting in searching for the Python resources consisted of attempts to read a pyvenv.cfg file in the current working directory.

I managed to look up the simple syntax for such a file, and it is something like this:

home = /opt/anaconda3 
include-system-site-packages = true 
version = 3.5.2

In fact, it is exactly like that in my case. If I place it in the root of the file tree (/pyvenv.cfg). That actually fixes the problem and the application runs as expected without any errors.

The mod_wsgi configuration details I settled on are as follows:

From /etc/httpd/conf.d/awsgi.conf:

LoadModule wsgi_module modules/mod_wsgi.so
WSGISocketPrefix /var/run/wsgi
WSGIApplicationGroup %{GLOBAL}
WSGIPythonHome /opt/anaconda3

And subsequently from /etc/httpd/conf.d/ssl.conf:

<VirtualHost _default_:443>
WSGIScriptAlias / /var/www/wsgi-scripts/scriptname.wsgi
WSGIDaemonProcess sampleapp user=apache group=apache threads=5
WSGIProcessGroup sampleapp
...
</VirtualHost>

Pretty basic.  Anyone have any ideas where I went wrong?  I am, of course, glad to attempt any suggestions with the current system and report back on its behavior. 

[Graham Dumpleton]

The response to this is complicated. The short answer is that Anaconda Python installations have broken the ability to use their Python distribution in an embedded system such as mod_wsgi. I don’t know that I am fully happy with the workaround I would have to use. If I can’t find a workaround, then only Anaconda can fix the issue, since it appears it is they who have broken the behaviour of the Python C API which is used when doing embedding. On that point they should be fixing it regardless of whether I can find a workaround.

As first step in understanding what is going on, lets first look at running command line Python. When you run command line Python, it will go through a very convoluted algorithm to work out where the Python run time has been installed.

From memory it first looks at the path of the ‘python’ execute run and from that goes up a directory and then looks for a ‘lib/pythonX.Y’ directory.

If the full path to the ‘python’ executable is not known due to how it was run, it will search through $PATH looking for a ‘python’ executable with same name as what was run. From that it again goes looking for the ‘lib/pythonX.Y’ directory. I think it also may even fallback to looking for generic ‘python’ executable name if program name is different.

Both of these checks can also be overridden by using the PYTHONHOME environment variable. This would be set to the same directory as ‘sys.prefix’ would give when dumped from the Python interpreter being used.

It is very rare situation with the command line Python that one of the first two checks would fail and PYTHONHOME would be required.

Now looking at a system where the Python interpreter is embedded in something else, such as is the case with mod_wsgi.

With embedded systems, the ‘python’ executable is not run. Instead you are running a custom application which is linked to the Python runtime library. To run the Python interpreter, the custom application would run Py_Initialize() C API function from the Python runtime library. This will in turn trigger the algorithm to find where the Python installation is located.

When you talk about an embedded system, this is where problems start to occur.

First up, the name of the application is not going to be ‘python’, nor is the path for the application going to be in the same ‘bin’ directory as where Python is located. The mod_wsgi case is even more complicated because the Python library isn’t even linked to the application (Apache httpd) being run. Instead the Python library is linked to an Apache module which is dynamically loaded into Apache only if required.

As a result, the first checks will fail as search relative to where ‘httpd' is located will not find a directory ‘lib/python3.5’. Even falling back to looking for ‘python’ executable will fail, as although that will be found in ‘/usr/bin’, that is the system Python which is older version, so again can’t find ‘lib/python3.5’.

The only fallback therefore when Python is installed in a non standard location is to tell Python where the Python installation is located.

For command line Python you would do that by setting PYTHONHOME environment variable. Setting process environment variables when using Apache is tricky though, also setting it carries some risk.

The risk here is that in setting PYTHONHOME that will be inherited by all child processes as well. This means that if somewhere down in your application you wanted to run a Python script which needed to use system Python, overriding PYTHONHOME environment variable could cause that script to fail as it will try and find the Python runtime in the wrong location.

The solution for embedded systems is a C API call Py_SetPythonHome(). Rather than set the environment variable, the custom application embedded Python can call this to set the location of the Python installation. This is what the WSGIPythonHome directive is triggering be done. That is, when WSGIPythonHome is set, it will call Py_SetPythonHome() with that value before Py_Initialize() is called so that the Python runtime can be found.

And this is now where our problem is. That is, Anaconda Python has somehow broken this C API and calling Py_SetPythonHome() is not overriding where it thinks the Python runtime is located as it should.

That said, they aren’t alone in breaking this. Python 3.3+ (???) has also broken Py_SetPythonHome() in the very specific case of where you use the builtin Python pyvenv support. That is, you can’t use Py_SetPythonHome() to refer to a Python virtual environment when you should be able to. This issue has still not been fixed in Python 3 (http://bugs.python.org/issue22213).

The workaround that mod_wsgi uses when it detects that Python 3 type pyvenv virtual environments are being used is to use Py_SetProgramName() instead. This is hooking into the first check where it tried to find the runtime directory relative to program name.

Problem is that for Anaconda you can’t use Py_SetProgramName() either, as they have broken that in some way as well and it doesn’t work either.

As far as I can see, using the proper C API for the Python library you cannot make Anaconda work properly for embedded systems in the case where the ‘python’ executable for Anaconda is not in the $PATH for the running program, as it is for Apache, or where PYTHONHOME environment is not being set.

If you understand what I am saying here, then you may have picked up that those two things can be used as workarounds. That is, if you can modify the system startup scripts for Apache such that the Anaconda bin directory is in $PATH, then it may work.

What will work for sure is setting PYTHONHOME environment variable. So you could modify system startup scripts to set that for Apache, but then you run that risk I mentioned before of causing sub processes that require system Python being screwed up.

You might be thinking, if PYTHONHOME works, why I can’t set that inside of mod_wsgi just before Py_Initialize() is called and then unset it after. That would work, but it isn’t as simple as just unsetting it as process environment variable. This is because Python will have already made a copy of it into os.environ of the main interpreter is creates. Any Python code which runs sub process where os.environ is inherited will still get it even if was unset a C level using C unsetenv().

So maybe it could be made to work by using the workaround of setting PYTHONHOME, but does mean that as well as unsetting the environment variable at C level, also need to remove it from os.environ in the main interpreter context as well. I had baulked at such a workaround in the past, but since Anaconda is still causing problems and not even sure that CPython might not further break C APIs for specifying this stuff, may be worth the trouble of seeing if can make it work that way.

As to why pyvenv.cfg in the root directory appears to work, that is likely going to be the case as the algorithm to work out where the Python installation is, has from memory some special checks as part of it that allows you to run a compiled Python interpreter out of the source directory where you install it. So it has some checks which look relative to the current directory as well. That pyvenv.cfg check may somehow be related to those checks. This would only work where you can place it in the current working directory of the application when run. The default for Apache for that is the root directory, but not sure that will always work. It will not work with mod_wsgi-express as a workaround and may also have problems with mod_wsgi daemon mode if the home directory option is used.

Anyway, that is the long and complicated response. I will have a go at embedding the PYTHONHOME environment variable workaround in mod_wsgi itself and see if that works. It will need to be tested against a range of different Python versions, as well as when using virtualenv and pyvenv though. 

Additional comments below.

That is probably the default directory and prefix for this environment anyway and so WSGISocketPrefix likely wouldn’t have been required.

WSGIApplicationGroup %{GLOBAL}
WSGIPythonHome /opt/anaconda3

And subsequently from /etc/httpd/conf.d/ssl.conf:

<VirtualHost _default_:443>
WSGIScriptAlias / /var/www/wsgi-scripts/scriptname.wsgi
WSGIDaemonProcess sampleapp user=apache group=apache threads=5

The user/group arguments wouldn’t have been needed anyway, as that is what mod_wsgi would default to anyway for this environment. That is, it inherits whatever is used with the User and Group directives in the Apache configuration.

WSGIProcessGroup sampleapp
...
</VirtualHost>

Pretty basic.  Anyone have any ideas where I went wrong?  I am, of course, glad to attempt any suggestions with the current system and report back on its behavior. 

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at https://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.

[Graham Dumpleton]

The workaround of setting PYTHONHOME inside of mod_wsgi will not work as Anaconda looks to be doing something even stranger than expected. I think what they are doing is a bit dubious and may break expectations of how the C API is meant to work, but it is actually in a different area to what I expected. That Py_SetPythonHome() and Py_SetProgramName() didn’t work was actually a side effect of an earlier problem that had spotted earlier.

See log and comments below.

# /opt/anaconda3/bin/mod_wsgi-express start-server --user apache --group root —log-to-terminal —log-level=debug

Server URL         : http://localhost:8000/

Server Root        : /tmp/mod_wsgi-localhost:8000:0

Server Conf        : /tmp/mod_wsgi-localhost:8000:0/httpd.conf

Error Log File     : /dev/stderr (debug)

Request Capacity   : 5 (1 process * 5 threads)

Request Timeout    : 60 (seconds)

Startup Timeout    : 15 (seconds)

Queue Backlog      : 100 (connections)

Queue Timeout      : 45 (seconds)

Server Capacity    : 20 (event/worker), 20 (prefork)

Server Backlog     : 500 (connections)

Locale Setting     : en_US.UTF-8

Could not find platform independent libraries <prefix>

Could not find platform dependent libraries <exec_prefix>

Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]

^^^^ This is occurring some time before the Python interpreter is being initialised.

What the error means is that Python was being partially initialised prior to Py_Intialize() being called. I am not sure what it is doing, but possibly trying to make use of Python codecs.

Turns out the problematic C API call is Py_GetVersion().

This C API function is meant to return version information, similar to what you get with sys.version.

In all CPython versions from the PSF, this function has been safe to call before Py_Initialize() as it just used very basic formatting functions to create the result. In the Anaconda Python they must have modified this function to use much higher level formatting functions which require that the codecs for the Python unicode system be initialised. These have failed to initialise though as Python hadn’t yet been initialised yet. This caused a cascade of failures as an erroneous state had been retained. Unfortunately as far as I know any modified Anaconda Python code isn’t available so isn’t possible to work out whether they have modified Py_GetVersion() or functions it in turn calls, as presumed and what they have done to cause the problem.

[Wed Sep 07 04:33:01 2016] [debug] src/server/mod_wsgi.c(8130): mod_wsgi (pid=11888): Socket for 'localhost:8000' is '/tmp/mod_wsgi-localhost:8000:0/wsgi.11888.0.1.sock'.

[Wed Sep 07 04:33:01 2016] [debug] src/server/mod_wsgi.c(8188): mod_wsgi (pid=11888): Listen backlog for socket '/tmp/mod_wsgi-localhost:8000:0/wsgi.11888.0.1.sock' is '100'.

[Wed Sep 07 04:33:01 2016] [info] mod_wsgi (pid=11890): Starting process 'localhost:8000' with uid=48, gid=0 and threads=5.

[Wed Sep 07 04:33:01 2016] [notice] Apache/2.2.15 (Unix) mod_wsgi/4.5.7 Python/3.5.2 configured -- resuming normal operations

[Wed Sep 07 04:33:01 2016] [info] Server built: Jul 18 2016 15:24:00

[Wed Sep 07 04:33:01 2016] [debug] prefork.c(1028): AcceptMutex: sysvsem (default: sysvsem)

[Wed Sep 07 04:33:01 2016] [debug] src/server/mod_wsgi.c(9782): mod_wsgi (pid=11890): Setting lang to en_US.UTF-8 for daemon process group localhost:8000.

[Wed Sep 07 04:33:01 2016] [debug] src/server/mod_wsgi.c(9795): mod_wsgi (pid=11890): Setting locale to en_US.UTF-8 for daemon process group localhost:8000.

Fatal Python error: Py_Initialize: Unable to get the locale encoding

ImportError: No module named 'encodings'

Current thread 0x00007f9bf14357e0 (most recent call first):

^C[Wed Sep 07 04:33:02 2016] [info] mod_wsgi (pid=11890): Process 'localhost:8000' has died but server is being stopped, deregister it.

[Wed Sep 07 04:33:02 2016] [info] mod_wsgi (pid=11890): Process 'localhost:8000' has been deregistered and will no longer be monitored.

[Wed Sep 07 04:33:02 2016] [info] removed PID file /tmp/mod_wsgi-localhost:8000:0/httpd.pid (pid=11888)

[Wed Sep 07 04:33:02 2016] [notice] caught SIGTERM, shutting down 

Anyway, can you please try installing and using:

     https://github.com/GrahamDumpleton/mod_wsgi/archive/develop.tar.gz

and see if that works for you. Let me know how you go.

It works for mod_wsgi-express now. I haven’t tried it with a manual installation.

Under mod_wsgi-express the WSGIPythonHome directive is always set. I am presuming you will still need to set that manually.

I have to now work out whether Anaconda is actually wrong, or whether technically Py_GetVersion() is not allowed to be called prior to Py_Initialize().

Right now the code is just commenting out the use of Py_GetVersion() that causes this. All this results in is loss of information in server string about Python version used, plus no warning about when using mismatched Python versions. That is, compile time vs run time. I can move logging of the Python version to where each process is initialised as INFO level logging. The mismatch check I can’t easily move as Py_Initialize() would likely block up if different minor version of Python, so can’t do it after Py_Initialize().

Graham

[Dylan Combs]

Graham -

That development module did, in fact, fix the problem.  Thanks for the mega-quick response; I apologize for my delay.

Did you ever figure out whether this is a technical fault with mod_wsgi or Anaconda Python?

-Dylan

[Graham Dumpleton]

It is a grey area.

The Python documentation doesn’t say you aren’t allowed to call Py_GetVersion() before Py_Initialize() is called. It has always been possible to call it before Python was initialised in CPython for as long as I can remember. It is only Anaconda Python which seems to have changed the behaviour of Py_GetVersion() for some reason. Unfortunately whatever changes they have made are not public so one will never know what they have done.

In these situations it is generally fruitless trying to get the Python re-distribution to change anything, so don’t have a choice but to change mod_wsgi to accomodate what change they have made.

Graham