What do you expect sys.executable to return when using mod_wsgi and virtual python environments?

[Molly Foley]

I'm having issues with a python package we have installed and I'm trying to determine if it's a problem with our mod_wsgi install or the python package itself. I am suspecting our mod_wsgi setup because the application works fine when debugging it locally with the exact same directory acting as the python environment that mod_wsgi is supposed to be using as python-home in our Apache config.

Part of my debugging process is having the WSGI application (Flask app) log a bunch of info:

• User running the app
• $PATH
• sys.path
• sys.prefix
• sys.executable
• sys.base_prefix

Output:

DEBUG [11/15/2024 06:37:38 PM] Running as user: hydro

DEBUG [11/15/2024 06:37:38 PM] $PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin

DEBUG [11/15/2024 06:37:38 PM] sys.path: ['/var/www/vhosts/climate-hydro-analytics-platform-v2/api', '/var/www/vhosts/climate-hydro-analytics-platform-v2', '/opt/miniforge3/envs/chap/lib/python312.zip', '/opt/miniforge3/envs/chap/lib/python3.12', '/opt/miniforge3/envs/chap/lib/python3.12/lib-dynload', '/opt/miniforge3/envs/chap/lib/python3.12/site-packages', '/var/www/vhosts/climate-hydro-analytics-platform-v2/api/models']

DEBUG [11/15/2024 06:37:45 PM] Prefix: /opt/miniforge3/envs/chap

DEBUG [11/15/2024 06:37:45 PM] Exec: /usr/bin/python3

DEBUG [11/15/2024 06:37:45 PM] Base Prefix: /opt/miniforge3/envs/chap

I am spotting two potential problems:

1. I see no /opt/miniforge3/bin or /opt/miniforge3/envs/chap/bin in the $PATH
2. The sys.executable seems to be pointing to the system installation of python instead of the virtual environment set as python-home

What's confusing me is that it's getting the prefix right, and the application does actually run and doesn't experience import errors, so I think it is still somehow finding the the right python packages (except for one which is responsible for the error). 

First question is is are these values expected for $PATH and sys.executable? 

If I login as the user above (hydro) and echo $PATH, I get a very different value where I do see our python env:

/opt/miniforge3/envs/chap/bin:/opt/miniforge3/condabin:/home/hydro/.local/bin:/home/hydro/bin:/opt/miniforge3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/var/cfengine/bin

I have attempted to SetEnv in Apache virtual hosts .conf file as well as export it as a new file in /etc/sysconfig, but the printed output above is always the same.

Here's our LoadModule directive:

<IfModule !wsgi_module>

    LoadModule wsgi_module /opt/miniforge3/lib/python3.12/site-packages/mod_wsgi/server/mod_wsgi-py312.cpython-312-x86_64-linux-gnu.so

    WSGIPythonHome "/opt/miniforge3"

    WSGIRestrictEmbedded On

</IfModule>


And our WSGI config in vhosts (there are two hosts, each using their own python env at python-home):

<VirtualHost *:443>

  ServerName myServer

  ## Vhost docroot

  DocumentRoot "/var/www/vhosts/climate-hydro-analytics-platform-v2"

  ## Alias declarations for resources outside the DocumentRoot

  Alias /jobs "/output/jobs"

  ## Logging

  ErrorLog "/var/log/httpd/chap-error.log"

  LogLevel info

  ServerSignature Off

  CustomLog "/var/log/httpd/chap-access.log" combined

  ## SSL directives

  SSLEngine on

  SSLCertificateFile      "/etc/pki/tls/certs/myServer.crt"

  SSLCertificateKeyFile   "/etc/pki/tls/private/myKey.key"

  SSLVerifyClient         none

  SSLCACertificatePath    "/etc/pki/tls/certs"

  SSLCACertificateFile    "/etc/pki/tls/certs/sectigo.ca.crt"

  ## WSGI configuration

  WSGIApplicationGroup %{GLOBAL}

  WSGIDaemonProcess chap display-name=%{GROUP} home=/var/www/vhosts/climate-hydro-analytics-platform-v2 python-home=/opt/miniforge3/envs/chap processes=10 threads=1 user=hydro

  WSGIProcessGroup chap

  WSGIScriptAlias / "/var/www/vhosts/climate-hydro-analytics-platform-v2/wsgi.py"

  #...more stuff

</VirtualHost>

<VirtualHost *:443>

  ServerName myServer2

  ## Vhost docroot

  DocumentRoot "/var/www/vhosts/hydrosource-2"

  ## Logging

  ErrorLog "/var/log/httpd/dataexplorer-error.log"

  LogLevel warn

  ServerSignature Off

  CustomLog "/var/log/httpd/dataexplorer-access.log" combined

  ## SSL directives

  SSLEngine on

  SSLCertificateFile      "/etc/pki/tls/certs/myServer.crt"

  SSLCertificateKeyFile   "/etc/pki/tls/private/myKey.key"

  SSLVerifyClient         none

  SSLCACertificatePath    "/etc/pki/tls/certs"

  SSLCACertificateFile    "/etc/pki/tls/certs/sectigo.ca.crt"

  ## WSGI configuration

  WSGIApplicationGroup %{GLOBAL}

  WSGIDaemonProcess dataexplorer display-name=%{GROUP} home=/var/www/vhosts/hydrosource-2 python-home=/opt/miniforge3/envs/dataexplorer processes=10 threads=1 user=hydro

  WSGIProcessGroup dataexplorer

  WSGIScriptAlias / "/var/www/vhosts/hydrosource-2/wsgi.py"

  #...more stuff

</VirtualHost>

I realize that conda isn't recommend and pip is, but we're using conda due to difficulties with managing scientific package dependencies (lots of C libraries) with pip. Given the error we're receiving, it is likely it's some C-library stuff conflicting somehow with whatever mod_wsgi is doing under the hood, so we may have to abandon mod_wsgi due to that. But I am very curious about the answer to my above question. 

If you are curious about the error, we have a python package called xarray where we call the method xarray.open_dataset(...).  This prompts the xarray package to import a package called netCDF4. This causes an import error like so:

DEBUG [11/15/2024 06:37:45 PM] Traceback (most recent call last):

  File "/var/www/vhosts/climate-hydro-analytics-platform-v2/api/blueprints/climate_viewer.py", line 69, in get_chart_data

    ctrl_ds = xarr.open_dataset(control, decode_times=False)

              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/opt/miniforge3/envs/chap/lib/python3.12/site-packages/xarray/backends/api.py", line 671, in open_dataset

    backend_ds = backend.open_dataset(

                 ^^^^^^^^^^^^^^^^^^^^^

  File "/opt/miniforge3/envs/chap/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 666, in open_dataset

    store = NetCDF4DataStore.open(

            ^^^^^^^^^^^^^^^^^^^^^^

  File "/opt/miniforge3/envs/chap/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 415, in open

    import netCDF4

  File "/opt/miniforge3/envs/chap/lib/python3.12/site-packages/netCDF4/__init__.py", line 3, in <module>

    from ._netCDF4 import *

ImportError: /opt/miniforge3/envs/chap/lib/python3.12/site-packages/netCDF4/../../.././libcurl.so.4: undefined symbol: nghttp2_option_set_no_rfc9113_leading_and_trailing_ws_validation

If I login as the hydro user,  activate the /opt/miniforge3/envs/chap environment, enter python and run the same piece of code (xarray.open_dataset(...)), it works without error. So there's something getting confused internally when going through mod_wsgi and Apache, but we have yet to figure out what after days of trying different things. You told me in my last question that PHP could be a problem, but we do not have PHP installed (and we are not using Anaconda official distribution either, but rather the open source distribution that utilizes conda and mamba called Miniforge3). 

Thanks for your continued support! I appreciate it.

[Graham Dumpleton]

Two reasons why see debug output you do.

1. Apache httpd starts as root if starting it from systemd and then drops privileges. Even if you use mod_wsgi daemon mode and tell it to run daemon processes as a specific user, that users login environment is never sourced and so will not pick up PATH changes in users local .bashrc/.zshrc/.profile/etc.

2. When Python is run in an embedded system, the "python" executable is never run and instead in this case the mod_wsgi module links to the Python library. In this case sys.executable has no meaning and can't be relied on for anything.

As to the missing nghttp2 symbol, it is likely due to Anaconda Python supplying its own copy of that library, but some other Apache module is linking to the system variant of that library and they are different versions. Any Python module installed in Anaconda Python will though have been compiled against the Anaconda version. Because though the system version gets loaded into the process first by way of the separate Apache module (or even Apache itself), the system one wins out and the Anaconda version of the library is never used, so if Python modules were built with expectation of a certain symbol existing, but that does not exist in system version of that library, then you will get that error.

If you want to test this theory, then disable mod_http2 in Apache since it seems to link to nghttp2. https://httpd.apache.org/docs/2.4/howto/http2.html

Make sure you do a full stop/start of Apache to ensure that system nghttp2 library is unloaded.

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/modwsgi/3fda8fd8-5847-4ee1-8849-80248a9ca006n%40googlegroups.com.

[Molly Foley]

Thanks Graham. If I remade the two environments using venv instead, wouldn’t it still break because an Apache module is still loading some other version of a package that overrides the version required by a dependency in my Python env? I guess I don’t get why venv could potentially work but conda breaks. Is it just because the “base” environment of venv is your system install of Python while the “base” environment of conda is, more or less, a virtual environment itself that doesn’t interact with the system install of Python? Is that where the conda copies of conflicting libraries occur?

You received this message because you are subscribed to a topic in the Google Groups "modwsgi" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/modwsgi/gYt2YLmm0cY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to modwsgi+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/modwsgi/53A671DA-388D-4EEA-A2F3-F2F268DE7601%40gmail.com.

[Graham Dumpleton]

If you don't use Anaconda Python and use standard Python distribution it will likely work fine. This is because the netCDF4 when installed would link with the system version of libcurl which would link with system libnghttp2 library. When you use Anaconda Python, it would link with libcurl/libnghttp2 shipped with Anaconda Python and it is that version which conflicts with mod_http2 in Apache httpd.

Note though that using "python -v venv" instead of "conda env" will not help unless the Python version being used at that point is system Python and not Anaconda distro based Python.

To understand, if you look in the directory "/opt/miniforge3/envs/chap/lib" of your virtual environment created using Anaconda Python you should see lots of shared libraries (or symlinks for them to actual Python distro). Follow where those links are pointing at and you will see the target directory in the Anaconda Python distro has its own versions of lots of libraries. It is those libraries being used which is the problem since mod_http2 in Apache httpd will not be linked to those but the system versions.

Hope this better explains it.

Graham

To view this discussion visit https://groups.google.com/d/msgid/modwsgi/CABLg_%3DOAdSHoAYV7z_A3GMt1k0bsvUxPckzcajJkRX0Zd7Wc2A%40mail.gmail.com.