ImportError: No module named site

[ryan west]

Hello,

I've spent a fair bit of time diagnosing this particular issue with Apache/mod_wsgi and could use some pointers. Any help is extremely appreciated.

Versions

• Ubuntu 14.04.6 LTS
• Python 2.7.8
• Python 3.6.8
• mod-wsgi 4.3.2 (installed via pip)

Virtualenvs

• /home/release/envs/stage-python2.7
• /home/release/envs/stage-python3.6

Depending on which version of python I want this instance to use, a symlink is created at /home/release/envs/stage that maps to either of the virtualenvs above. In this example, I am simply trying to get it working with python 2.7.

Apache Logs

ryanwest@el8-stage-experimental-webapp1:/home/release/envs$ sudo tail /var/log/apache2/error.log
[Wed Nov 06 16:38:51.586673 2019] [:info] [pid 2978:tid 139776709330816] mod_wsgi (pid=2978): Python home /home/release/envs/stage/bin.
[Wed Nov 06 16:38:51.586681 2019] [:info] [pid 2978:tid 139776709330816] mod_wsgi (pid=2978): Initializing Python.
[Wed Nov 06 16:38:51.585441 2019] [:info] [pid 2976:tid 139776709330816] mod_wsgi (pid=2976): Starting process 'http.www.stage.domain.com' with uid=33, gid=33 and threads=1.
[Wed Nov 06 16:38:51.588878 2019] [:info] [pid 2976:tid 139776709330816] mod_wsgi (pid=2976): Python home /home/release/envs/stage/bin.
[Wed Nov 06 16:38:51.588886 2019] [:info] [pid 2976:tid 139776709330816] mod_wsgi (pid=2976): Initializing Python.
[Wed Nov 06 16:38:51.589546 2019] [:info] [pid 2973:tid 139776709330816] mod_wsgi (pid=2973): Python home /home/release/envs/stage/bin.
[Wed Nov 06 16:38:51.589559 2019] [:info] [pid 2973:tid 139776709330816] mod_wsgi (pid=2973): Initializing Python.
ImportError: No module named site
ImportError: No module named site
ImportError: No module named site

Apache Conf (/etc/apache2/sites-available/stage.conf)

ryanwest@el8-stage-experimental-webapp1:/home/release/envs$ cat /etc/apache2/sites-available/stage.conf
# LB TERMINATED
<VirtualHost *:80>

    # apache default of 8190 is too small because we have some long uri
    LimitRequestLine 81900

    ServerName www.stage.domain.com

    DocumentRoot "/opt/domain/sites/www.stage.domain.com/current"

    ErrorLog  "/var/log/domain/www.stage.domain.com/www.error.django.log"
 
    LogLevel debug

    WSGIDaemonProcess http.www.stage.domain.com user=www-data maximum-requests=5000 processes=30 threads=1 python-eggs=/tmp display-name='%{GROUP}' python-home=/home/release/envs/stage/bin
    WSGIProcessGroup http.www.stage.domain.com
    WSGIApplicationGroup http.www.stage.domain.com
    WSGIImportScript /opt/domain/sites/www.stage.domain.com/wsgi_imports.py process-group=http.www.stage.domain.com application-group=http.www.stage.domain.com
    WSGIScriptAlias / /opt/domain/sites/www.stage.domain.com/virtualenv.wsgi
    # APP-8453: WSGI doesn't pass through auth headers by default
    WSGIPassAuthorization On

    XSendFile on
    XSendFilePath /mnt/www.stage.domain.com/media
    XSendFilePath /opt/domain/sites/www.stage.domain.com/current/media

    RewriteEngine on
    ProxyRequests On
    SSLProxyEngine on

    RewriteCond %{DOCUMENT_ROOT}/downtime-enabled.html -l
    RewriteCond %{REQUEST_URI} !\.(css|jpg|gif|png)$
    RewriteCond %{HTTP_HOST} !^\w+1\.
    RewriteCond %{SCRIPT_FILENAME} !downtime-enabled.html
    RewriteRule ^.*$ %{DOCUMENT_ROOT}/downtime-enabled.html [L]

</VirtualHost>

In this case, /home/release/envs/stage is a symlink to a python2.7 virtualenv:

ryanwest@el8-stage-experimental-webapp1:/home/release/envs$ ls -l
total 8
lrwxrwxrwx 1 root    root   34 Nov  6 12:11 stage -> /home/release/envs/stage-python2.7
drwxr-xr-x 6 release root 4096 Nov  6 15:41 stage-python2.7
drwxr-xr-x 4 release root 4096 Nov  6 12:10 stage-python3.6

ryanwest@el8-stage-experimental-webapp1:/home/release/envs$ ls -l stage-python2.7
total 16
drwxr-xr-x 2 release root 4096 Nov  6 14:27 bin
drwxr-xr-x 2 release root 4096 Nov  6 12:10 include
drwxr-xr-x 3 release root 4096 Nov  6 12:10 lib
drwxr-xr-x 3 root    root 4096 Nov  6 12:15 man

ryanwest@el8-stage-experimental-webapp1:/home/release/envs$ stage-python2.7/bin/python --version
Python 2.7.8
ryanwest@el8-stage-experimental-webapp1:/home/release/envs$ stage-python3.6/bin/python --version
Python 3.6.8

I realize that pointing python-home to the bin directory is strange. When I point it to simply /home/release/envs/stage, the apache error logs default to saying python home is /usr/bin/python. As a side note, my Apache logs don't seem to be providing me with many pertinent details. I have tried increasing the verbosity using LogLevel debug with no success.

What am I doing wrong? Thank you!

Ryan

[Graham Dumpleton]

Ensure you have read:

https://modwsgi.readthedocs.io/en/develop/user-guides/virtual-environments.html

The mod_wsgi module has to be compiled for the Python version you want to use as it links to the Python library, the Python executable is never itself run. You cannot use mod_wsgi compiled for a specific Python version with virtual environments for a different Python version.

On 7 Nov 2019, at 11:54 am, ryan west <ryan...@gmail.com> wrote:

Hello,

I've spent a fair bit of time diagnosing this particular issue with Apache/mod_wsgi and could use some pointers. Any help is extremely appreciated.

Versions

• Ubuntu 14.04.6 LTS
• Python 2.7.8
• Python 3.6.8
• mod-wsgi 4.3.2 (installed via pip)

Virtualenvs

• /home/release/envs/stage-python2.7
• /home/release/envs/stage-python3.6

Depending on which version of python I want this instance to use, a symlink is created at /home/release/envs/stage that maps to either of the virtualenvs above. In this example, I am simply trying to get it working with python 2.7.

You can't do that.

Suggest you use mod_wsgi-express instead and run separate instances of that for each Python version and then as necessary use the main system Apache to proxy to those separate instances.

Apache Logs

ryanwest@el8-stage-experimental-webapp1:/home/release/envs$ sudo tail /var/log/apache2/error.log
[Wed Nov 06 16:38:51.586673 2019] [:info] [pid 2978:tid 139776709330816] mod_wsgi (pid=2978): Python home /home/release/envs/stage/bin.
[Wed Nov 06 16:38:51.586681 2019] [:info] [pid 2978:tid 139776709330816] mod_wsgi (pid=2978): Initializing Python.
[Wed Nov 06 16:38:51.585441 2019] [:info] [pid 2976:tid 139776709330816] mod_wsgi (pid=2976): Starting process 'http.www.stage.domain.com' with uid=33, gid=33 and threads=1.
[Wed Nov 06 16:38:51.588878 2019] [:info] [pid 2976:tid 139776709330816] mod_wsgi (pid=2976): Python home /home/release/envs/stage/bin.
[Wed Nov 06 16:38:51.588886 2019] [:info] [pid 2976:tid 139776709330816] mod_wsgi (pid=2976): Initializing Python.
[Wed Nov 06 16:38:51.589546 2019] [:info] [pid 2973:tid 139776709330816] mod_wsgi (pid=2973): Python home /home/release/envs/stage/bin.
[Wed Nov 06 16:38:51.589559 2019] [:info] [pid 2973:tid 139776709330816] mod_wsgi (pid=2973): Initializing Python.
ImportError: No module named site
ImportError: No module named site
ImportError: No module named site

The error is specifically because you are mixing Python versions.

The Python executable is never invoked. What the location is reported as within the process doesn't matter.

As a side note, my Apache logs don't seem to be providing me with many pertinent details. I have tried increasing the verbosity using LogLevel debug with no success.

What am I doing wrong? Thank you!

Ryan

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/cd2d57fa-18c3-449f-92fe-47ccb46acf36%40googlegroups.com.

[ryan west]

Hi Graham.

Thanks very much for the prompt reply.

When using a Python virtual environment with mod_wsgi, it is very important that it has been created using the same Python installation that mod_wsgi was originally compiled for. It is not possible to use a Python virtual environment to force mod_wsgi to use a different Python version, or even a different Python installation.

You cannot for example force mod_wsgi to use a Python virtual environment created using Python 3.5 when mod_wsgi was originally compiled for Python 2.7. This is because the Python library for the Python installation it was originally compiled against is linked directly into the mod_wsgi module. In other words, Python is embedded within mod_wsgi. When mod_wsgi is used it does not run the command line python program to run the interpreter and thus why you can’t force it to use a different Python installation.

After reading this, I still find myself a bit confused as to what is happening under the hood, though I suspect that this is a gap in my knowledge rather than a deficiency in the documentation. If I am in a virtualenv for a specific version of python, wouldn't a pip install mod-wsgi install a version of the module that is compiled against a similar version of python? It seems like a reasonable assumption, and one that generally speaking has worked well for me.

In my use-case, I do not need to run the application in python 2 and 3 simultaneously, side-by-side on the same instance. It is completely acceptable for me to have either or. Do you still recommend using mod_wsgi-express?

Thanks again Graham. I will more thoroughly review the documentation page that you linked below.

Ryan

To unsubscribe from this group and stop receiving emails from it, send an email to mod...@googlegroups.com.

[Graham Dumpleton]

On 7 Nov 2019, at 12:52 pm, ryan west <ryan...@gmail.com> wrote:

Hi Graham.

Thanks very much for the prompt reply.

When using a Python virtual environment with mod_wsgi, it is very important that it has been created using the same Python installation that mod_wsgi was originally compiled for. It is not possible to use a Python virtual environment to force mod_wsgi to use a different Python version, or even a different Python installation.

You cannot for example force mod_wsgi to use a Python virtual environment created using Python 3.5 when mod_wsgi was originally compiled for Python 2.7. This is because the Python library for the Python installation it was originally compiled against is linked directly into the mod_wsgi module. In other words, Python is embedded within mod_wsgi. When mod_wsgi is used it does not run the command line python program to run the interpreter and thus why you can’t force it to use a different Python installation.

After reading this, I still find myself a bit confused as to what is happening under the hood, though I suspect that this is a gap in my knowledge rather than a deficiency in the documentation. If I am in a virtualenv for a specific version of python, wouldn't a pip install mod-wsgi install a version of the module that is compiled against a similar version of python? It seems like a reasonable assumption, and one that generally speaking has worked well for me.

The problem I see is that you have said you have two virtual environments of:

• /home/release/envs/stage-python2.7

• /home/release/envs/stage-python3.6

You may have a symlink to one or the other, but how are you controlling which mod_wsgi module is being loaded by Apache using the LoadModule line.

That LoadModule line is going to have to be to a file inside of the site-packages directory of the respective virtual environment when switch between them, but that directory path has a further directory (below those root directories for the virtual environment), which has the Python version in it. Eg:

    /home/release/envs/stage-python2.7/lib/python2.7/site-packages/mod_wsgi/.....

    /home/release/envs/stage-python3.6/lib/python3.6/site-packages/mod_wsgi/.....

So just changing the symlink to the root of the virtual environment is not enough, the LoadModule line in Apache would also need to change.

This suggests the LoadModule line is loading a system Python version, possibly Python 2.7. Which means that when change the stage link to 3.6 it will fail, as will be still running Python 2.7, but can't find all the 2.7 files in the Python 3.6 virtual environment since no lib/python2.7 directory exists. Thus why site module fails to load.

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/63e2568a-aeb8-4b24-81b4-61012fe45b88%40googlegroups.com.