mod_wsgi fails on import, using system Python not venv

[A McBain]

Hi, I looked at previous messages and others on StackOverflow but none seem to solve my issue.

I have an app I wrote working perfectly fine under Python 3.11 with mod_wsgi and Apache 2.

I did a bunch of development on the app (upgraded django, new features), and set up a new checkout of that on my server, with its own virtual environment (using venv). It uses effectively the same config (different subdomain) in Apache2 as the original older copy, but the new one fails with an import error while the old one is still chugging along.

I double checked the instructions at https://docs.djangoproject.com/en/5.0/howto/deployment/wsgi/modwsgi/

I've also re-verified the mod_wsgi I installed (via pip) matches the Python version I'm using (both are Python 3.11).

I also tried:

• Removing the python-path argument
  • The mod_wsgi docs suggest I don't need that if I specify python-home?
• Setting WSGIApplicationGroup to %{GLOBAL}
  

Unfortunately the error didn't change at all after trying those.

I've attached the relevant apache2 config section which includes all the mod_wsgi-setup (the rest is just redirects, ssl stuff, aliases, etc.)

I also attached the error from the log. It looks like it's trying to use the system Python instead of the one that exists in my .env (virtual environment) directory.

I'm banging my head as to why this worked before but not now so any help is much appreciated. Thank you! 🙂

Other details:

• wsgi.prod.py is the default wsgi.py, just modified to load settings.prod.py
• The server is Devuan Daedalus
  

[Graham Dumpleton]

On 13 May 2024, at 2:02 PM, A McBain <mcbai...@gmail.com> wrote:

Hi, I looked at previous messages and others on StackOverflow but none seem to solve my issue.

I have an app I wrote working perfectly fine under Python 3.11 with mod_wsgi and Apache 2.

I did a bunch of development on the app (upgraded django, new features), and set up a new checkout of that on my server, with its own virtual environment (using venv). It uses effectively the same config (different subdomain) in Apache2 as the original older copy, but the new one fails with an import error while the old one is still chugging along.

When you say "old one is still chugging along" can you confirm you are saying that the existing one is still running on the same server at the same time, or it is a completely new server you have set up.

I double checked the instructions at https://docs.djangoproject.com/en/5.0/howto/deployment/wsgi/modwsgi/

I've also re-verified the mod_wsgi I installed (via pip) matches the Python version I'm using (both are Python 3.11).

I also tried:

• Removing the python-path argument
  • The mod_wsgi docs suggest I don't need that if I specify python-home?

Removing python-path if it refers to site-packages directory of virtual environment specified to python-home is okay as path would be redundant. If python-path is used to tell it where your Django project is, that still needs to stay.

• Setting WSGIApplicationGroup to %{GLOBAL}
  

That is recommended, but if you are hosting multiple WSGI applications on the same server, they would need to be delegated to different mod_wsgi daemon process groups.

Unfortunately the error didn't change at all after trying those.

I've attached the relevant apache2 config section which includes all the mod_wsgi-setup (the rest is just redirects, ssl stuff, aliases, etc.)

I also attached the error from the log. It looks like it's trying to use the system Python instead of the one that exists in my .env (virtual environment) directory.

The virtual environment doesn't have a copy of Python stdlib files so they are still imported from the system Python the virtual environment was created from. The virtual environment effectively only has its own site-packages directory.

I'm banging my head as to why this worked before but not now so any help is much appreciated. Thank you! 🙂

Other details:

• wsgi.prod.py is the default wsgi.py, just modified to load settings.prod.py
• The server is Devuan Daedalus

At the start of your WSGI script file add:

    import sys

    print('sys.prefix', repr(sys.prefix))

    print('sys.path', repr(sys.path))

and check Apache logs for what paths it is using.

Compare that to the working site.

Post what you learn from that.

Graham

[A McBain]

Apologies, I noticed a small issue with the previously attached config. Unfortunately I don't get to be so lucky as to have that solve my problem. 😄 (I'd asked to load wsgi.py instead of wsgi.prod.py while only granting access to the latter.)

--
You received this message because you are subscribed to a topic in the Google Groups "modwsgi" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/modwsgi/IJp7zr6SjtY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/1856a102-140f-4174-a326-73d170d719bdn%40googlegroups.com.

[A McBain]

The old one is running on the same physical server, just with its own Apache config and own subdomain. As to having multiple things running, that's why I set WSGIApplicationGroup to %{SERVER} but just in case I changed it explicitly to "enfilade.asmcbain.net" (no effect, but shouldn't hurt anything to keep the change).

This is the result of the wsgi.prod.py printouts:

[Sun May 12 21:29:11.788518 2024] [wsgi:error] [pid 5671:tid 140047682610880] sys.prefix '/home/asmcbain/enfilade/.env'
[Sun May 12 21:29:11.788639 2024] [wsgi:error] [pid 5671:tid 140047682610880] sys.path ['/home/asmcbain/enfilade', '/usr/lib/python311.zip', '/usr/lib/python3.11', '/usr/lib/python3.11/lib-dynload', '/home/asmcbain/enfilade/.env/lib/python3.11/site-packages']

Alastair

--
You received this message because you are subscribed to a topic in the Google Groups "modwsgi" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/modwsgi/IJp7zr6SjtY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to modwsgi+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/7184C263-1522-4583-A98B-EC9806B0FA66%40gmail.com.

[Graham Dumpleton]

On 13 May 2024, at 2:33 PM, A McBain <mcbai...@gmail.com> wrote:

The old one is running on the same physical server, just with its own Apache config and own subdomain. As to having multiple things running, that's why I set WSGIApplicationGroup to %{SERVER} but just in case I changed it explicitly to "enfilade.asmcbain.net" (no effect, but shouldn't hurt anything to keep the change).

The WSGIApplicationGroup directive sets which Python interpreter context is used. It is distinct from the daemon process group.

You want each WSGI application instance to run in a different mod_wsgi daemon process group if using the main Python interpreter context (application group %{GLOBAL}).

So what you want is for each VirtualHost to use a different named mod_wsgi dameon process group.

<VirtualHost ...>

        ServerName site-1.example.com

        WSGIDaemonProcess site-1          \

                user=asmcbain group=asmcbain             \

                display-name='%{GROUP}'                  \

                lang='en_US.UTF-8'                       \

                locale='en_US.UTF-8'                     \

                threads=5                                \

                queue-timeout=45                         \

                socket-timeout=60                        \

                connect-timeout=15                       \

                request-timeout=60                       \

                inactivity-timeout=0                     \

                startup-timeout=15                       \

                deadlock-timeout=60                      \

                graceful-timeout=15                      \

                eviction-timeout=0                       \

                restart-interval=0                       \

                shutdown-timeout=5                       \

                maximum-requests=0                       \

                python-home=/home/asmcbain/enfilade/.env \

                python-path=/home/asmcbain/enfilade

        WSGIProcessGroup site-1

        WSGIApplicationGroup %{GLOBAL}

        WSGIScriptAlias / /home/asmcbain/enfilade/atcid/wsgi.py

</VirtualHost>

<VirtualHost ...>

        ServerName site-2.example.com

        WSGIDaemonProcess site-2        \

                user=asmcbain group=asmcbain             \

                display-name='%{GROUP}'                  \

                lang='en_US.UTF-8'                       \

                locale='en_US.UTF-8'                     \

                threads=5                                \

                queue-timeout=45                         \

                socket-timeout=60                        \

                connect-timeout=15                       \

                request-timeout=60                       \

                inactivity-timeout=0                     \

                startup-timeout=15                       \

                deadlock-timeout=60                      \

                graceful-timeout=15                      \

                eviction-timeout=0                       \

                restart-interval=0                       \

                shutdown-timeout=5                       \

                maximum-requests=0                       \

                python-home=/home/asmcbain/enfilade/.env \

                python-path=/home/asmcbain/enfilade

        WSGIProcessGroup site-2

        WSGIApplicationGroup %{GLOBAL}

        WSGIScriptAlias / /home/asmcbain/enfilade/atcid/wsgi.py

</VirtualHost>

So each site should have different name to WSGIDaemonProcess, with matching WSGIProcessGroup, and WSGIApplicationGroup should be %{GLOBAL} for both, not {SERVER}.

The {SERVER} value has a different meaning and best avoiding it.

So lets make sure each is running in separate daemon process groups.

BTW, also make sure that somewhere outside of all VirtualHost definitions you add:

    WSGIRestrictEmbedded On

This will cause an error if mod_wsgi daemon process groups aren't being configured properly.

Graham

You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/CAGV_ScocfNc%3DSv7W7Ha3MbrM_0wq43v8U0AM3JOuNtsfP9vK%3Dw%40mail.gmail.com.

[A McBain]

On Sun, May 12, 2024 at 10:05 PM A McBain <mcbai...@gmail.com> wrote:

> OK, I updated both the old site and the new one to make
> WSGIApplicationGroup %{GLOBAL}. They always had different
> WSGIProcessGroup values.
>
> I also added WSGIRestrictEmbedded to the base Apache config.
>
> Apache restarts fine, and the old site loads up but the new one still
> prints the import math error even after the changes. 😕
>
> I've attached the relevant bits to both.
>
> - Alastair

Originally I'd compiled this Python 3.11 myself, but I've just double
checked and verified Python 3.11 and the -dev package are from
official packages and not my own manual build+install.

I also checked that inside or outside of the virtual env (results are
the same) I can run the Python interpreter shell and do:

import math
math.ceil(1.1)

I don't get any errors.

- Alastair

> > To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/41CA1CF3-8A06-46BF-A7AE-2CA74FCC2196%40gmail.com.

[A McBain]

OK, I updated both the old site and the new one to make
WSGIApplicationGroup %{GLOBAL}. They always had different
WSGIProcessGroup values.

I also added WSGIRestrictEmbedded to the base Apache config.

Apache restarts fine, and the old site loads up but the new one still
prints the import math error even after the changes. 😕

I've attached the relevant bits to both.

- Alastair

>

> To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/41CA1CF3-8A06-46BF-A7AE-2CA74FCC2196%40gmail.com.

[A McBain]

Update to that. The modules loaded are different between the daemon
one and the regular interpreter.

Doing "print(sys.builtin_module_names)" results in the following for
the daemon version:

[Sun May 12 22:31:33.744235 2024] [wsgi:error] [pid 7430:tid
140134038087360] ('_abc', '_ast', '_codecs', '_collections',
'_functools', '_imp', '_io', '_locale', '_operator', '_signal',
'_sre', '_stat', '_string', '_symtable', '_thread', '_tokenize',
'_tracemalloc', '_warnings', '_weakref', 'atexit', 'builtins',
'errno', 'faulthandler',
'gc', 'itertools', 'marshal', 'posix', 'pwd', 'sys', 'time', 'xxsubtype')

And this substantially longer list for the one I just run from the CLI myself:

('_abc', '_ast', '_bisect', '_blake2', '_codecs', '_collections',
'_csv', '_datetime', '_elementtree', '_functools', '_heapq', '_imp',
'_io', '_locale', '_md5', '_opcode', '_operator', '_pickle',
'_posixsubprocess', '_random', '_sha1', '_sha256', '_sha3', '_sha512',
'_signal', '_socket', '_sre', '_stat', '_statistics', '_string',
'_struct', '_symtable', '_thread', '_tokenize', '_tracemalloc',
'_warnings', '_weakref', 'array', 'atexit', 'binascii', 'builtins',
'cmath', 'errno', 'faulthandler', 'fcntl', 'gc', 'grp', 'itertools',
'marshal', 'math', 'posix', 'pwd', 'pyexpat', 'select', 'spwd', 'sys',
'syslog', 'time', 'unicodedata', 'xxsubtype', 'zlib')

Now I just have to figure out why.

[Graham Dumpleton]

Ensure you do a complete stop and start of Apache and not just reload or restart in case Apache had cached a reference to a broken Python library variant from your self compiled build.

If going to build your own Python from source code ensure you read. A bit dated, but still relevant. Ignore that talks about Docker, still applies for any Python build.

Also, where is LoadFile line in Apache picking up mod_wsgi.so from?

Graham

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/CAGV_ScoxJu%3DxKwijvbPjE78BnZZD4s2ppRBT0-jCErCgdr8v%2Bg%40mail.gmail.com.

[A McBain]

On Sun, May 12, 2024 at 10:57 PM Graham Dumpleton <graham.d...@gmail.com> wrote:
>
> Ensure you do a complete stop and start of Apache and not just reload or restart in case Apache had cached a reference to a broken Python library variant from your self compiled build.
>
> If going to build your own Python from source code ensure you read. A bit dated, but still relevant. Ignore that talks about Docker, still applies for any Python build.
>
> Installing a custom Python version into a Docker image.
> blog.dscpl.com.au
>

> Also, where is LoadFile line in Apache picking up mod_wsgi.so from?
>
> Graham

I don't intend to build from source again if I can help it. That was a PITA. 😄 I only did it way back since I believe at the time Python 3.11 wasn't available as a package for Devuan/Debian yet...? I honestly don't remember. Thank you for the link though if I ever do need. 🙂

I had been just doing "service apache2 restart" yeah, but I just tried a proper stop, then a start, and the state is the same. 😑

So best I can tell this is where I'm at:

• CLI-run Python, the older site, and new site, have identical paths except for the first entry (CLI = '', apps = their app dir)
  
• Both apps don't import cmath and a bunch of other modules but the CLI Python does
• All prefixes point to the correct virtual env root
  
• The old app still runs even after a full stop and start of Apache

I apologize for all this. I don't have any idea what's going on and I've tried everything I can think of, what you've asked / suggested, and it's still breaking my brain.

- Alastair

> To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/55163B95-DCAF-4129-BD81-4D477F4FC9F6%40gmail.com.

[A McBain]

On Sun, May 12, 2024 at 11:08 PM A McBain <mcbai...@gmail.com> wrote:
>
>
>
> On Sun, May 12, 2024 at 10:57 PM Graham Dumpleton <graham.d...@gmail.com> wrote:
> >
> > Ensure you do a complete stop and start of Apache and not just reload or restart in case Apache had cached a reference to a broken Python library variant from your self compiled build.
> >
> > If going to build your own Python from source code ensure you read. A bit dated, but still relevant. Ignore that talks about Docker, still applies for any Python build.
> >
> > Installing a custom Python version into a Docker image.
> > blog.dscpl.com.au
> >
> > Also, where is LoadFile line in Apache picking up mod_wsgi.so from?
> >
> > Graham
>
> I don't intend to build from source again if I can help it. That was a PITA. 😄 I only did it way back since I believe at the time Python 3.11 wasn't available as a package for Devuan/Debian yet...? I honestly don't remember. Thank you for the link though if I ever do need. 🙂
>
> I had been just doing "service apache2 restart" yeah, but I just tried a proper stop, then a start, and the state is the same. 😑
>
> So best I can tell this is where I'm at:
>
> CLI-run Python, the older site, and new site, have identical paths except for the first entry (CLI = '', apps = their app dir)
> Both apps don't import cmath and a bunch of other modules but the CLI Python does
> All prefixes point to the correct virtual env root
> The old app still runs even after a full stop and start of Apache
>
> I apologize for all this. I don't have any idea what's going on and I've tried everything I can think of, what you've asked / suggested, and it's still breaking my brain.
>
> - Alastair

Oh, the load command. Right, so that's at "/etc/apache2/mods-enabled/wsgi.load"

The contents are this:

#LoadModule wsgi_module /usr/lib/apache2/modules/mod_wsgi.so
LoadModule wsgi_module
/home/asmcbain/.local/lib/python3.11/site-packages/mod_wsgi/server/mod_wsgi-py311.cpython-311-x86_64-linux-gnu.so

[Graham Dumpleton]

The mod_wsgi.so specified by LoadFile is what dictates version (and in what location) of Python is being used.

Right now you are using one under your home directory. I would not be surprised if it is compiled for the version of Python you built yourself from source code and that is causing the problem.

Ensuring you have no virtual environments activated, run the following:

pip3 uninstall mod_wsgi

Run this multiple times until it says nothing to uninstall.

Then try rebuilding/reinstall mod_wsgi again but when you run pip install use the --no-cache-dir option so it doesn't use old cached build.

Then update LoadFile line to use whatever location it now gets installed to if you use a different location (eg., a dedicated virtual environment, or system Python directory.

Ensure you stop and start Apache when changed.

Graham

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/CAGV_Scr3A_PxtgAtamReunLZ9zjG03OFG%2BZZFpQWV5aSzmo16Q%40mail.gmail.com.

[A McBain]

On Sun, May 12, 2024 at 11:19 PM Graham Dumpleton
<graham.d...@gmail.com> wrote:
>
> The mod_wsgi.so specified by LoadFile is what dictates version (and in what location) of Python is being used.
>
> Right now you are using one under your home directory. I would not be surprised if it is compiled for the version of Python you built yourself from source code and that is causing the problem.
>
> Ensuring you have no virtual environments activated, run the following:
>
> pip3 uninstall mod_wsgi
>
> Run this multiple times until it says nothing to uninstall.
>
> Then try rebuilding/reinstall mod_wsgi again but when you run pip install use the --no-cache-dir option so it doesn't use old cached build.
>
> Then update LoadFile line to use whatever location it now gets installed to if you use a different location (eg., a dedicated virtual environment, or system Python directory.
>
> Ensure you stop and start Apache when changed.
>
> Graham

OK I uninstalled the old one (there was only the .local copy). I ran
the install (with the flag) and it's now residing in the more expected
/usr/local/... python3.11 site-packages directory.

Unfortunately no dice.

I also did a full shutdown and boot. Nope. 😕

> To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/F97967BE-2139-4E76-A9DC-4C7A9B3A397E%40gmail.com.

[A McBain]

On Sun, May 12, 2024 at 11:37 PM A McBain <mcbai...@gmail.com> wrote:
>
> On Sun, May 12, 2024 at 11:19 PM Graham Dumpleton
> <graham.d...@gmail.com> wrote:
> >
> > The mod_wsgi.so specified by LoadFile is what dictates version (and in what location) of Python is being used.
> >
> > Right now you are using one under your home directory. I would not be surprised if it is compiled for the version of Python you built yourself from source code and that is causing the problem.
> >
> > Ensuring you have no virtual environments activated, run the following:
> >
> > pip3 uninstall mod_wsgi
> >
> > Run this multiple times until it says nothing to uninstall.
> >
> > Then try rebuilding/reinstall mod_wsgi again but when you run pip install use the --no-cache-dir option so it doesn't use old cached build.
> >
> > Then update LoadFile line to use whatever location it now gets installed to if you use a different location (eg., a dedicated virtual environment, or system Python directory.
> >
> > Ensure you stop and start Apache when changed.
> >
> > Graham
>
> OK I uninstalled the old one (there was only the .local copy). I ran
> the install (with the flag) and it's now residing in the more expected
> /usr/local/... python3.11 site-packages directory.
>
> Unfortunately no dice.
>
> I also did a full shutdown and boot. Nope. 😕

OK so for transparency it is working now! But heck if I know exactly why.

I know I probably caused this mess and I'm not sure how to fix it
exactly, so I'm just going to live in my mess for the time being,
since this is just a home dev server, and proper projects go to an
offsite server I purposefully haven't messed up.

Also, thank you very much for helping me debug this, I really appreciate it.

For anyone else I'll go through what I found in case it helps them
with their own weird environments.

Python versions on my server:
2.7.18 - the system one
3.11.1 - the one I compiled, only available at /usr/local/bin/python3.11
3.12.2 - the package/distro Python, and what is used when I do
"python3 --version", available at /usr/bin/python3 and
/usr/bin/python3.11

Inside the virtual environment folder of each app, which I'll just
call Old App and New App, there's a pyvenv.cfg file. Old App appears
to have been created with virtualenv, and New App was created with
venv, but testing shows that doesn't matter.

Old App had different file contents in pyvenv.cfg, presumably because
it was created prior to installing the package Python (3.11.2):

home = /usr/local/bin
implementation = CPython
version_info = 3.11.1.final.0
virtualenv = 20.17.1
include-system-site-packages = false
base-prefix = /usr/local
base-exec-prefix = /usr/local
base-executable = /usr/local/bin/python3.11

New App had this in pyvenv.cfg (after I recreated it using virtualenv):

home = /usr/bin
implementation = CPython
version_info = 3.11.2.final.0
virtualenv = 20.17.1+ds
include-system-site-packages = false
base-prefix = /usr
base-exec-prefix = /usr
base-executable = /usr/bin/python3

Once I changed /usr to /usr/local and python3 to python3.11 in the New
App, the import error disappeared and left me only with logical errors
due to mistakes in wsgi.prod.py and settings-prod.py leftover from
debugging. Fixing those let the app start up just fine.

Something weird is going on with the Python 3.11.2, or the interaction
between it and the Python 3.11.1, but as per above since I have the
app working I don't have the energy right this minute to untangle
that.

Thanks again,
Alastair

[Graham Dumpleton]

Since Python is embedded inside of Apache by mod_wsgi, ie., the Python library is linked into the process and the CLI Python is not run/used, all virtual environments you use with it must be created from the same version/directory installation of Python as mod_wsgi was compiled for. So it is sort of understandable that things would break or behave strangely if your two virtual environments were created off different Python installations.

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/CAGV_ScqPGkJ2r_UKL0rQHf6RHXLvfoSWrpmVk-tfc0gqCtvb%2BA%40mail.gmail.com.