Update in Python killed my server - tried a reinstall with python3.9

[Isaac Rutel]

So as the title says, I didn't pay attention to a python install and I think it removed the python 3.8 I had been using (site went down and now trying to fix).

I ended up creating a new environment and going through the "fresh installation" to generate the appropriate static files.  In order to get the correct versions, I used the requirements.txt document and it loaded the appropriate packages (pip install -r requirements.txt) while I was in the venv (setup to use python 3.9).

Ran the various tests from the install section and got "OK"s for the packages and installed modules.

To make a long story short, I updated the virtual server to point to the new virtual environment and it gives me a traceback error (apache2/error.log):

[Wed Nov 23 15:38:50.574889 2022] [mpm_event:notice] [pid 707380:tid 140557775439744] AH00492: caught SIGWINCH, shutting down gracefully
[Wed Nov 23 15:38:50.689603 2022] [mpm_event:notice] [pid 709668:tid 139843898124160] AH00489: Apache/2.4.52 (Ubuntu) mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
[Wed Nov 23 15:38:50.689800 2022] [core:notice] [pid 709668:tid 139843898124160] AH00094: Command line: '/usr/sbin/apache2'
[Wed Nov 23 15:38:50.707000 2022] [wsgi:error] [pid 709669:tid 139843898124160] mod_wsgi (pid=709669): Failed to exec Python script file '/home/<user>/web/qatrackplus/qatrack/wsgi.py'.
[Wed Nov 23 15:38:50.707045 2022] [wsgi:error] [pid 709669:tid 139843898124160] mod_wsgi (pid=709669): Exception occurred processing WSGI script '/home/<user>/web/qatrackplus/qatrack/wsgi.py'.
[Wed Nov 23 15:38:50.710544 2022] [wsgi:error] [pid 709669:tid 139843898124160] Traceback (most recent call last):
[Wed Nov 23 15:38:50.710573 2022] [wsgi:error] [pid 709669:tid 139843898124160]   File "/home/<user>/web/qatrackplus/qatrack/wsgi.py", line 23, in <module>
[Wed Nov 23 15:38:50.710577 2022] [wsgi:error] [pid 709669:tid 139843898124160]     from django.core.wsgi import get_wsgi_application
[Wed Nov 23 15:38:50.710586 2022] [wsgi:error] [pid 709669:tid 139843898124160] ModuleNotFoundError: No module named 'django'

<user> is used instead of my installed user.

So, I notice that after the SIGWINCH (not sure what this is) is recovers into using python3.10 which I assume is apache2's environment.  However, I noticed in another discussion that there is a conflict between the mod_wsgi modules that apache may be using and the one I want it to use.

There was comment about installing mod_wsgi directly (and I think using it as a selector for which version it to use for webpages (mod_wsgi.server) but I wasn't able to install the module on its own, so I couldn't go down that route.

I am at a loss for what to do next.  Any help would be greatly appreciated.

Thanks!

Isaac

[Isaac Rutel]

Sorry, forgot to mention that I installed the same version of Django for both environments (2.2.18) so it would hopefully see the correct module, but it doesn't matter. I also did a check of the installed components in both the venv and the apache2 environment (running python10), and they both find the django module and with the version I installed. So, again I am not sure why this isn't finding django in general, it is there 2x in the same version.

[tbe...@gmail.com]

Hi Isaac,

I'm not sure what you meand with "apache2 environment" could you please check your site specific apache config which should be in /etc/apache2/sites-enabled/qatrack.conf

There should be a reference to your Qatrack+ venv:  WSGIDaemonProcess qatrackplus python-home=/home/<user>/venvs/qatrack31 python-path=/home/<user>/web/qatrackplus

regards

Thomas

[Isaac Rutel]

Thank you for the response. I updated my WSGIDaemonProcess (manually) to match my newly installed venv (with the python3.9 environment variables), which is now python-home=/home/<user>/.venvs/qatrack31v2, but the python-path is the same, since I didn’t change the /home/<user>/web/qatrackplus (the python file location for qatrackplus).

I was hoping that this redirection would be OK for my environment selection. I am curious to know whether my change in both the venv name and/or the location (.venvs vs venvs) would cause issues in any of the files or config options. I didn’t think it would be an issue if I redirected to my newly installed virtual environment, but obviously it isn’t working, hence this question thread.

I was also wondering if I am missing a call to the new environment somewhere?

By apache2 environment, I mean that Apache is started on boot from the root system under the main environment variables, so it calls/uses python 3.10 in the non venv I set up. However, as I mentioned above, the call for Django (through the WSGI calls SHOULD be the venv python 3.9, given the setting in the site-enabled config. 

As I mentioned in the original post, I did see issues surrounding not finding the encoding module (for a QATrack+ install, which was fixed by installing mod_wsgi and using it as a server to select (force) the use of the older library file, but mod_wsgi won’t install on my system. So I couldn’t try that solution on my system. 

Thanks again for the help,

Isaac

Sent from my iPhone

On Nov 24, 2022, at 1:51 AM, tbe...@gmail.com <tbe...@gmail.com> wrote:



--
You received this message because you are subscribed to a topic in the Google Groups "QATrack+" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/qatrack/4gxedDeb6mo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to qatrack+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qatrack/b60dfd6d-8115-4348-ab68-bf95b00b2e9bn%40googlegroups.com.

[Randle Taylor]

Hi Isaac,

It's not 100% clear to me what's going on but the the Python version of your virtualenv needs to match the Python version that Apache uses (see Graham's, author of mod_wsgi's answer here): https://stackoverflow.com/a/40951638).

Another option for you would be to forgo mod_wsgi and use Apache (or nginx) to serve static assets and reverse proxy requests to gunicorn (a standalone wsgi server).   There are many guides online on how to set this up and then it wouldn't matter the version of Python that Apache/mod_wsgi is using since gunicorn gets installed in your virtualenv and Apache doesn't need to involve Python at all then.

Randy

You received this message because you are subscribed to the Google Groups "QATrack+" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qatrack+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qatrack/107EFE3A-6821-43DC-805C-E2D2920B291F%40yahoo.com.

[tbe...@gmail.com]

Hi Randy,

as far as I rember, Python 3.10 was introduced with Ubuntu 22.04. My installation is still on 20.04 because I am not sure if QATrack+ and/or pylinac both support python 3.10.

Is it possible to install QAtrack on Ubuntu 22.04 or possible to upgrade an existing installation to Ubuntu 22.04?

regards

Thomas

[Geoff Scott]

Morning 

As far as I know the version of python you are using is not supported it need to be 3.9.x version 3.10.x is too new. I ran into this problem when I installed for the first time. 

Sent from Outlook for iOS

---

From: qat...@googlegroups.com <qat...@googlegroups.com> on behalf of tbe...@gmail.com <tbe...@gmail.com>
Sent: Wednesday, November 30, 2022 8:46:35 AM
To: QATrack+ <qat...@googlegroups.com>
Subject: Re: [QATrack+ 2963] Re: Update in Python killed my server - tried a reinstall with python3.9

 

To view this discussion on the web visit https://groups.google.com/d/msgid/qatrack/91fd29e7-bf3c-4057-b2a6-a491621b4e54n%40googlegroups.com.

[Isaac Rutel]

Thank you, I was aware that python3.9 is not supported, that is why there is a virtual environment activated.  As initially stated, it appeared that apache was still using the upgraded python version on the system (3.10), however, I was able to modify that through the update and compilation/install of mod_wsgi.  Currently, the apache logs now show a 3.9x version of python. It appears that the last post I submitted was deleted? Maybe this is the confusino for the version.

The update to the system now shows a 3.9x version of python, however, I am still not able to see the app page through a browser.  I am currently getting indications that the system is performing some functions, but it is still not serving to the host address.

In the previous post, I added the last of the information from the supervisor-django-q.err.log, but maybe there was something that caused the post removal (so I am hesitant to re-paste here).

The gist of the log shows processes being run but for the last entry there is an ?identifier in brackets:[bravo-taco-beast-failed] (This is made up from the actual in case there is an issue with listing these values), but the "failed" portion made me wonder if this was indicative of an issue.  There is "fourteen" as an entry for "f" if this is an attempt at radio call clarity, so "failed" doesn't need to be used.  As for the only other issue, there is a warning call about matplotlib and access to a writable directory, but that appears to be a non-fatal warning, so I am not sure if that is an issue.  The fix seems to be code side, not permission in the os side, so I hadn't tried to hunt down making the appropriate folder with updated permissions.

There does appear to be a q cluster running in the log . . . so again, I am not sure why I am not seeing something in my browser.

[Isaac Rutel]

Sorry, I meant I was aware that python 3.10 is not supported.  The virtual environment has python 3.9 installed in an attempt to get around this issue, given the os update forced python 3.10 on my server.

[Randle Taylor]

Hi Isaac,

Did you have any luck with this?  Django Q is not responsible for serving the web pages so that is not the problem.  It still sounds like an Apache issue.  If you try to visit the page in your browser then look in the apache error log file (/var/log/apache2/error.log) you should have some indication of what the problem is.

Randy

On Sun, 4 Dec 2022 at 16:34, 'Isaac Rutel' via QATrack+ <qat...@googlegroups.com> wrote:

Thanks Randy,

  I saw the same article independently (but it is nice to have confirmation), so I d/l'ed the tarball for mod_wsgi and installed it using the correct flag to link it to my virtual environment python.  The good news, is that I am no longer getting the error in the apache error log . . . however, I am still not able to get it to show the application when pointing at it in my browser.  I am able to confirm (in the apache error log) that I am running:

Apache/2.4.52 (Ubuntu) mod_wsgi/4.9.4 Python/3.9, and this is the updated version of the mod_wsgi from the initial install (mod_wsgi/4.9.0 and python/3.10), so it is an improvement on the apache/mod_wsgi/python side, since everything appears to be in appropriate levels.  Now, I didn't d/l the mod_wsgi version exactly, so is this going to be an issue, even if I compiled it in my virtual environment?

The bad news is that I am still not getting the page/application in the browser and now I don't appear to have an error to point to my issue.  Although I did look at the supervisor-django-q.err.log and in the latest restart I got this set of reports:

Running run_service_event_review_notices task at 2022-11-28 15:07:54.188773 for notices between 2022-11-28 15:07:54.188773 and 2022-11-28 15:22:54.188773
queueing from qatrack
Processed [emma-beryllium-timing-glucose]
Processed [fillet-fourteen-river-oxygen]
MainProcess got signal UNKNOWN
Q Cluster lemon-robin-alaska-undress stopping.
Q Cluster lemon-robin-alaska-undress has stopped.
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-i1dqy6qi because the default path (/var/www/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Q Cluster black-carbon-timing-connecticut starting.
Process-1:1 ready for work at 40978
Process-1:2 ready for work at 40979
Process-1:3 monitoring at 40980
Process-1 guarding cluster black-carbon-timing-connecticut
Process-1:4 pushing tasks at 40981
Q Cluster black-carbon-timing-connecticut running.
Broker in an atomic transaction
Broker in an atomic transaction
Enqueued 1777
Pushed ('rugby-oven-west-mobile', '8aa36ffbdb7a444cad8f00d0a81ea69f')
Process-1 created a task from schedule [QATrack+ Scheduling Notices]
Broker in an atomic transaction
Broker in an atomic transaction
Enqueued 1778
Pushed ('hotel-sodium-fourteen-triple', '421b6d6b648b4163b02823f6bb7daf88')
Process-1 created a task from schedule [QATrack+ Review Notices]
Broker in an atomic transaction
Broker in an atomic transaction
Enqueued 1779
Pushed ('lima-march-bacon-failed', '32c4f976c6a444a5b75ede2785f6fdcf')
Process-1 created a task from schedule [QATrack+ Service Event Review Notices]
queueing from qatrack
Process-1:1 processing [rugby-oven-west-mobile]
Running run_scheduling_notices task at 2022-11-28 15:22:36.414293 for notices between 2022-11-28 15:22:36.414293 and 2022-11-28 15:37:36.414293
queueing from qatrack
Process-1:2 processing [hotel-sodium-fourteen-triple]
Running run_review_notices task at 2022-11-28 15:22:36.429436 for notices between 2022-11-28 15:22:36.429436 and 2022-11-28 15:37:36.429436
queueing from qatrack
Process-1:1 processing [lima-march-bacon-failed]
Running run_service_event_review_notices task at 2022-11-28 15:22:36.442877 for notices between 2022-11-28 15:22:36.442877 and 2022-11-28 15:37:36.442877
Processed [rugby-oven-west-mobile]
Processed [hotel-sodium-fourteen-triple]
Processed [lima-march-bacon-failed]

I am not sure if the "-failed" is an indicator of failure or just an identification of the process?

I also see that the Q cluster stops (which may be why I am not seeing the webpages?), but I though the following issues with the Matplotlib temporary file creation was a warning and is not a fatal error to cause the shutdown? Then it starts the Q cluster again and then it looks like it runs some notices for QA Track+ (like it is running), until I get to the final process and the appearance of the word failed, which may or may not be a true failure.

I will look at the static serving method you mention, but it may be more than I am willing to bit off at the moment.  As always, any help from the community is appreciated.

Cheers,

Isaac

To view this discussion on the web visit https://groups.google.com/d/msgid/qatrack/8c39c58f-0de0-447c-8709-bd3bcc8c4f38n%40googlegroups.com.