Another Post about mod_wsgi Logging in a Sad Environment

101 views
Skip to first unread message

Matt J

unread,
Aug 13, 2021, 6:01:01 PM8/13/21
to modwsgi
Hello All,

Firstly let me say that I have done my best over the past couple weeks to find a solution to this problem, but have come up with no concrete solutions, mostly because of limitations in my corporate environment. I've found many helpful resources such as this one https://www.youtube.com/watch?v=H6Q3l11fjU0, but am coming straight to the source for this question. 

How can I change the log location of mod_wsgi output (the app output) without defining my app under a separate virtual host?

I am hosting a Django Application using Python 3.7.10 on a RHEL7 server using Apache version 2.4 and mod_wsgi version 4.8.0. This server uses only a single virtual host to host multiple corporate applications (java, perl, python etc), and this server hosts them typically using proxies and locations under the single virtual host server01.corp.com:443. Because domain name is important and cannot be changed due to "its the way we have always done things", my django app must be hosted at server01.corp.com and aliased to a script path /apps/djangosite. Therefore, a conf.d for the site looks like this:
-----------------------------------------------------------------
<Location "/apps/djangosite ">
    #How all other apps are hosted on server01.corp.com using a proxy
    #ProxyPass "http://127.0.0.1:9889" retry=0
    #ProxyPassReverse "http://127.0.0.1:9889"
   <IF "req('Authorization') =~ /^Bearer/">
       #stuff
   </IF>
   <ELSE>
        #stuff but with ldap
   </ELSE>

    Require valid-user
    # Forward the REMOTE_USER env var as a request header
    RequestHeader set X-Remote-User %{REMOTE_USER}s
    RequestHeader set X-Remote-Host %{REMOTE_HOST}s

    #Add some variables for Django
    #RequestHeader set X-Script-Name /apps/djangosite/
    #RequestHeader set X-Forwarded-Script-Name /apps/djangosite/
</Location>

#mod_wsgi Specific Entries
<Directory /fullpath/mysite/static>
    Options FollowSymLinks Includes
    Require all granted
</Directory>

Alias /apps/djangosite/static/ /fullpath/mysite/static/

<Directory /fullpath/mysite>
    <Files wsgi.py>
         Require all granted
    </Files>
</Directory>

WSGIDaemonProcess djangosite-api processes=2 threads=15 python-home=/fullpath/envs/djangosite_api/ user=corpuser home=/fullpath/djangosite python-path=/fullpath:/fullpath:/fullpath
WSGIProcessGroup djangosite-api
WSGIScriptAlias /apps/djangosite /fullpath/mysite/wsgi.py process-group=compute-api
WSGIRestrictEmbedded On
WSGIApplicationGroup %{GLOBAL}

#pull mod_wsgi out of ssl_error_log
SetEnvIf Request_URI ^/apps/djangosite(/.*|$) djangosite-api
CustomLog /fullpath/logs/djangosite.debug.log combined env=djangosite-api
LogLevel debug wsgi:Trace8
-----------------------------------------------------------------

 Logging Directives in settings.py
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}',
            'style': '{',
        },
    },
    'handlers': {
        'file': {
            'level': 'DEBUG',
            #'class':'logging.StreamHandler'
            'class': 'logging.FileHandler',
            'filename': ' /fullpath/logs/djangosite.debug.log  '
        }
    },
    'root': {
        'handlers': ['file'],
        'level': 'DEBUG',
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'INFO',
            'propagate': True,
        }
    }
}

And so here is the problem: With over 10 other deployed apps on server01.corp.com, the ssl_error_log of this one virtual host is pretty quick to bury messages, and I want to be able to get my apps log in a specific location, rather then sifting through the hosts log. If I was able to create another host, this wouldnt be an issue obviously. I can get django debug info from the logging directive, and some very slight request logging from setting the custom log above, but I will never be able to get stack traces or wsgi startup info unless I go to the ssl_error_log. I have read the warnings on a portable application using stderr/stdout, and have tried to make the application as portable as the development environment will allow. I cannot find a way to change the log location of a specific module in apache. I tried gunicorn as a alternative to mod_wsgi, but mod_wsgi has a cleaner setup for us and apache/gunicorn struggled to host djangosite on anything other than the root /. 

Is there any way to circumvent this inability to create another virtualhost while still having full control of logging?

Thank you for your time,
Matt J

Graham Dumpleton

unread,
Aug 13, 2021, 6:13:18 PM8/13/21
to mod...@googlegroups.com
Since you tried gunicorn and your other sub sites do it, why not use Apache as a proxy to mod_wsgi-express?

Use of mod_wsgi-express was already mentioned in that video you linked. You can find some blog posts specifically about the topic of using it behind Apache as a proxy at:


Just ignore that it talks about the mod_wsgi-express running in docker as that doesn't need to be the case.

Finally, since part of the issue seems to relate to the mount point of the application, mod_wsgi-express does have a --mount-point option so that you can when it is behind the proxy still have it mounted as a sub path to make things potentially easier.

If you are concerned about performance, your application is robust and you don't need the protections that daemon mode of mod_wsgi-express offers for auto recovery (which you aren't using in your current manual Apache setup with daemon mode), you can set the backend mod_wsgi-express to use embedded mode using --embedded-mode. This will eliminate some of the extra latency of using a proxy to backend Apache to daemon mode.

Using mod_wsgi-express means that also not subject to using the older mod_wsgi version that RHEL ships with as installing it from PyPi instead.

So as first step perhaps get your application working under mod_wsgi-express and see if it works and ask questions about anything you aren't quite sure about or can't get working.

Also see:


Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/fc14abce-0f43-4e35-a91e-05310847947an%40googlegroups.com.

Matt J

unread,
Aug 16, 2021, 5:26:31 PM8/16/21
to modwsgi
Hi Graham,

Thank you for your response. I am now using mod_wsgi-express, and running into what I believe is a final issue - SSL errors. Please see this new configuration:

conf.d/djangosite.conf - it can be noted that most, but not all other apps on this server use localhost in the proxy pass, but I believe it should be the hostname for our case. Maybe you can correct me on this. 
<Location "/apps/onequeue">
   ProxyPass "http://server01.corp.com:9889" retry=0
   ProxyPassReverse "http://server01.corp.com:9889"
   <IF "req('Authorization') =~ /^Bearer/">
        stuff
   </IF>
   <ELSE>
        stuff but with LDAP
   </ELSE>

    Require valid-user
    RequestHeader set X-Remote-User %{REMOTE_USER}s
    RequestHeader set X-Remote-Host %{REMOTE_HOST}s
</Location>

conf/httpd.conf - only relevant items shown (save a bunch of location-specific ldap crap) 
# LoadModule foo_module modules/mod_foo.so
Include conf.modules.d/*.conf
#MOD_WSGI installed from SOURCE using source python3.7 with shared flags
LoadFile /deployment/tools/lib/libpython3.7m.so.1.0
#LoadModule wsgi_module modules/mod_wsgi.so

#MOD_WSGI-EXPRESS installed from PyEnv with mod_wsgi installed via PIP
LoadModule wsgi_module modules/mod_wsgi-py37.cpython-37m-x86_64-linux-gnu.so
WSGIPythonHome "/deployment/apps/envs/djangosite-api"

Include conf.d/*.conf

ServerName server01.corp.com:80

<VirtualHost *:80>
   Redirect permanent / https://server01.corp.com/
</VirtualHost>


conf.d/ssl.conf - All items shown (defaults may be commented)

Listen 443 https

SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog
SSLSessionCache         shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout  300

SSLRandomSeed startup file:/dev/urandom  256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin

<VirtualHost _default_:443>

ErrorLog logs/ssl_error_log
LogLevel warn
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
CustomLog "|/sbin/rotatelogs /var/log/httpd/access.%Y%m%d%H%M%S 10M" combined

SSLEngine on

SSLProtocol all -SSLv2 -SSLv3

SSLCipherSuite HIGH:3DES:!aNULL:!MD5:!SEED:!IDEA

SSLCertificateFile /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt

SSLCertificateKeyFile /etc/pki/tls/private/server.key

<Files ~ "\.(cgi|shtml|phtml|php3?)$">
    SSLOptions +StdEnvVars
</Files>
<Directory "/var/www/cgi-bin">
    SSLOptions +StdEnvVars
</Directory>

BrowserMatch "MSIE [2-5]" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0
</VirtualHost>



Steps Taken during Installation: 
export APXS=/usr/bin/apxs
pip install --proxy=http://proxy.corp.com:80 --trusted-host pypi.python.org --trusted-host files.pythonhosted.org mod_wsgi
/deployment/apps/envs/djangosite-api/bin/mod_wsgi-express install-module (output placed in above httpd.conf, I know this part is all fine)

Issue Occuring: 
Alas, we are at the command I am trying to use to test the server: (I have an identical variant for manage.py runmodwsgi that produces the same error).

setup @
mod_wsgi-express setup-server /deployment/source/djangosite/mysite/wsgi.py --reload-on-changes --port=9889 --https-port=443 --https-only --server-name=server01.corp.com --mount-point=/apps/djangosite --url-alias /apps/djangosite/static /deployment/source/djangosite/mysite/static/ --ssl-certificate-file=/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt --ssl-certificate-key-file=/etc/pki/tls/private/server.key --user=opc --group=cad --server-root=/deployment/source/djangosite/mysite/mod_wsgi-express
start server @
/deployment/source/mysite/mod_wsgi-express/apachectl start
command output:
(13)Permission denied: AH00072: make_sock: could not bind to address [::]:443
(13)Permission denied: AH00072: make_sock: could not bind to address 0.0.0.0:443
no listening sockets available, shutting down
AH00015: Unable to open logs

In this case it cannot use 443.
 
setup again @ (without specify --https-port and --https-only)
mod_wsgi-express setup-server /deployment/source/djangosite/mysite/wsgi.py --reload-on-changes --port=9889 --server-name server01.corp.com --mount-point=/apps/djangosite --url-alias /apps/djangosite/static /deployment/source/djangosite/mysite/static/ --ssl-certificate-file=/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt --ssl-certificate-key-file=/etc/pki/tls/private/server.key --user opc --group cad --server-root=/deployment/source/djangosite/mysite/mod_wsgi-express  --startup-log --access-log
start server @
/deployment/source/mysite/mod_wsgi-express/apachectl start
web browser output: (chrome)
[URL CHANGED BY BROWSER] 

server01.corp.com sent an invalid response. ERR_SSL_PROTOCOL_ERROR

In this case, I get an ambiguous ssl error. I do know that the browser changing my URL can be a proxy issue, but am unsure how to solve it in this case. 

So here my question is, how/what am I doing wrong to serve my app using mod_wsgi-express to upgrade insecure requests and use https behind an apache proxy?

As a side note, being able to use the apachectl start and stop with systemd will be a blessing. I'm not sure exactly how you are able to recreate/serve an apache module outside of apache like this but its very cool, very smart, and I cant wait to show my friends it working. 
Along the way, I used these resources:

https://pypi.org/project/mod-wsgi/
https://groups.google.com/g/modwsgi/c/nVNtX4zJGBY
https://github.com/GrahamDumpleton/mod_wsgi/issues/411
Aforementioned blog posts



Thank you for your time,

Matthew Johnson

Graham Dumpleton

unread,
Aug 16, 2021, 5:49:17 PM8/16/21
to mod...@googlegroups.com
See comments below.

On 17 Aug 2021, at 7:26 am, Matt J <matthew.ja...@outlook.com> wrote:

Hi Graham,

Thank you for your response. I am now using mod_wsgi-express, and running into what I believe is a final issue - SSL errors. Please see this new configuration:

conf.d/djangosite.conf - it can be noted that most, but not all other apps on this server use localhost in the proxy pass, but I believe it should be the hostname for our case. Maybe you can correct me on this. 
<Location "/apps/onequeue">
   ProxyPass "http://server01.corp.com:9889" retry=0
   ProxyPassReverse "http://server01.corp.com:9889"

Note that based on this you are proxying to backend as HTTP.

   <IF "req('Authorization') =~ /^Bearer/">
        stuff
   </IF>
   <ELSE>
        stuff but with LDAP
   </ELSE>

    Require valid-user
    RequestHeader set X-Remote-User %{REMOTE_USER}s
    RequestHeader set X-Remote-Host %{REMOTE_HOST}s
</Location>

conf/httpd.conf - only relevant items shown (save a bunch of location-specific ldap crap) 
# LoadModule foo_module modules/mod_foo.so
Include conf.modules.d/*.conf
#MOD_WSGI installed from SOURCE using source python3.7 with shared flags
LoadFile /deployment/tools/lib/libpython3.7m.so.1.0

You don't need this LoadFile line.

#LoadModule wsgi_module modules/mod_wsgi.so

#MOD_WSGI-EXPRESS installed from PyEnv with mod_wsgi installed via PIP
LoadModule wsgi_module modules/mod_wsgi-py37.cpython-37m-x86_64-linux-gnu.so
WSGIPythonHome "/deployment/apps/envs/djangosite-api"

You don't need to include mod_wsgi in the front end Apache anymore if the only thing using mod_wsgi is now as a backend using mod_wsgi-express.
You don't need to run 'mod_wsgi-express install-module' if using mod_wsgi-express as backend. When mod_wsgi-express start-server is run it will use version of mod_wsgi.so file installed in the Python virtual environment.

In other words, there doesn't need to be anything in the front end Apache for mod_wsgi, only the config to set it up as a proxy which are just Apache directives.


Issue Occuring: 
Alas, we are at the command I am trying to use to test the server: (I have an identical variant for manage.py runmodwsgi that produces the same error).

setup @
mod_wsgi-express setup-server /deployment/source/djangosite/mysite/wsgi.py --reload-on-changes --port=9889 --https-port=443 --https-only --server-name=server01.corp.com --mount-point=/apps/djangosite --url-alias /apps/djangosite/static /deployment/source/djangosite/mysite/static/ --ssl-certificate-file=/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt --ssl-certificate-key-file=/etc/pki/tls/private/server.key --user=opc --group=cad --server-root=/deployment/source/djangosite/mysite/mod_wsgi-express

You are only proxying HTTP, not HTTPS, the backend doesn't need: --https-port=443 --https-only --server-name=server01.corp.com --ssl-certificate-file=/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt --ssl-certificate-key-file=/etc/pki/tls/private/server.key

Your proxy is for /apps/onequeue but you are using --mount-point=/apps/djangosite which doesn't match.

You are not using any of the options talked about in the two blog posts about using mod_wsgi-express behind a proxy.

start server @
/deployment/source/mysite/mod_wsgi-express/apachectl start
command output:
(13)Permission denied: AH00072: make_sock: could not bind to address [::]:443
(13)Permission denied: AH00072: make_sock: could not bind to address 0.0.0.0:443
no listening sockets available, shutting down
AH00015: Unable to open logs

In this case it cannot use 443.

Correct.

 
setup again @ (without specify --https-port and --https-only)
mod_wsgi-express setup-server /deployment/source/djangosite/mysite/wsgi.py --reload-on-changes --port=9889 --server-name server01.corp.com --mount-point=/apps/djangosite --url-alias /apps/djangosite/static /deployment/source/djangosite/mysite/static/ --ssl-certificate-file=/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt --ssl-certificate-key-file=/etc/pki/tls/private/server.key --user opc --group cad --server-root=/deployment/source/djangosite/mysite/mod_wsgi-express  --startup-log --access-log

As above, you don't need: --server-name server01.corp.com -ssl-certificate-file=/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt --ssl-certificate-key-file=/etc/pki/tls/private/server.key

start server @
/deployment/source/mysite/mod_wsgi-express/apachectl start
web browser output: (chrome)
[URL CHANGED BY BROWSER] 


This the fact you are seeing the backend port is probably because you are missing all the special options to mod_wsgi-express when using proxying mentioned in:


It is very important you have the options as to which proxy headers to trust. There is a chance your Django project code needs to change if it doesn't do URL reconstruction properly with functions/template tags supplied by Django for the purpose.

server01.corp.com sent an invalid response. ERR_SSL_PROTOCOL_ERROR

If the backend is running on the same machine and not a separate machine on a private network, we will also need to tell mod_wsgi-express to only listen on 127.0.0.1 and the proxy will need to use http://127.0.0.1:9889 and not http://server01.corp.com:9889 as the backend shouldn't be directly contactable by the browser. I will need to double check what needs to be done to ensure it only listens for connections on 127.0.0.1.

In this case, I get an ambiguous ssl error. I do know that the browser changing my URL can be a proxy issue, but am unsure how to solve it in this case. 

So here my question is, how/what am I doing wrong to serve my app using mod_wsgi-express to upgrade insecure requests and use https behind an apache proxy?

First up you need those special options to mod_wsgi-express saying what proxy headers you need to trust per the blog posts.


As a side note, being able to use the apachectl start and stop with systemd will be a blessing. I'm not sure exactly how you are able to recreate/serve an apache module outside of apache like this but its very cool, very smart, and I cant wait to show my friends it working. 

Am not a systemd expert, but you should be able to set up systemd with a config that or start runs:

/deployment/source/mysite/mod_wsgi-express/apachectl start

and for stop runs:

/deployment/source/mysite/mod_wsgi-express/apachectl stop

Matt J

unread,
Aug 18, 2021, 11:54:32 AM8/18/21
to modwsgi
Hi Graham, 

Ahh, my mistake, missed those two article about proxying. This now works as expected. On typical apachectl start and stop the application runs as expected. Hoewever, I cant seem to have the service start correctly from systemd. Interestingly, it reports the same information as a manual start, but chooses to die. I know you said you are not a systemd expert, but would you know what is causing mod_wsgi-express to deregister itself immediately?

SETUP COMMAND:
/my/envs/compute-api-prod/bin/python3.7 manage.py runmodwsgi --setup-only --reload-on-changes --port=9889 --user opc --group faim --document-root /prod-path/compute/mysite/ --server-root=/var/run/compute-express --startup-log --access-log --log-level debug

NORMAL START & LOG:
/var/run/compute-express/apachectl start
&
[Wed Aug 18 09:41:26.979537 2021] [wsgi:debug] [pid 18790:tid 140170992916608] src/server/mod_wsgi.c(8512): mod_wsgi (pid=18790): Socket for 'localhost:9889' is '/var/run/compute-express/wsgi.18790.u43809.1.sock'.
[Wed Aug 18 09:41:26.980273 2021] [wsgi:debug] [pid 18790:tid 140170992916608] src/server/mod_wsgi.c(8581): mod_wsgi (pid=18790): Listen backlog for socket '/var/run/compute-express/wsgi.18790.u43809.1.sock' is '100'.
[Wed Aug 18 09:41:26.980979 2021] [wsgi:info] [pid 18793:tid 140170992916608] mod_wsgi (pid=18793): Starting process 'localhost:9889' with threads=5.
[Wed Aug 18 09:41:26.981185 2021] [mpm_event:notice] [pid 18790:tid 140170992916608] AH00489: Apache/2.4.6 (Red Hat Enterprise Linux) mod_wsgi/4.9.0 Python/3.7 configured -- resuming normal operations
[Wed Aug 18 09:41:26.981208 2021] [mpm_event:info] [pid 18790:tid 140170992916608] AH00490: Server built: Oct  8 2020 21:27:40
[Wed Aug 18 09:41:26.981242 2021] [core:notice] [pid 18790:tid 140170992916608] AH00094: Command line: 'httpd (mod_wsgi-express)    -f /var/run/compute-express/httpd.conf -E /var/run/compute-express/startup_log -D MOD_WSGI_ACCESS_LOG -D MOD_WSGI_KEEP_ALIVE -D MOD_WSGI_MPM_ENABLE_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_WORKER_MODULE -D MOD_WSGI_MPM_EXISTS_PREFORK_MODULE'
[Wed Aug 18 09:41:26.981359 2021] [wsgi:debug] [pid 18793:tid 140170992916608] src/server/mod_wsgi.c(10262): mod_wsgi (pid=18793): Setting lang to en_US.ISO8859-1 for daemon process group localhost:9889.
[Wed Aug 18 09:41:26.981421 2021] [wsgi:debug] [pid 18793:tid 140170992916608] src/server/mod_wsgi.c(10275): mod_wsgi (pid=18793): Setting locale to en_US.ISO8859-1 for daemon process group localhost:9889.
[Wed Aug 18 09:41:26.981650 2021] [wsgi:info] [pid 18793:tid 140170992916608] mod_wsgi (pid=18793): Python home /usr/wwwprod/tapeout/envs/compute-api-prod.
[Wed Aug 18 09:41:26.982282 2021] [wsgi:info] [pid 18793:tid 140170992916608] mod_wsgi (pid=18793): Initializing Python.
[Wed Aug 18 09:41:27.045970 2021] [wsgi:info] [pid 18793:tid 140170992916608] mod_wsgi (pid=18793): Attach interpreter ''.
[Wed Aug 18 09:41:27.120445 2021] [wsgi:info] [pid 18793:tid 140170992916608] mod_wsgi (pid=18793): Imported 'mod_wsgi'.
[Wed Aug 18 09:41:27.120545 2021] [wsgi:info] [pid 18793:tid 140170992916608] mod_wsgi (pid=18793, process='localhost:9889', application=''): Loading Python script file '/var/run/compute-express/handler.wsgi'.
[Wed Aug 18 15:41:31.154269 2021] [wsgi:error] [pid 18793:tid 140170992916608] monitor (pid=18793): Starting change monitor.
[Wed Aug 18 15:41:31.155924 2021] [wsgi:debug] [pid 18793:tid 140170611386112] src/server/mod_wsgi.c(9142): mod_wsgi (pid=18793): Started thread 0 in daemon process 'localhost:9889'.
[Wed Aug 18 15:41:31.155990 2021] [wsgi:debug] [pid 18793:tid 140170602993408] src/server/mod_wsgi.c(9142): mod_wsgi (pid=18793): Started thread 1 in daemon process 'localhost:9889'.
[Wed Aug 18 15:41:31.156039 2021] [wsgi:debug] [pid 18793:tid 140170594600704] src/server/mod_wsgi.c(9142): mod_wsgi (pid=18793): Started thread 2 in daemon process 'localhost:9889'.
[Wed Aug 18 15:41:31.156122 2021] [wsgi:debug] [pid 18793:tid 140170586208000] src/server/mod_wsgi.c(9142): mod_wsgi (pid=18793): Started thread 3 in daemon process 'localhost:9889'.
[Wed Aug 18 15:41:31.156174 2021] [wsgi:debug] [pid 18793:tid 140170577815296] src/server/mod_wsgi.c(9142): mod_wsgi (pid=18793): Started thread 4 in daemon process 'localhost:9889'.

SYSTEMD SERVICE:
[Unit]
Description = OneQueue mod_wsgi-express Server
Wants=multi-user.target
After=multi-user.target

[Service]
PermissionsStartOnly = true
PIDFile = /run/compute-api.pid

User = opc
WorkingDirectory = /prod-path/compute/
RuntimeDirectory=compute-api

Environment="DJANGO_SETTINGS_MODULE=mysite.settings.webapi"
Environment="PYTHONPATH=/path/python/lib:/path/python:/prod-path/"

ExecStart =/usr/bin/bash /var/run/compute-express/apachectl start
ExecReload =/usr/bin/bash /var/run/compute-express/apachectl stop;/usr/bin/bash /var/run/compute-express/apachectl start
ExecStop =/usr/bin/bash /var/run/compute-express/apachectl stop
Restart=no


PrivateTmp = true

[Install]
WantedBy = multi-user.target

SYSTEMD START & LOG:
[Wed Aug 18 09:40:08.957672 2021] [wsgi:debug] [pid 25040:tid 140552998975616] src/server/mod_wsgi.c(8512): mod_wsgi (pid=25040): Socket for 'localhost:9889' is '/var/run/compute-express/wsgi.25040.u43809.1.sock'.
[Wed Aug 18 09:40:08.959159 2021] [wsgi:debug] [pid 25040:tid 140552998975616] src/server/mod_wsgi.c(8581): mod_wsgi (pid=25040): Listen backlog for socket '/var/run/compute-express/wsgi.25040.u43809.1.sock' is '100'.
[Wed Aug 18 09:40:08.959714 2021] [mpm_event:notice] [pid 25040:tid 140552998975616] AH00489: Apache/2.4.6 (Red Hat Enterprise Linux) mod_wsgi/4.9.0 Python/3.7 configured -- resuming normal operations
[Wed Aug 18 09:40:08.959730 2021] [mpm_event:info] [pid 25040:tid 140552998975616] AH00490: Server built: Oct  8 2020 21:27:40
[Wed Aug 18 09:40:08.959749 2021] [core:notice] [pid 25040:tid 140552998975616] AH00094: Command line: 'httpd (mod_wsgi-express)    -f /var/run/compute-express/httpd.conf -E /var/run/compute-express/startup_log -D MOD_WSGI_ACCESS_LOG -D MOD_WSGI_KEEP_ALIVE -D MOD_WSGI_MPM_ENABLE_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_WORKER_MODULE -D MOD_WSGI_MPM_EXISTS_PREFORK_MODULE'
[Wed Aug 18 09:40:08.959720 2021] [wsgi:info] [pid 25049:tid 140552998975616] mod_wsgi (pid=25049): Starting process 'localhost:9889' with threads=5.
[Wed Aug 18 09:40:08.960082 2021] [wsgi:debug] [pid 25049:tid 140552998975616] src/server/mod_wsgi.c(10262): mod_wsgi (pid=25049): Setting lang to en_US.ISO8859-1 for daemon process group localhost:9889.
[Wed Aug 18 09:40:08.960133 2021] [wsgi:debug] [pid 25049:tid 140552998975616] src/server/mod_wsgi.c(10275): mod_wsgi (pid=25049): Setting locale to en_US.ISO8859-1 for daemon process group localhost:9889.
[Wed Aug 18 09:40:08.960359 2021] [wsgi:info] [pid 25049:tid 140552998975616] mod_wsgi (pid=25049): Python home /usr/wwwprod/tapeout/envs/compute-api-prod.
[Wed Aug 18 09:40:08.960988 2021] [wsgi:info] [pid 25049:tid 140552998975616] mod_wsgi (pid=25049): Initializing Python.
[Wed Aug 18 09:40:08.999896 2021] [wsgi:info] [pid 25049:tid 140552998975616] mod_wsgi (pid=25049): Attach interpreter ''.
[Wed Aug 18 09:40:09.041311 2021] [wsgi:info] [pid 25049:tid 140552998975616] mod_wsgi (pid=25049): Imported 'mod_wsgi'.
[Wed Aug 18 09:40:09.041414 2021] [wsgi:info] [pid 25049:tid 140552998975616] mod_wsgi (pid=25049, process='localhost:9889', application=''): Loading Python script file '/var/run/compute-express/handler.wsgi'.
[Wed Aug 18 09:40:09.073390 2021] [wsgi:info] [pid 25040:tid 140552998975616] mod_wsgi (pid=25049): Process 'localhost:9889' to be deregistered, as server is restarting or being shutdown.
[Wed Aug 18 09:40:09.073439 2021] [wsgi:info] [pid 25040:tid 140552998975616] mod_wsgi (pid=25049): Process 'localhost:9889' has been deregistered and will no longer be monitored.
[Wed Aug 18 09:40:09.078388 2021] [core:info] [pid 25040:tid 140552998975616] AH00096: removed PID file /var/run/compute-express/httpd.pid (pid=25040)
[Wed Aug 18 09:40:09.078407 2021] [mpm_event:notice] [pid 25040:tid 140552998975616] AH00491: caught SIGTERM, shutting down
Broken pipe


I am not too worried about getting this going as I do have the apachectl itself, but I feel it has to be pretty close... If you happen to know the issue you here i'll add another coffee to my donation :)

Thanks,
Matt  

Graham Dumpleton

unread,
Aug 18, 2021, 6:19:40 PM8/18/21
to mod...@googlegroups.com
Try:

ExecReload =/usr/bin/bash /var/run/compute-express/apachectl restart

Commands line start/stop will happen in background so running them too close in time possibly means that the start was replacing the process ID file before stop was run and so is shutting down new instance and not the old one.

On 19 Aug 2021, at 1:54 am, Matt J <matthew.ja...@outlook.com> wrote:

Hi Graham, 

Ahh, my mistake, missed those two article about proxying. This now works as expected. On typical apachectl start and stop the application runs as expected. Hoewever, I cant seem to have the service start correctly from systemd. Interestingly, it reports the same information as a manual start, but chooses to die. I know you said you are not a systemd expert, but would you know what is causing mod_wsgi-express to deregister itself immediately?

SETUP COMMAND:
/my/envs/compute-api-prod/bin/python3.7 manage.py runmodwsgi --setup-only --reload-on-changes --port=9889 --user opc --group faim --document-root /prod-path/compute/mysite/ --server-root=/var/run/compute-express --startup-log --access-log --log-level debug

NORMAL START & LOG:
/var/run/compute-express/apachectl start
&

You didn't need the & to run it in background as will already run in background.

Matt J

unread,
Aug 19, 2021, 11:00:57 AM8/19/21
to modwsgi
Graham,

The same behavior seems to exist with the restart option. It should also be noted that the log above resulted from calling systemd's start for the service. Although, in this log trace using 'restart', it did create the daemon process before immediately exiting. Either way in both cases, it exits itself. 
I was not running the process in the background using &, just a poor choice of syntax to separate the command & logs.

ExecReload =/usr/bin/bash /var/run/compute-express/apachectl restart
produces ->
[Thu Aug 19 08:52:29.209357 2021] [wsgi:debug] [pid 28348:tid 140134250059904] src/server/mod_wsgi.c(8512): mod_wsgi (pid=28348): Socket for 'localhost:9889' is '/var/run/compute-express/wsgi.28348.u43809.1.sock'.
[Thu Aug 19 08:52:29.210213 2021] [wsgi:debug] [pid 28348:tid 140134250059904] src/server/mod_wsgi.c(8581): mod_wsgi (pid=28348): Listen backlog for socket '/var/run/compute-express/wsgi.28348.u43809.1.sock' is '100'.
[Thu Aug 19 08:52:29.210965 2021] [wsgi:info] [pid 28354:tid 140134250059904] mod_wsgi (pid=28354): Starting process 'localhost:9889' with threads=5.
[Thu Aug 19 08:52:29.211192 2021] [mpm_event:notice] [pid 28348:tid 140134250059904] AH00489: Apache/2.4.6 (Red Hat Enterprise Linux) mod_wsgi/4.9.0 Python/3.7 configured -- resuming normal operations
[Thu Aug 19 08:52:29.211217 2021] [mpm_event:info] [pid 28348:tid 140134250059904] AH00490: Server built: Oct  8 2020 21:27:40
[Thu Aug 19 08:52:29.211251 2021] [core:notice] [pid 28348:tid 140134250059904] AH00094: Command line: 'httpd (mod_wsgi-express)    -f /var/run/compute-express/httpd.conf -E /var/run/compute-express/startup_log -D MOD_WSGI_ACCESS_LOG -D MOD_WSGI_KEEP_ALIVE -D MOD_WSGI_MPM_ENABLE_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_WORKER_MODULE -D MOD_WSGI_MPM_EXISTS_PREFORK_MODULE'
[Thu Aug 19 08:52:29.211382 2021] [wsgi:debug] [pid 28354:tid 140134250059904] src/server/mod_wsgi.c(10262): mod_wsgi (pid=28354): Setting lang to en_US.ISO8859-1 for daemon process group localhost:9889.
[Thu Aug 19 08:52:29.211437 2021] [wsgi:debug] [pid 28354:tid 140134250059904] src/server/mod_wsgi.c(10275): mod_wsgi (pid=28354): Setting locale to en_US.ISO8859-1 for daemon process group localhost:9889.
[Thu Aug 19 08:52:29.211666 2021] [wsgi:info] [pid 28354:tid 140134250059904] mod_wsgi (pid=28354): Python home /usr/wwwprod/tapeout/envs/compute-api-prod.
[Thu Aug 19 08:52:29.212351 2021] [wsgi:info] [pid 28354:tid 140134250059904] mod_wsgi (pid=28354): Initializing Python.
[Thu Aug 19 08:52:29.270992 2021] [wsgi:info] [pid 28354:tid 140134250059904] mod_wsgi (pid=28354): Attach interpreter ''.
[Thu Aug 19 08:52:29.324631 2021] [wsgi:info] [pid 28348:tid 140134250059904] mod_wsgi (pid=28354): Process 'localhost:9889' to be deregistered, as server is restarting or being shutdown.
[Thu Aug 19 08:52:29.324654 2021] [wsgi:info] [pid 28348:tid 140134250059904] mod_wsgi (pid=28354): Process 'localhost:9889' has been deregistered and will no longer be monitored.
[Thu Aug 19 08:52:29.325246 2021] [core:info] [pid 28348:tid 140134250059904] AH00096: removed PID file /var/run/compute-express/httpd.pid (pid=28348)
[Thu Aug 19 08:52:29.325255 2021] [mpm_event:notice] [pid 28348:tid 140134250059904] AH00491: caught SIGTERM, shutting down
[Thu Aug 19 08:52:29.379217 2021] [wsgi:info] [pid 28354:tid 140134250059904] mod_wsgi (pid=28354): Imported 'mod_wsgi'.
[Thu Aug 19 08:52:29.379526 2021] [wsgi:debug] [pid 28354:tid 140134026278656] src/server/mod_wsgi.c(9142): mod_wsgi (pid=28354): Started thread 0 in daemon process 'localhost:9889'.
[Thu Aug 19 08:52:29.379561 2021] [wsgi:debug] [pid 28354:tid 140134026278656] src/server/mod_wsgi.c(9123): mod_wsgi (pid=28354): Exiting thread 0 in daemon process 'localhost:9889'.
[Thu Aug 19 08:52:29.379580 2021] [wsgi:debug] [pid 28354:tid 140134017885952] src/server/mod_wsgi.c(9142): mod_wsgi (pid=28354): Started thread 1 in daemon process 'localhost:9889'.
[Thu Aug 19 08:52:29.379606 2021] [wsgi:debug] [pid 28354:tid 140134017885952] src/server/mod_wsgi.c(9123): mod_wsgi (pid=28354): Exiting thread 1 in daemon process 'localhost:9889'.
[Thu Aug 19 08:52:29.379647 2021] [wsgi:debug] [pid 28354:tid 140134009493248] src/server/mod_wsgi.c(9142): mod_wsgi (pid=28354): Started thread 2 in daemon process 'localhost:9889'.
[Thu Aug 19 08:52:29.379679 2021] [wsgi:debug] [pid 28354:tid 140134009493248] src/server/mod_wsgi.c(9123): mod_wsgi (pid=28354): Exiting thread 2 in daemon process 'localhost:9889'.
[Thu Aug 19 08:52:29.379679 2021] [wsgi:debug] [pid 28354:tid 140134001100544] src/server/mod_wsgi.c(9142): mod_wsgi (pid=28354): Started thread 3 in daemon process 'localhost:9889'.
[Thu Aug 19 08:52:29.379693 2021] [wsgi:debug] [pid 28354:tid 140134001100544] src/server/mod_wsgi.c(9123): mod_wsgi (pid=28354): Exiting thread 3 in daemon process 'localhost:9889'.
[Thu Aug 19 08:52:29.379738 2021] [wsgi:debug] [pid 28354:tid 140133992605440] src/server/mod_wsgi.c(9142): mod_wsgi (pid=28354): Started thread 4 in daemon process 'localhost:9889'.
[Thu Aug 19 08:52:29.379767 2021] [wsgi:debug] [pid 28354:tid 140133992605440] src/server/mod_wsgi.c(9123): mod_wsgi (pid=28354): Exiting thread 4 in daemon process 'localhost:9889'.
[Thu Aug 19 08:52:29.379795 2021] [wsgi:info] [pid 28354:tid 140134250059904] mod_wsgi (pid=28354): Shutdown requested 'localhost:9889'.
[Thu Aug 19 08:52:29.380348 2021] [wsgi:info] [pid 28354:tid 140134250059904] mod_wsgi (pid=28354): Stopping process 'localhost:9889'.
[Thu Aug 19 08:52:29.380361 2021] [wsgi:info] [pid 28354:tid 140134250059904] mod_wsgi (pid=28354): Destroying interpreters.
[Thu Aug 19 08:52:29.380371 2021] [wsgi:info] [pid 28354:tid 140134250059904] mod_wsgi (pid=28354): Cleanup interpreter ''.
[Thu Aug 19 08:52:29.380385 2021] [wsgi:info] [pid 28354:tid 140134250059904] mod_wsgi (pid=28354): Terminating Python.
[Thu Aug 19 08:52:29.385102 2021] [wsgi:info] [pid 28354:tid 140134250059904] mod_wsgi (pid=28354): Python has shutdown.
[Thu Aug 19 08:52:29.385118 2021] [wsgi:info] [pid 28354:tid 140134250059904] mod_wsgi (pid=28354): Exiting process 'localhost:9889'.

Thanks,
Matt 

Matt J

unread,
Aug 19, 2021, 11:40:07 AM8/19/21
to modwsgi
Graham,

Let me answer my own question this time :) Maybe this could be added to a future article for other users to enjoy.

The service must be of Type=oneshot. This is a slight variation of the default service type 'simple' as it starts follow up units after the main process exists, the apachectl. The default assumes apachectl is the main process which is obviously not the case. Services of type oneshot should be followed up usually with RemainAfterExit= such that the process is not assumed dead. 

Full Service:
[Unit]
Description = OneQueue mod_wsgi-express Server
Wants=multi-user.target
After=multi-user.target

[Service]
Type=oneshot
RemainAfterExit=yes
PIDFile = /run/compute-api.pid

User = opc
Group = faim
WorkingDirectory = /opc1/proteus/projects/scripts/SRC/PACKAGE_LIBS/python/compute/
RuntimeDirectory=compute-api

Environment="DJANGO_SETTINGS_MODULE=mysite.settings.webapi"
Environment="PYTHONPATH=/prod/python/lib:/sw/python/lib:/eda/python:/SRC/python"

ExecStart =/usr/bin/bash /var/run/compute-express/apachectl start
ExecReload =/usr/bin/bash /var/run/compute-express/apachectl restart
ExecStop =/usr/bin/bash /var/run/compute-express/apachectl stop
Restart=no
PrivateTmp = true

[Install]
WantedBy = multi-user.target

Thank you for all your help. I found both the default mod_wsgi and mod_wsgi-express to not only be great utilities but superbly supported and documented (although mod_wsgi-express is really itching for a single source of documentation like mod_wsgi's homespace). I look forward to monitoring this project and recommending it to others. 

Many thanks,
Matt Johnson 


Graham Dumpleton

unread,
Aug 19, 2021, 8:40:46 PM8/19/21
to mod...@googlegroups.com
I figured it may be the issue with the difference in two operating modes of systemd but I didn't know offhand the options to set which was used so wasn't sure what to look for. Anyway, great that you worked it out.

Reply all
Reply to author
Forward
0 new messages