client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs

[Lou King]

Everything was running along fine, then all of a sudden I received complaints that people were getting 403 forbidden from my server. 

I looked in all the /var/log, server logs, etc., but didn't think to look at /tmp/mod_wsgi-localhost:8001:0/error_log until after restarting the service (I'm using systemd). I knew the 403 errors started at 8:21 Jan 21 by viewing the virtual host's request log, but after restarting the service found the following in the mod_wsgi log:

[Mon Jan 20 21:53:13.249739 2020] [wsgi:error] [pid 20392:tid 140270704191232] [remote 198.199.82.162:35964] [2020-01-20 21:53:13,249] INFO in __init__: 66.249.79.119: GET http://routes.loutilities.com/robots.txt/routes 403
[Mon Jan 20 21:53:13.549214 2020] [wsgi:error] [pid 20392:tid 140270695798528] [remote 198.199.82.162:35968] [2020-01-20 21:53:13,549] INFO in __init__: 66.249.79.121: GET http://routes.loutilities.com/fsrc/routes 200
[Tue Jan 21 04:47:53.399163 2020] [wsgi:error] [pid 20392:tid 140270704191232] [remote 198.199.82.162:42670] [2020-01-21 04:47:53,398] INFO in __init__: 173.252.87.47: GET http://routes.loutilities.com/ 302
[Tue Jan 21 04:47:55.573795 2020] [wsgi:error] [pid 20392:tid 140270695798528] [remote 198.199.82.162:42674] [2020-01-21 04:47:55,573] INFO in __init__: 173.252.95.26: GET http://routes.loutilities.com/ 302
[Tue Jan 21 08:21:05.017486 2020] [authz_core:error] [pid 20393:tid 140271178082048] [client 198.199.82.162:49262] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https://steeplechasers.org/training/running-routes/
[Tue Jan 21 08:21:09.020530 2020] [authz_core:error] [pid 20420:tid 140271178348288] [client 198.199.82.162:49264] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https://steeplechasers.org/training/running-routes/
[Tue Jan 21 08:21:09.952862 2020] [authz_core:error] [pid 20393:tid 140271099369216] [client 198.199.82.162:49266] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https://steeplechasers.org/training/running-routes/
[Tue Jan 21 08:21:10.811980 2020] [authz_core:error] [pid 20420:tid 140271178082048] [client 198.199.82.162:49268] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https://steeplechasers.org/training/running-routes/
[Tue Jan 21 08:21:19.475125 2020] [authz_core:error] [pid 20393:tid 140271177549568] [client 198.199.82.162:49270] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https://steeplechasers.org/training/running-routes/
[Tue Jan 21 08:21:23.131160 2020] [authz_core:error] [pid 20420:tid 140271178614528] [client 198.199.82.162:49272] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https://steeplechasers.org/training/running-routes/
[Tue Jan 21 08:21:24.440572 2020] [authz_core:error] [pid 20420:tid 140271177549568] [client 198.199.82.162:49276] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https://steeplechasers.org/training/running-routes/
[Tue Jan 21 08:23:38.045572 2020] [authz_core:error] [pid 20420:tid 140271099369216] [client 198.199.82.162:49322] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs
[Tue Jan 21 08:36:40.705930 2020] [authz_core:error] [pid 20393:tid 140271098836736] [client 198.199.82.162:49516] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https://m.facebook.com/
[Tue Jan 21 08:36:41.109523 2020] [authz_core:error] [pid 20393:tid 140271177283328] [client 198.199.82.162:49518] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https://routes.loutilities.com/fsrc/routes
[Tue Jan 21 08:47:53.231357 2020] [authz_core:error] [pid 20420:tid 140271098836736] [client 198.199.82.162:49730] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs

i.e., everything running along fine until the request 8:21. 

As mentioned I didn't look into this log until after restarting the service, so I didn't get a chance to inspect the /tmp/mod_wsgi-localhost:8001:0/htdocs directory to see what the issue was.

I did systemctl status on the service before the restart and all looked well. 

[lking@loutility-server-digitalocean ~]$ sudo systemctl status vhost-routetility-sandbox.service
● vhost-routetility-sandbox.service - start routetility www proxy server
   Loaded: loaded (/etc/systemd/system/vhost-routetility-sandbox.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2020-01-20 11:53:57 EST; 23h ago
 Main PID: 19331 (start-wsgi-serv)
   CGroup: /system.slice/vhost-routetility-sandbox.service
           ├─19331 /bin/bash /root/bin/start-wsgi-server runningroutes sandbox.routes.loutilities.com routesmgr routesmgr 8002
           ├─19332 httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost:8002:0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E...
           ├─19335 httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost:8002:0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E...
           ├─19978 httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost:8002:0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E...
           └─19979 (wsgi:localhost:8002:0)  -f /tmp/mod_wsgi-localhost:8002:0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E...


Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Server Conf        : /tmp/mod_wsgi-localhost:8002:0/ht...onf
Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Error Log File     : /tmp/mod_wsgi-localhost:8002:0/er...rn)
Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Request Capacity   : 5 (1 process * 5 threads)
Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Request Timeout    : 60 (seconds)
Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Startup Timeout    : 15 (seconds)
Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Queue Backlog      : 100 (connections)
Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Queue Timeout      : 45 (seconds)
Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Server Capacity    : 20 (event/worker), 20 (prefork)
Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Server Backlog     : 500 (connections)
Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Locale Setting     : en_US.UTF-8
Hint: Some lines were ellipsized, use -l to show in full.

Have you heard of this before? I will definitely check /tmp/mod_wsgi-localhost:8001:0/htdocs directory if this happens again.

[Lou King]

Oops, posted the wrong servics log, sorry it should have been

[lking@loutility-server-digitalocean ~]$ sudo systemctl status vhost-routetility-www.service
● vhost-routetility-www.service - start routetility www proxy server
   Loaded: loaded (/etc/systemd/system/vhost-routetility-www.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2020-01-20 12:11:29 EST; 23h ago
 Main PID: 20389 (vhost-routetili)
   CGroup: /system.slice/vhost-routetility-www.service
           ├─20389 /bin/bash /root/bin/vhost-routetility-www
           ├─20390 httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost:8001:0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E...
           ├─20392 (wsgi:localhost:8001:0)  -f /tmp/mod_wsgi-localhost:8001:0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E...
           ├─20393 httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost:8001:0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E...
           └─20420 httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost:8001:0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E...


Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389]: Server Conf        : /tmp/mod_wsgi-localhost:8001:...onf
Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389]: Error Log File     : /tmp/mod_wsgi-localhost:8001:...rn)
Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389]: Request Capacity   : 5 (1 process * 5 threads)
Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389]: Request Timeout    : 60 (seconds)
Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389]: Startup Timeout    : 15 (seconds)
Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389]: Queue Backlog      : 100 (connections)
Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389]: Queue Timeout      : 45 (seconds)
Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389]: Server Capacity    : 20 (event/worker), 20 (prefork)
Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389]: Server Backlog     : 500 (connections)
Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389]: Locale Setting     : en_US.UTF-8

Hint: Some lines were ellipsized, use -l to show in full.

[Lou King]

and for completeness

[lking@loutility-server-digitalocean applogs]$ sudo cat /root/bin/vhost-routetility-www
#!/bin/bash
source /var/www/www.routes.loutilities.com/venv/bin/activate
mod_wsgi-express start-server --server-name proxysvr.loutilities.com --port 8001 --user routesmgr --group routesmgr /var/www/www.routes.loutilities.com/runningroutes/runningroutes/runningroutes.wsgi --working-directory /var/www/www.routes.loutilities.com/runningroutes/runningroutes/
deactivate

[Graham Dumpleton]

If using mod_wsgi-express as a system service, ensure you set —server-root so that files aren’t placed in /tmp. Using /tmp as a persistent location can fail if the Linux system implements a cron job that removes stuff under /tmp if older than a certain date.

The note about using —server-root in this case is mentioned in:

https://pypi.org/project/mod-wsgi/

along with other suggestions about using generated config for integration into system init scripts.

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/aaa686b3-265c-44ad-8604-95551cc35d31%40googlegroups.com.

[Lou King]

Thanks. I want to make sure I understand. It seems like at https://pypi.org/project/mod-wsgi/ "Running mod_wsgi-express as root" it says the process is similar to:

• run mod_wsgi-express ... --server-root=<wsgi-dir>
• create systemd service file which runs <wsgi-dir>/apachectl start/stop commands

i.e., the mod_wsgi-express command is executed once to create the environment, and the <wsgi-dir>/apachectl command is used when service is started/stopped/restarted

Am I getting this right?

Graham

To unsubscribe from this group and stop receiving emails from it, send an email to mod...@googlegroups.com.

[Lou King]

I may have answered my own question. When I run the mod_wsgi-express command it seems to run in the foreground. I don't understand the use of the <wsgi-dir>apachectl.

$ sudo /root/bin/init-mod_wsgi-express runningroutes sandbox.routes.loutilities.com routesmgr routesmgr 8002
Server URL         : http://proxysvr.loutilities.com:8002/
Server Root        : /etc/mod_wsgi-express/sandbox.routes.loutilities.com
Server Conf        : /etc/mod_wsgi-express/sandbox.routes.loutilities.com/httpd.conf
Error Log File     : /etc/mod_wsgi-express/sandbox.routes.loutilities.com/error_log (warn)

Request Capacity   : 5 (1 process * 5 threads)

Request Timeout    : 60 (seconds)
Startup Timeout    : 15 (seconds)
Queue Backlog      : 100 (connections)
Queue Timeout      : 45 (seconds)

Server Capacity    : 20 (event/worker), 20 (prefork)

Server Backlog     : 500 (connections)
Locale Setting     : en_US.UTF-8

where /root/bin/init-mod_wsgi-express contains

$ sudo cat /root/bin/init-mod_wsgi-express
#!/bin/bash
if [[ $# -lt 5 ]] ; then
   echo "usage:"
   echo "    init-mod_wsgi-express project servername user group port"
   exit 0
fi


source /var/www/$2/venv/bin/activate
mod_wsgi-express start-server --server-name proxysvr.loutilities.com --port $5 --user $3 --group $4 /var/www/$2/$1/$1/$1.wsgi --working-directory /var/www/$2/$1/$1/ --server-root /etc/mod_wsgi-express/$2
deactivate

[Lou King]

Oops, sorry. I completely missed the use of setup-server instead of start-server. Sorry for the noise.

Graham

[Tue Jan 21 08:21:09.020530 2020] [authz_core:error] [pid 20420:tid 140271178348288] [client 198.199.82.162:49264] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https://steeplechasers.org/training/running-routes/</spa