Apache server goes down multiple times a week

[Juan Khawly]

Hello,

I've been running into this problem for a while.

CONTEXT 

I have an application developed in python (3.10), django 4.0.3, using mod_wsgi and apache. The application is in a DEV environment and hosted in AWS EC2. Currently, it does not receive traffic at all. 

Installation of Mod WSGI

apt-get install -y apache2-dev

Setup out of the VENV

mod_wsgi-express install-module

editing: /etc/apache2/mods-available/wsgi.load

LoadModule wsgi_module "/usr/lib/apache2/modules/mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so"
WSGIPythonHome "/data/home/user/environment/venv"

Module Enabled

a2enmod wsgi

PROBLEM

The application works perfect most of the time. Couple of times a week, without traffic the apache server goes down into 503. Usually it is preceded by a random request but it does not always happen that way. I am assuming that is Slow DDOS but I want to make sure it is not miss configuration of the WSGI.

access.log example

error.log example

I masked the internal routes

This is one of the errors:

[Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid 140518453380864] [client 118.126.82.157:37722] Timeout when reading response headers from daemon process 'XXXXX': /XXX/XXXX/XXXXX/XXXXX/XXXXXX/wsgi.py

Another type of error:

[Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid 140518453380864] [client 118.126.82.157:37722] Timeout when reading response headers from daemon process 'XXXXXXX': /XXX/XXXX/XXXXX/XXXXXX/XXXXXXX/wsgi.py

SOLUTION 

If I restart the server, all works again until next failure.

I've enabled the following modules, in case it is SlowDDOS

modreqtimeout

libapache2-mod-qos

libapache2-mod-security2.

Any recommendation?

Thanks,

Juan Khawly

[Graham Dumpleton]

Would need to see the mod_wsgi configuration you are using to configure the WSGI application, including how WSGIDaemonProcess is configured and whether you are using WSGIApplicationGroup. Also, what errors are in the Apache error log when the 503 errors occur.

On 2 Sep 2022, at 4:57 am, Juan Khawly <juan...@gmail.com> wrote:

Hello,

I've been running into this problem for a while.

CONTEXT 

I have an application developed in python (3.10), django 4.0.3, using mod_wsgi and apache. The application is in a DEV environment and hosted in AWS EC2. Currently, it does not receive traffic at all. 

Installation of Mod WSGI

apt-get install -y apache2-dev

Setup out of the VENV

mod_wsgi-express install-module

editing: /etc/apache2/mods-available/wsgi.load

LoadModule wsgi_module "/usr/lib/apache2/modules/mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so"
WSGIPythonHome "/data/home/user/environment/venv"

Module Enabled

a2enmod wsgi

PROBLEM

The application works perfect most of the time. Couple of times a week, without traffic the apache server goes down into 503. Usually it is preceded by a random request but it does not always happen that way. I am assuming that is Slow DDOS but I want to make sure it is not miss configuration of the WSGI.

access.log example

<access.PNG>

error.log example

I masked the internal routes

This is one of the errors:

[Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid 140518453380864] [client 118.126.82.157:37722] Timeout when reading response headers from daemon process 'XXXXX': /XXX/XXXX/XXXXX/XXXXX/XXXXXX/wsgi.py

Another type of error:

[Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid 140518453380864] [client 118.126.82.157:37722] Timeout when reading response headers from daemon process 'XXXXXXX': /XXX/XXXX/XXXXX/XXXXXX/XXXXXXX/wsgi.py

SOLUTION 

If I restart the server, all works again until next failure.

I've enabled the following modules, in case it is SlowDDOS

modreqtimeout

libapache2-mod-qos

libapache2-mod-security2.

Any recommendation?

Thanks,

Juan Khawly

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com.
<access.PNG>

[Juan Khawly]

Hello Graham, 

I'm going to try to address your questions:

Inside my Virtual Host

        Alias /static /data/home/user/project/frontend/build/static
        <Directory /data/home/user/project/frontend/build/static>
                Require all granted
        </Directory>

        <Directory /data/home/user/project/my_project>
                <Files wsgi.py>
                        Require all granted
                </Files>
        </Directory>

        WSGIScriptAlias / /data/home/user/project/my_project/wsgi.py
        WSGIDaemonProcess my_project python-path=/data/home/user/project python-home=/data/home/user/environment/venv
        WSGIProcessGroup my_project

Inside apache2.conf

WSGIApplicationGroup %{GLOBAL}

On the apache/error.log

When I get the 503 on the access.log, these are the types of errors seen on the error.log

One type of error

[Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid 140518453380864] [client 118.126.82.157:37722] Timeout when reading response headers from daemon process 'my_project': /data/home/project/my_project/my_project/wsgi.py

Another type of error

[Thu Sep 01 04:27:00.053558 2022] [wsgi:error] [pid 3267:tid 140518595991296] (11)Resource temporarily unavailable: [client 172.31.17.102:31880] mod_wsgi (pid=3267): Unable to connect to WSGI daemon process ' my_project  ' on '/var/run/apache2/wsgi.2385.1.1.sock' after multiple attempts as listener backlog limit was exceeded or the socket does not exist.

Thanks,

Juan Khawly

[Juan Khawly]

Any ideas? 

Thanks

[Graham Dumpleton]

Sorry, seems I didn't see your update.

Add an option:

    request-timeout=60

to the WSGIDaemonProcess.

Set the value (in seconds) greater than you would expect your HTTP requests to normally run.

What will happen is that when the average running time for all possible concurrent requests exceeds that timeout value, the daemon process will be forcibly restarted. This will have the effect of unblocking the process and a new one will be started in its place. So acts as a fail safe to ensure your application keeps running.

What this will also do is attempt to dump out Python stack traces for what all the request handler threads were doing when the process is restarted. This will hopefully allow you to work out why your request handlers are getting stuck, be it they are getting stuck on a lock, or waiting on a backend service.

In short, your request handlers are getting stuck and not completing. Over time these are building up and the thread pool for handling requests is exhausted and so the process stops handling requests.

Graham

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com.

[Juan Khawly]

Graham,

Going to make that change, monitor and keep this chat updated with the result.

2 Questions:

1) The option request-timeout = 60 is included inside the virtual host along with the rest of the Daemon code right ?

2) Under no traffic, do you have any idea of why this problem could happen? As I explained, it is usually, but not always, preceded by couple of GET Request from a random IP (bot requests) to random urls. My assumption was Slow DDOS and this is why I enabled modreqtimeout, mod security and mod qos. But at this point I'm clueless of how to diagnose.

Thanks

Juan Khawly

[Graham Dumpleton]

On 7 Sep 2022, at 11:43 pm, Juan Khawly <juan...@gmail.com> wrote:

Graham,

Going to make that change, monitor and keep this chat updated with the result.

2 Questions:

1) The option request-timeout = 60 is included inside the virtual host along with the rest of the Daemon code right ?

It is an option to be added to the existing WSGIDaemonProcess directive.

2) Under no traffic, do you have any idea of why this problem could happen? As I explained, it is usually, but not always, preceded by couple of GET Request from a random IP (bot requests) to random urls. My assumption was Slow DDOS and this is why I enabled modreqtimeout, mod security and mod qos. But at this point I'm clueless of how to diagnose.

No idea. If it was truly a slow DDOS attack the request wouldn't actually show in the access logs because Apache only logs requests on completion. So am not sure one could say is related to those other requests. I would say it is more likely that over time a trickle of requests come in to your application as normal which block and slowly use up capacity. Hopefully the stack trace created when get a forced restart due to request timeout will show where. Just keep in mind that since the request timeout will cause auto recovery you may not notice it occurred, so you will need to periodically check Apache error logs yourself. Make sure that have info LogLevel for the virtual host so get more useful information out of mod_wsgi.

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com.

[Juan Khawly]

Makes total sense. 

Just added the option to the DaemonProcess and LogLevel info to the virtual host. I will be monitoring the the logs and report back in a couple days for reference.

Appreciate your help.

Juan Khawly

[Graham Dumpleton]

As logs show, you have a problem with thread locking related to logging subsystem of Python.

What do you have in your wsgi.py file?

The messages suggest you are calling Django's get_wsgi_application() on every request, which is a bad idea. It should only be called once at top level scope in wsgi.py, not in a request handler function.

Graham

On 10 Sep 2022, at 2:41 am, Juan Khawly <juan...@gmail.com> wrote:

Graham,

After adding the timeout, as you said, the server auto recovers from the problem. 

After mod_wsgi is logging info level. My error log now gives traces of where the problem is. I'm attaching my error.log from today (sep 09),  any ideas? 

Thanks for the support.

Juan Khawly

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/4c08f3cf-54f6-43a3-809d-a463723c274cn%40googlegroups.com.
<error.log>

[Juan Khawly]

Graham, 

It does click now. Couple of months ago, I had to modify the original wsgi.py file to include Environmental variables read from my Virtual Host.

__________________________________________________________________________

I changed from: 

On wsgi.py

import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
application = get_wsgi_application()

to

On Virtual Host

SetEnv VAR1 xxxx

SetEnv VAR2 yyyy

On wsgi.py

import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', ' project  .settings')

def application(environ, start_response):
    os.environ['VAR1'] = environ.get('VAR1', '')
    os.environ['VAR2'] = environ.get('VAR2', '')
    _application = get_wsgi_application()
    return _application(environ, start_response)

__________________________________________________________________________

I think I found this solution on a forum and it worked and never expected that it would yield on such consequences.

Do you have any suggestion on the right way to do this ? I remember testing multiple options and this was the only one that worked.

Thanks,

Juan Khawly

[Graham Dumpleton]

Be aware that what you are doing is not technically safe because environment variables are global and so if different URLs ended up in different values for the variables, then a different request could change it before you got to use it.

That said, use:

import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', ' project  .settings')

_application = get_wsgi_application()

def application(environ, start_response):
    os.environ['VAR1'] = environ.get('VAR1', '')
    os.environ['VAR2'] = environ.get('VAR2', '')

    return _application(environ, start_response)

In other words, just call get_wsgi_application() once outside of the function.

Graham

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/a69d70f4-faac-4515-b991-90590efd7b95n%40googlegroups.com.

[Juan Khawly]

Graham,

If I take the the  get_wsgi_application() out of the function, my site throws error an internal error. Looking inside the logs, it all comes down that my app is not receiving the value of VAR1 or VAR2 that are further used in my settings.py to setup the environment. VAR1 and VAR2 are environment variables that help me recognize if I am in LOCAL, DEV or PROD.

Also, I honestly didn't understand this comment if you can help me see more clearly. "Be aware that what you are doing is not technically safe because environment variables are global and so if different URLs ended up in different values for the variables, then a different request could change it before you got to use it." 

My goal is having environmental variables inside my virtual host and pass them to the Django app. I tried this using the environment variables files that Linux uses but I was never successful to pass them to my Django app.

[Graham Dumpleton]

On 11 Sep 2022, at 2:25 am, Juan Khawly <juan...@gmail.com> wrote:

Graham,

If I take the the  get_wsgi_application() out of the function, my site throws error an internal error. Looking inside the logs, it all comes down that my app is not receiving the value of VAR1 or VAR2 that are further used in my settings.py to setup the environment. VAR1 and VAR2 are environment variables that help me recognize if I am in LOCAL, DEV or PROD.

Also, I honestly didn't understand this comment if you can help me see more clearly. "Be aware that what you are doing is not technically safe because environment variables are global and so if different URLs ended up in different values for the variables, then a different request could change it before you got to use it." 

SetEnv is not technically for setting environment variables. How they are used is dependent on the specific Apache request handler. For CGI scripts they do translate to process environment variables since each CGI request is handled in a separate process invocation. For PHP, mod_wsgi, mod_perl etc, anything set using SetEnv is specific to the request variables and using them to set environment variables is not recommended since different contexts in the Apache configuration could use SetEnv to pass different values for the same variable name. The consequence of this if a request comes in which has one value for the variable and sets a process wide environment variable but before it gets to use that environment variable another parallel request came in which also overrides the environment variable, but with a different value based on what SetEnv dictates, then the first request may end up seeing a different value to what you would expect to see. In other words, in a multi threaded system you will have race conditions because there is no mutex locking on the usage of process wide environment variables.

The recommended way of doing things is for different environment types have a different WSGI script file as entry point. That WSGI script can then set once any process environment variables before importing the real wsgi.py module for the project. Do note this still only works where different instances of your application are delegated to run in different mod_wsgi daemon process groups. You cannot do this if using embedded mode, or where you have only a single mod_wsgi daemon process group with multiple instances of your web application.

        WSGIDaemonProcess my_project_production python-path=/data/home/user/project python-home=/data/home/user/environment/venv

        WSGIScriptAlias /prod /data/home/user/project/my_project/wsgi_production.py process-group=my_project_production application-group=%{GLOBAL}

        WSGIDaemonProcess my_project_development python-path=/data/home/user/project python-home=/data/home/user/environment/venv

        WSGIScriptAlias /dev /data/home/user/project/my_project/wsgi_development.py process-group=my_project_development application-group=%{GLOBAL}

Since multiple mod_wsgi daemon process groups are going to be required, an alternative way is to base decisions off the name of the mod_wsgi daemon process group. A wsgi.py file could thus do:

    import mod_wsgi

    if mod_wsgi.process_group() == "my_project_production":

        ...

    elif mod_wsgi.process_group() == "my_project_development":

        ...

The important thing is use different daemon process groups for different instances of the application using different global configuration, use of Python sub interpreters for separation is not enough.

My goal is having environmental variables inside my virtual host and pass them to the Django app. I tried this using the environment variables files that Linux uses but I was never successful to pass them to my Django app.

If you mean environment variables set in your user login shell environment then no it will not as Apache runs as a completely different user. If you want to override environment variables for Apache, they would need to be set in the systemd/Apache startup scripts.

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/33b4cf84-410f-48d8-a062-4d1ef5e92a50n%40googlegroups.com.

[Juan Khawly]

Graham,

Thanks for the deep explanation. This was really helpful to understand better the architecture. 

In my case, my PROD and DEV applications are in separate machines. I don't think I have to create multiple daemon processes. 

This is what I settled down:

1) Use /etc/environment file to setup my environment variables. There are other processes living in the server that will need to use this file, not only the DJANGO APP. I dont like the idea to create a separate wsgi_dev, wsgi_prod and duplicate the vars. 

2) Modify the wsgi.py to:

  a) Call application = get_wsgi_application() only one time.

  b) Created a small routine that opens the previous file and reads the VARS to store them ONE TIME at os.environ['VAR1'], os.environ['VAR2']

  c) Remove VARS from the virtual host since I wont be able to pass them to the DJANGO App.

I will report back if something goes wrong, but thanks for your time and help

Juan Khawly

[Graham Dumpleton]

Yeah, that is what people often do. I didn't want to dictate where you stored the config as people do things different.

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/e77b2844-6948-457b-8a13-65ef7698c377n%40googlegroups.com.