403 Status when Maximum-Requests is reached and Apache restarts process

[Lalith Maddali]

Hi All,

      Occassionally users reported that UI was very slow. When we dig deeper we found that  some requests never got a response back, rather after loadbalancer timed out it returned 504 status code.

Most API calls are high in request size (due to users saving a lot of text data back to server). I saw some blogs saying that large request sizes can bloat the memory consumed by django process. We were definitely hitting peak memory due to this. 

We introduced maximum-request setting to kill process once it reaches this limit. It was fine until we hit this weird issue of timeouts for seemingly simple API calls.

Digging further, we found that in Apache logs there are 403 status codes, it happens right during the time Apache restarts the processes as maximum-request setting is reached. We could see some logs of django process start logs around the same time.

Question is why do we have 403? In our other production servers this was never an issue, only on this one application, Apache keeps the requests dangling and later 403 and not sending to the active processes. This is requiring us to periodically do apache graceful restarts. This is not elegant solution, but any insight into what to improve would be great.

We also have New Relic installed as middleware. We found that a process restart takes more than 20 seconds. Is this the root cause? How do we deal with improving process restart?

Thanks,
Lalith

[Graham Dumpleton]

Can you supply the mod_wsgi configuration you are using in Apache configuration file.

I would need to see mod_wsgi directives such as WSGIDaemonProcess, WSGIProcessGroup, WSGIApplicationGroup, WSGIScriptAlias and how they are used in the context of Apache configuration.

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/4cb4025a-7dc6-4993-8dd6-78f7492a0607n%40googlegroups.com.

[Lalith Maddali]

Thank you for the prompt reply Graham.

Box capacity: 4 CPU, 16 GB RAM, 14 GB Swap space

Python version: 2.7

Django version: 1.8.4 final
Mod_wsgi version: 4.7.1

Here is Apache configuration. I have masked the internal names with xxx, yyy, bbb but it should be consistent across the whole file. 

WSGISocketPrefix /var/run/

WSGIPassAuthorization On

<Directory "/var/www/html/xxx/yyy">    

    #Options FollowSymLinks    

    #AllowOverride All    

    Options Indexes FollowSymLinks    

    AllowOverride All

</Directory>

<VirtualHost *:443>    

    ServerName yyy.bbb.com    

    ServerAlias yyy.staging.bbb.com     

    DocumentRoot "/var/www/html/xxx/yyy"    

    WSGIDaemonProcess yyy.bbb.com \

        user=apache group=apache \

        processes=6 threads=20 \

        display-name='%{GROUP}' \

        maximum-requests=1800 graceful-timeout=720 socket-timeout=300 \

        python-path=/var/www/html/xxx/:/var/www/html/xxx/ccc    

    WSGIProcessGroup  yyy.bbb.com    

    WSGIScriptAlias / /var/www/html/xxx/yyy/wsgi.py    

    SetHandler wsgi-script    SSLEngine on    

</VirtualHost>

Please let me know if there is anything else I can provide. Thank you for your help!

Thanks,
Lalith

[Graham Dumpleton]

Will possible look and comment more later, but the very first thing you want to do is change:

    WSGIProcessGroup  yyy.bbb.com    
    WSGIScriptAlias / /var/www/html/xxx/yyy/wsgi.py   

to:

    WSGIScriptAlias / /var/www/html/xxx/yyy/wsgi.py process-group=yyy.bbb.com application-group=%{GLOBAL}

The most important bit is setting the application group to %{GLOBAL}. This avoids problems with some third party modules which aren't designed to work in Python sub interpreters and which can cause deadlocks with threads causing them to hang and slowly you run out of capacity.

Other comments are you should not need:

    SetHandler wsgi-script

and it is recommend to set outside of the VirtualHost:

    WSGIRestrictEmbedded On

This last one disables Python in main Apache child worker processes, saving on memory and startup times for those. Because you are using daemon mode you don't need embedded mode enabled.

Using maximum-requests is actually usually not a good idea because when site under load then it causes process to restart too frequently, making things worse.

You should instead look at restart-interval option for periodic restarts.

Also perhaps want to look at request-timeout option to force process restarts when requests run too long and/or hang.

What is the average response time, plus longest response times you expect?

If you haven't watch the talk:

https://www.youtube.com/watch?v=H6Q3l11fjU0

Graham

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/a76046f7-8f30-4756-9a97-6bcc5fca2024n%40googlegroups.com.

[Lalith Maddali]

Graham, I Really appreciate the response. I will work with my team to test these changes and get back to you!

Thanks,
Lalith

[Lalith Maddali]

Regarding your question on average response time and longest resposnse.

Average response times for most of the queries is a maximum of 600 ms.

Longest response times is about 30 seconds and some apis that integrate with 3rd party apps take 5 seconds average. So we overall have some long running calls > 1 sec due to the transactional nature of the application.

Thanks,
Lalith

[Graham Dumpleton]

While researching things, also read:

http://blog.dscpl.com.au/2014/02/vertically-partitioning-python-web.html

It can be advantageous to run multiple daemon process groups and use Location/WSGIProcessGroup to direct subsets of the URL namespace to a different process group and application instance.

By doing this you can better tune each respective daemon process groups for the type of traffic it receives, with a candidate being to segment out long running requests to their own process group so can set high request timeouts, with lower request timeout in core process group handling everything else.

Graham

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/16761a72-0ab4-4396-bd70-fe1aceaf663cn%40googlegroups.com.