Handling High Number of Requests

[Andrew Charles]

Ubuntu 18.04.04

Apache 2.4.29 (event)

mod_wsgi 4.5.17

Python 3.6.8

Django 2.2.10

WSGIScriptAlias / ...wsgi.py
WSGIDaemonProcess name processes=8 threads=30 queue-timeout=45 socket-timeout=60 request-timeout=60 inactivity-timeout=0 startup-timeout=45 deadlock-timeout=60 graceful-timeout=15 eviction-timeout=0 python-path=...base/ python-home=...virtualenv/
WSGIProcessGroup name

AWS ec2 c5.xlarge 4 CPUs 8GB Mem (ASG autoscaling between 4 and 10 instances) behind an ELB

Averaging 100,000,000 requests per day (107 mil today)

We have a few django api endpoints that are very simple, which only hit a local or separate redis cache, no db hits. We relay data to firehose but use django-q to offload those tasks. Requests take around 200ms but a fair number are 400-500ms. The ELB reports the average as 60ms. Each instances uses between 4-5GB mem. I've been trying to get more performance out of our instances and reduce our 5XXs. I previously tried 3 processes and the default (15) threads. I've been researching the best ways to change settings but it seems like it's unique to every setup and there's no easy rules to follow. I'm looking for suggestions or at least someone to tell me I'm on the right track.

[Jason Garber]

Hello,

What does the math say at peak times?  100,000,000 is a lot in a day, and it would not take much disturbance to spike requests.

I assume the clients are not slow?  Hanging clients consume a valuable thread.  We use nginx buffering request/response to mitigate that and free up thread as soon as possible.

Keep in mind python gil will bottleneck threads if your processes are at all cpu bound...  I might consider more processes and less threads per process.  

Any pattern around when you get 5xx errors?   Can you spin up 20 extra instances for an hour to see if that cures them?

Just some thoughts...

Jason

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/f7e65e6b-8975-4292-ba05-1e347945a6d1%40googlegroups.com.

[Graham Dumpleton]

Are you using any sort of application performance monitoring (APM) product, such as APM products from New Relic, DataDog, AppDyamics or Elastic?

What specifically are the 5xx errors you are getting?

As Jason mentioned less threads per process and more processes is always generally better. Even if you don't use a full APM product, if you have a way of collecting metrics, you could extract metrics data out of mod_wsgi using its event callbacks. One of the things you can work out from that is capacityutilisation. This will show whether you have way too many threads per process, with potential for bottlenecking on one process due to uneven distribution of requests across processes and GIL side effects if things are a bit CPU bound.

Also, do you have:

    WSGIRestrictEmbedded On

set so that you avoid starting Python interpreters in the main Apache child processes.

If you only have the one Python WSGI application, you should also be setting:

    WSGIApplicationGroup %{GLOBAL}

Graham

[Andrew Charles]

Peak is 125,000 per minute. I'm really not sure about the 5XXs, the ELB is reporting them but I don't see too many in the apache logs. Seems to be 502s or 503s. They're definitely lessened with more instances.

To unsubscribe from this group and stop receiving emails from it, send an email to mod...@googlegroups.com.

[Andrew Charles]

I'm not using any APMs. Mostly I have a cron checking for non-200s on a simple health check endpoint or relying on AWS metrics.

I've added the following settings and will see how it goes: WSGIRestrictEmbedded On, WSGIApplicationGroup %{GLOBAL}, WSGIPythonOptimize 1

I'll also try out more processes and less threads.

On Monday, February 17, 2020 at 10:31:23 PM UTC-7, Graham Dumpleton wrote:

Are you using any sort of application performance monitoring (APM) product, such as APM products from New Relic, DataDog, AppDyamics or Elastic?

What specifically are the 5xx errors you are getting?

As Jason mentioned less threads per process and more processes is always generally better. Even if you don't use a full APM product, if you have a way of collecting metrics, you could extract metrics data out of mod_wsgi using its event callbacks. One of the things you can work out from that is capacityutilisation. This will show whether you have way too many threads per process, with potential for bottlenecking on one process due to uneven distribution of requests across processes and GIL side effects if things are a bit CPU bound.

Also, do you have:

    WSGIRestrictEmbedded On

set so that you avoid starting Python interpreters in the main Apache child processes.

If you only have the one Python WSGI application, you should also be setting:

    WSGIApplicationGroup %{GLOBAL}

Graham

On 18 Feb 2020, at 12:55 pm, Andrew Charles <andrew...@antyc.ca> wrote:

Ubuntu 18.04.04

Apache 2.4.29 (event)

mod_wsgi 4.5.17

Python 3.6.8

Django 2.2.10

WSGIScriptAlias / ...wsgi.py
WSGIDaemonProcess name processes=8 threads=30 queue-timeout=45 socket-timeout=60 request-timeout=60 inactivity-timeout=0 startup-timeout=45 deadlock-timeout=60 graceful-timeout=15 eviction-timeout=0 python-path=...base/ python-home=...virtualenv/
WSGIProcessGroup name

AWS ec2 c5.xlarge 4 CPUs 8GB Mem (ASG autoscaling between 4 and 10 instances) behind an ELB

Averaging 100,000,000 requests per day (107 mil today)

We have a few django api endpoints that are very simple, which only hit a local or separate redis cache, no db hits. We relay data to firehose but use django-q to offload those tasks. Requests take around 200ms but a fair number are 400-500ms. The ELB reports the average as 60ms. Each instances uses between 4-5GB mem. I've been trying to get more performance out of our instances and reduce our 5XXs. I previously tried 3 processes and the default (15) threads. I've been researching the best ways to change settings but it seems like it's unique to every setup and there's no easy rules to follow. I'm looking for suggestions or at least someone to tell me I'm on the right track.

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mod...@googlegroups.com.

[Andrew Charles]

Upping the processes increases the memory usage quite a lot. I tried 16 procs and 15 threads and ran out.

[Graham Dumpleton]

What Apache MPM are you using? What are the MPM settings in Apache? Eg:

* https://httpd.apache.org/docs/2.4/mod/event.html

Is MaxMemFree directive set in Apache configuration?

How much memory does each mod_wsgi daemon process take?

How much memory does each normal Apache child process take?

Do you have log level set to info so you know how often mod_wsgi daemon processes are getting restarted due to request timeouts?

Graham

-- 
You received this message because you are subscribed to the Google Groups "modwsgi" group.

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/d52202b0-1641-40ce-a903-f9c5e252223d%40googlegroups.com.

[Andrew Charles]

The default one, and a2query -M outputs event. I haven't changed any apache settings like MaxMemFree.

In htop each mod_wsgi proc takes ~960M VIRT, each apache proc takes ~1980-2060M VIRT

I've been testing out different procs/threads and 10 procs with 5 threads seems to be the same as 8 procs with 30 threads.

I'll see what the logs output with info.

On Wednesday, February 19, 2020 at 3:10:53 PM UTC-7, Graham Dumpleton wrote:

What Apache MPM are you using? What are the MPM settings in Apache? Eg:

* https://httpd.apache.org/docs/2.4/mod/event.html

Is MaxMemFree directive set in Apache configuration?

How much memory does each mod_wsgi daemon process take?

How much memory does each normal Apache child process take?

Do you have log level set to info so you know how often mod_wsgi daemon processes are getting restarted due to request timeouts?

Graham

On 20 Feb 2020, at 8:36 am, Andrew Charles <andrew...@antyc.ca> wrote:

Upping the processes increases the memory usage quite a lot. I tried 16 procs and 15 threads and ran out.

On Monday, February 17, 2020 at 7:13:24 PM UTC-7, Andrew Charles wrote:

Ubuntu 18.04.04

Apache 2.4.29 (event)

mod_wsgi 4.5.17

Python 3.6.8

Django 2.2.10

WSGIScriptAlias / ...wsgi.py
WSGIDaemonProcess name processes=8 threads=30 queue-timeout=45 socket-timeout=60 request-timeout=60 inactivity-timeout=0 startup-timeout=45 deadlock-timeout=60 graceful-timeout=15 eviction-timeout=0 python-path=...base/python-home=...virtualenv/
WSGIProcessGroup name

AWS ec2 c5.xlarge 4 CPUs 8GB Mem (ASG autoscaling between 4 and 10 instances) behind an ELB

Averaging 100,000,000 requests per day (107 mil today)

We have a few django api endpoints that are very simple, which only hit a local or separate redis cache, no db hits. We relay data to firehose but use django-q to offload those tasks. Requests take around 200ms but a fair number are 400-500ms. The ELB reports the average as 60ms. Each instances uses between 4-5GB mem. I've been trying to get more performance out of our instances and reduce our 5XXs. I previously tried 3 processes and the default (15) threads. I've been researching the best ways to change settings but it seems like it's unique to every setup and there's no easy rules to follow. I'm looking for suggestions or at least someone to tell me I'm on the right track.

-- 
You received this message because you are subscribed to the Google Groups "modwsgi" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mod...@googlegroups.com.

[Graham Dumpleton]

It is resident memory (RSS), not virtual memory which is more important to know.

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/3e061ff2-11aa-4b35-af39-650b17690e46%40googlegroups.com.

[Andrew Charles]

Sorry I should've included that. mod_wsgi proc is ~400M RES and apache proc is ~1600-1700 (no M) RES.

There's definitely errors in the logs now I'll try to paste some. The file is huge though (2.3GB error 2.4GB other_vhosts_access) I'll paste a snippet.

On Wednesday, February 19, 2020 at 5:07:25 PM UTC-7, Graham Dumpleton wrote:

It is resident memory (RSS), not virtual memory which is more important to know.

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/3e061ff2-11aa-4b35-af39-650b17690e46%40googlegroups.com.

[Graham Dumpleton]

On 20 Feb 2020, at 11:16 am, Andrew Charles <andrew....@antyc.ca> wrote:

Sorry I should've included that. mod_wsgi proc is ~400M RES and apache proc is ~1600-1700 (no M) RES.

There's definitely errors in the logs now I'll try to paste some. The file is huge though (2.3GB error 2.4GB other_vhosts_access) I'll paste a snippet.

Am mainly worried about whether you see evidence of daemon process restarts over time for some reason as that will affect performance if restarting often. You can set LogLevel just in your VirtualHost to limit amount output. So long as the WSGIDaemonProcess directives are inside of the VirtualHost I will get what am interested in.

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/aff5e970-bc14-48d5-b769-49c0ac11ab12%40googlegroups.com.

[Andrew Charles]

Not sure if this should be in an attached file or not:

[Thu Feb 20 00:18:30.165197 2020] [wsgi:error] [pid 21732:tid 140156156389120] [client 172.30.2.15:34800] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.factahead.com/pt/15120-papa-beijo/1/?utm_source=taboola&utm_campaign=3815502&utm_medium=premiumprogrammatic-bandab&utm_content=2881672882

[Thu Feb 20 00:18:38.578370 2020] [wsgi:error] [pid 21842:tid 140156156389120] [client 172.30.2.15:35266] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.laprensa.com.ni/2020/02/14/economia/2640873-la-marseillaise-uno-de-los-restaurantes-mas-emblematicos-de-managua-anuncia-su-cierre?fbclid=IwAR1_ICdMFRE1tz8v4OfpHI-hqJ2B2LhqgyR6vLs_AXrdVESyWCiRPJr6iAs
[Thu Feb 20 00:18:41.774913 2020] [reqtimeout:info] [pid 21842:tid 140155922872064] [client 172.30.2.15:35084] AH01382: Request body read timeout
[Thu Feb 20 00:18:41.774934 2020] [wsgi:error] [pid 21842:tid 140155922872064] [client 172.30.2.15:35084] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.factahead.com/pt/15120-papa-beijo/1/?utm_source=taboola&utm_campaign=3815502&utm_medium=premiumprogrammatic-bandab&utm_content=2881672882
[Thu Feb 20 00:18:44.278709 2020] [wsgi:error] [pid 21842:tid 140156057155328] [client 172.30.2.15:35502] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.bitelists.com/es/12-alimentos-que-aumentan-tus-niveles-de-azucar-en-la-sangre/4/
[Thu Feb 20 00:18:45.572466 2020] [wsgi:error] [pid 21732:tid 140156181567232] [client 172.30.2.15:35494] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://nczas.com/2020/02/19/to-nie-szalenstwo-tylko-opetanie-feministki-ujawnily-prawdziwe-oblicze-swojego-ruchu-2/?fbclid=IwAR32iGhxryTsx52NUdKpAMqyZDegMW_W2LtS2nDL0aGmICHLOJSSBjnoQD8
[Thu Feb 20 00:18:46.373248 2020] [wsgi:error] [pid 21781:tid 140155931264768] [client 172.30.0.30:59762] mod_wsgi (pid=21781): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.megaatualizado.com/ex-bbbs-voltam-ao-bbb20-saiba-os-nomes-dos-escolhidos-pela-producao/?slot=abae6a196602aad&utm_campaign=707262786961&utm_medium=abae6a196602aad&insta=true
[Thu Feb 20 00:18:46.991408 2020] [wsgi:error] [pid 21732:tid 140156156389120] [client 172.30.2.15:35552] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.c-sharpcorner.com/article/sociamedia-link-share-in-android-using-androi/?fbclid=IwAR1a1g3nu_1wFTZMzozDD46uZY427SbgPRbbTgSKmGZ_4D3ZmDi3MNrtbZQ
[Thu Feb 20 00:18:47.058587 2020] [wsgi:error] [pid 21574:tid 140156031977216] [client 172.30.0.30:59734] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.c-sharpcorner.com/article/sociamedia-link-share-in-android-using-androi/?fbclid=IwAR1a1g3nu_1wFTZMzozDD46uZY427SbgPRbbTgSKmGZ_4D3ZmDi3MNrtbZQ
[Thu Feb 20 00:18:51.453268 2020] [wsgi:error] [pid 21842:tid 140155931264768] [client 172.30.1.233:53202] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://bustednewspaper.com/bryant-roy-martin-2020-02-19-arrest-booking-mugshot-williamson-county-texas/?fbclid=IwAR2owCqOWJDODO9kckt6fHdCCZ5jtbbz-9cPLzyOPg6G0Us4-SXnbjUpTXY
[Thu Feb 20 00:18:55.987232 2020] [reqtimeout:info] [pid 21842:tid 140155939657472] [client 172.30.2.15:35448] AH01382: Request body read timeout
[Thu Feb 20 00:18:55.987259 2020] [wsgi:error] [pid 21842:tid 140155939657472] [client 172.30.2.15:35448] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://androidwaves.com/download-whatsapp-plus-apk/
[Thu Feb 20 00:18:59.264830 2020] [wsgi:error] [pid 21732:tid 140155956442880] [client 172.30.1.233:53554] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.jugantor.com/national/280323/%E0%A6%AE%E0%A6%BF%E0%A6%A5%E0%A7%8D%E0%A6%AF%E0%A6%BE%E0%A6%B0-%E0%A6%86%E0%A6%B6%E0%A7%8D%E0%A6%B0%E0%A7%9F-%E0%A6%A8%E0%A6%BF%E0%A7%9F%E0%A7%87-%E0%A6%AB%E0%A6%BE%E0%A6%81%E0%A6%B8%E0%A6%B2%E0%A7%87%E0%A6%A8-%E0%A6%A8%E0%A6%BE%E0%A6%9C%E0%A6%AE%E0%A7%81%E0%A6%B2-%E0%A6%B9%E0%A7%81%E0%A6%A6%E0%A6%BE-%E0%A6%A6%E0%A7%81%E0%A6%A6%E0%A6%95%E0%A7%87%E0%A6%B0-%E0%A6%AE%E0%A6%BE%E0%A6%AE%E0%A6%B2%E0%A6%BE
[Thu Feb 20 00:19:14.526240 2020] [wsgi:error] [pid 21732:tid 140155956442880] [client 172.30.1.233:53910] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://mdundo.com/search?q=Maleek+bery+mp3+music
[Thu Feb 20 00:19:14.909726 2020] [wsgi:error] [pid 21732:tid 140156006799104] [client 172.30.2.15:36402] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://rolloid.net/descubre-por-que-no-deberias-estrechar-la-mano-de-los-demas/?fbclid=IwAR1lHJG-RSByoKj9fQlqQ61UNkL7lDniO3tvm8oYI44Aebafto0FjJgx4Zs
[Thu Feb 20 00:19:21.594903 2020] [reqtimeout:info] [pid 21732:tid 140155931264768] [client 172.30.2.15:36400] AH01382: Request body read timeout
[Thu Feb 20 00:19:21.594919 2020] [wsgi:error] [pid 21732:tid 140155931264768] [client 172.30.2.15:36400] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://androidwaves.com/download-whatsapp-plus-apk/
[Thu Feb 20 00:19:23.590906 2020] [reqtimeout:info] [pid 21574:tid 140156189959936] [client 172.30.1.233:53982] AH01382: Request body read timeout
[Thu Feb 20 00:19:23.590922 2020] [wsgi:error] [pid 21574:tid 140156189959936] [client 172.30.1.233:53982] mod_wsgi (pid=21574): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.megaatualizado.com/bbb20-lucas-se-vinga-apos-eliminacao-e-faz-trolagem-que-interrompe-programa/?slot=abae6a196602aad&utm_campaign=7176746E6A61&utm_medium=abae6a196602aad&insta=true
[Thu Feb 20 00:19:24.170869 2020] [wsgi:error] [pid 21842:tid 140155990013696] [client 172.30.1.233:53698] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://senhorfinancas.com/content/m2-m/os-30-brasileiros-mais-ricos-do-mundo/19/?utm_source=fb&utm_campaign=23844152138870012&utm_content=23844152138920012&utm_medium=Facebook_Mobile_Feed
[Thu Feb 20 00:19:24.482930 2020] [wsgi:error] [pid 21574:tid 140156147996416] [client 172.30.0.30:60984] mod_wsgi (pid=21574): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.dailyconservative.com/daily-blog/breaking-bloomberg-fails-its-over.htm
[Thu Feb 20 00:19:26.090899 2020] [reqtimeout:info] [pid 21842:tid 140156164781824] [client 172.30.0.30:60380] AH01382: Request body read timeout
[Thu Feb 20 00:19:26.090918 2020] [wsgi:error] [pid 21842:tid 140156164781824] [client 172.30.0.30:60380] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://androidwaves.com/download-whatsapp-plus-apk/
[Thu Feb 20 00:19:26.944255 2020] [wsgi:error] [pid 21575:tid 140156006799104] [client 172.30.0.30:61114] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://psxbrasil.com.br/noticias/sony-oferece-temas-da-celebracao-do-jogador-playstation-na-ps-store/?fbclid=IwAR1V0yl-OmQTYNoxyy6tRbdW0vkd6mK5Sk1IheCoBcPDPIvvVza9gUDddas
[Thu Feb 20 00:19:30.560903 2020] [wsgi:error] [pid 21842:tid 140156006799104] [client 172.30.0.30:61164] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://urdu.siasat.pk/news/siasi/2020-02-19/news-10140?fbclid=IwAR16FBKUaOP1DXKPsd0XOgplMdCe5GOGXTkPXd3Bv7vdTvp52KclrSMhRyw
[Thu Feb 20 00:19:35.561877 2020] [wsgi:error] [pid 21732:tid 140156181567232] [client 172.30.1.233:54534] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://diariodopoder.com.br/cpmi-das-fake-news-ouve-dona-da-empresa-acusada-de-fraude-nas-eleicoes-de-2018/?fbclid=IwAR35QpC5wFfT4ACv1wQBlUG_XSHwc9FUKZyQtHdwNsGmh1NHoJoO8njFAgY
[Thu Feb 20 00:19:39.018895 2020] [reqtimeout:info] [pid 21574:tid 140156189959936] [client 172.30.2.15:36978] AH01382: Request body read timeout
[Thu Feb 20 00:19:39.018926 2020] [wsgi:error] [pid 21574:tid 140156189959936] [client 172.30.2.15:36978] mod_wsgi (pid=21574): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.solitar.io/solitario-spider2
[Thu Feb 20 00:19:41.365865 2020] [wsgi:error] [pid 21842:tid 140155939657472] [client 172.30.1.233:54840] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://theloadedbaze.com/study-in-norway-admissions-scholarships-financial-aid-visa-learn-everything/
[Thu Feb 20 00:19:42.353707 2020] [wsgi:error] [pid 21842:tid 140155981620992] [client 172.30.1.233:54822] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://androidwaves.com/download-gbwhatsapp-pro/
[Thu Feb 20 00:19:44.789410 2020] [wsgi:error] [pid 21732:tid 140156057155328] [client 172.30.2.15:37356] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: http://mzamin.com/article.php?mzamin=213786&fbclid=IwAR3cydSQOBfrZjctKGz5zS4tOsBUbquHSP4aVXmBYHr6y3BEQkC09iQi_nQ
[Thu Feb 20 00:19:46.226905 2020] [reqtimeout:info] [pid 21732:tid 140155973228288] [client 172.30.1.233:54634] AH01382: Request body read timeout
[Thu Feb 20 00:19:46.226929 2020] [wsgi:error] [pid 21732:tid 140155973228288] [client 172.30.1.233:54634] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.laprensa.com.ni/2020/02/19/hoy/2642554-pobladores-detienen-y-entregan-a-la-policia-a-hombre-que-intento-violar-a-una-nina-de-10-anos
[Thu Feb 20 00:19:48.785390 2020] [wsgi:error] [pid 21842:tid 140156015191808] [client 172.30.0.30:61332] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.kikiers.com/post/726e65b3-fa74-4e7a-8a3e-5061c38ba55f/story
[Thu Feb 20 00:19:49.226899 2020] [reqtimeout:info] [pid 21842:tid 140156164781824] [client 172.30.0.30:61398] AH01382: Request body read timeout
[Thu Feb 20 00:19:49.226917 2020] [wsgi:error] [pid 21842:tid 140156164781824] [client 172.30.0.30:61398] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.notilogia.com/2016/10/como-solicitar-datos-filiatorios-en-el-saime.html
[Thu Feb 20 00:19:49.740949 2020] [wsgi:error] [pid 21842:tid 140155922872064] [client 172.30.1.233:54788] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: http://www.dainikamadershomoy.com/post/242420
[Thu Feb 20 00:19:50.519210 2020] [wsgi:error] [pid 21842:tid 140156048762624] [client 172.30.1.233:54956] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.forever-mom.com/de/15116/13/?utm_source=taboola&utm_campaign=3802287&utm_medium=general-anzeiger-bonn&utm_content=2881144027
[Thu Feb 20 00:19:53.881251 2020] [wsgi:error] [pid 21842:tid 140155914479360] [client 172.30.0.30:61838] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: http://www.gaz.com.br/conteudos/esportes/2020/02/18/162009-medalha_de_ouro_domina_prova_em_capao_da_canoa.html.php?fbclid=IwAR1OcnXgjMecywsfgw-3ansx8xYWccQt1KGaUL92G6Tw-FXnNMZSC-VUt4U
[Thu Feb 20 00:19:54.960333 2020] [wsgi:error] [pid 21842:tid 140156023584512] [client 172.30.0.30:61906] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://mdundo.com/a/6734?fbclid=IwAR0YynriYASx5criuBy0ouqvsjFpxvfcXAgq0ChHGmbegnrgnT3ZIrjq5Ss
[Thu Feb 20 00:19:56.125280 2020] [wsgi:error] [pid 21842:tid 140156173174528] [client 172.30.1.233:55120] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://nczas.com/2020/02/19/kenia-polska-i-usa-czyli-miedzynarodowa-afera-murzyn-przebral-sie-za-generala-i-zostal-wpuszczony-na-teren-polskich-zakladow-zbrojeniowych/?fbclid=IwAR2Rt4L_c6ZDZl3tSGT-9TFEBrBGhfiaHZyc0cQizjdDT0axsZ-JaEl5QFs
[Thu Feb 20 00:19:56.818904 2020] [reqtimeout:info] [pid 21732:tid 140156006799104] [client 172.30.1.233:54992] AH01382: Request body read timeout
[Thu Feb 20 00:19:56.818919 2020] [wsgi:error] [pid 21732:tid 140156006799104] [client 172.30.1.233:54992] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://androidwaves.com/download-gbwhatsapp-pro/
[Thu Feb 20 00:20:00.319812 2020] [wsgi:error] [pid 21842:tid 140156040369920] [client 172.30.0.30:62098] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.kikiers.com/post/988e5bce-872f-425e-a9b1-59c6d3d14209/story
[Thu Feb 20 00:20:03.718683 2020] [wsgi:error] [pid 21574:tid 140156048762624] [client 172.30.0.30:62432] mod_wsgi (pid=21574): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.allgoodwedding.com/pt/14290?utm_source=taboola&utm_campaign=3356905&utm_medium=berimbaunoticias&utm_content=2856719506
[Thu Feb 20 00:20:07.582588 2020] [wsgi:error] [pid 21839:tid 140156023584512] [client 172.30.1.233:55644] mod_wsgi (pid=21839): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://androidwaves.com/download-gbwhatsapp-pro/
[Thu Feb 20 00:20:07.759872 2020] [wsgi:error] [pid 21842:tid 140155931264768] [client 172.30.1.233:55358] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: http://www.dainikamadershomoy.com/post/241655
[Thu Feb 20 00:20:09.014927 2020] [reqtimeout:info] [pid 21732:tid 140155939657472] [client 172.30.1.233:54430] AH01382: Request body read timeout
[Thu Feb 20 00:20:09.014945 2020] [wsgi:error] [pid 21732:tid 140155939657472] [client 172.30.1.233:54430] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://kievportal.com/2019/12/07/sepa-como-es-una-tarjeta-de-credito-online-y-sin-consulta-al-buro/
[Thu Feb 20 00:20:12.144720 2020] [wsgi:error] [pid 21732:tid 140156147996416] [client 172.30.0.30:62462] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.apost.com/en/blog/mom-waits-for-autistic-son-to-speak-in-thanksgiving-play-but-mic-is-snatched-away-from-him-by-teacher/25007/?fbclid=IwAR0moquCZ5kxSHQmlBku-lEiMIMu2yGV-qHCdhoLJw25cnkTRk3SZcigPrs
[Thu Feb 20 00:20:12.813229 2020] [wsgi:error] [pid 21732:tid 140156057155328] [client 172.30.2.15:38174] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.bitelists.com/pt-br/10-maneiras-faceis-de-parar-os-sinais-de-envelhecimento/2/
[Thu Feb 20 00:20:15.306259 2020] [wsgi:error] [pid 21732:tid 140155964835584] [client 172.30.0.30:62358] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.jailbase.com/arrested/north-dakota/williams-county/jeremy-arnold-berger-2019-07-16-berger-jeremy-arnold
[Thu Feb 20 00:20:16.249325 2020] [wsgi:error] [pid 21732:tid 140156048762624] [client 172.30.1.233:55738] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.bitelists.com/es/10-beneficios-de-la-granada/2/
[Thu Feb 20 00:20:24.113422 2020] [wsgi:error] [pid 21732:tid 140155964835584] [client 172.30.1.233:55650] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://androidwaves.com/
[Thu Feb 20 00:20:24.416880 2020] [wsgi:error] [pid 21842:tid 140156015191808] [client 172.30.1.233:56112] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.megaatualizado.com/motorista-cita-anitta-para-justificar-assedio-e-ela-rebate-nada-justifica/?slot=abae6a196602aad&utm_campaign=71796976776B&utm_medium=abae6a196602aad&insta=true
[Thu Feb 20 00:20:29.470934 2020] [reqtimeout:info] [pid 21575:tid 140156173174528] [client 172.30.2.15:38360] AH01382: Request body read timeout
[Thu Feb 20 00:20:29.470970 2020] [wsgi:error] [pid 21575:tid 140156173174528] [client 172.30.2.15:38360] mod_wsgi (pid=21575): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://androidwaves.com/category/mods/
[Thu Feb 20 00:20:29.900675 2020] [wsgi:error] [pid 21574:tid 140156048762624] [client 172.30.1.233:56036] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://punchnew.com/wish-i-could-just-disappear-with-my-kid-and-never-come-back-davidos-second-babymama-amanda/?fbclid=IwAR1ebx06gRnVYX4vY0CROLhTADfoHvbCGvIk_GITkLR47YTtkG-ta3vfhK8
[Thu Feb 20 00:20:39.081623 2020] [wsgi:error] [pid 21574:tid 140155939657472] [client 172.30.2.15:39006] mod_wsgi (pid=21574): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: http://forum.skyscraperpage.com/forumdisplay.php?f=163
[Thu Feb 20 00:20:44.266922 2020] [reqtimeout:info] [pid 21841:tid 140155956442880] [client 172.30.1.233:56428] AH01382: Request body read timeout
[Thu Feb 20 00:20:44.266951 2020] [wsgi:error] [pid 21841:tid 140155956442880] [client 172.30.1.233:56428] mod_wsgi (pid=21841): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.factahead.com/es/15120/?utm_source=taboola&utm_campaign=3815505&utm_medium=cmi-unotv&utm_content=2881673254
[Thu Feb 20 00:20:58.786332 2020] [wsgi:error] [pid 21842:tid 140156164781824] [client 172.30.0.30:63864] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.rotamapas.com.br/como-ir-de-aripuana-a-humaita-am
[Thu Feb 20 00:21:01.614912 2020] [reqtimeout:info] [pid 21842:tid 140156023584512] [client 172.30.2.15:39334] AH01382: Request body read timeout
[Thu Feb 20 00:21:01.614936 2020] [wsgi:error] [pid 21842:tid 140156023584512] [client 172.30.2.15:39334] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.boredpanda.com/before-after-weight-transformations-my-600-lb-life/?utm_source=google&utm_medium=organic&utm_campaign=organic
[Thu Feb 20 00:21:04.511702 2020] [wsgi:error] [pid 21842:tid 140156031977216] [client 172.30.1.233:57396] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: http://sandesh.com/navsari-married-woman-took-last-step-in-life/
[Thu Feb 20 00:21:07.230263 2020] [wsgi:error] [pid 21842:tid 140156173174528] [client 172.30.2.15:39822] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.boredpanda.com/retired-mi5-agent-confesses-on-his-deathbed-i-killed-princess-diana-because-she-knew-secrets-of-royalty-video-3/
[Thu Feb 20 00:21:09.343700 2020] [wsgi:error] [pid 21781:tid 140156015191808] [client 172.30.0.30:64022] mod_wsgi (pid=21781): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.megaatualizado.com/marilia-mendonca-pega-todos-de-surpresa-ao-revelar-doenca-de-seu-filho/2/
[Thu Feb 20 00:21:10.561797 2020] [wsgi:error] [pid 21842:tid 140155956442880] [client 172.30.1.233:57310] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.megaatualizado.com/apos-relacionamento-com-o-jogador-ex-de-neymar-surge-gravida-e-anuncia-a-chegada-do-herdeiro/2/
[Thu Feb 20 00:21:12.690901 2020] [reqtimeout:info] [pid 21842:tid 140155981620992] [client 172.30.2.15:39728] AH01382: Request body read timeout
[Thu Feb 20 00:21:12.690914 2020] [wsgi:error] [pid 21842:tid 140155981620992] [client 172.30.2.15:39728] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.factahead.com/pt/13624-miss-belvedere/10/?utm_source=taboola&utm_campaign=3815525&utm_medium=oantagonista&utm_content=2881680495
[Thu Feb 20 00:21:14.381162 2020] [wsgi:error] [pid 21732:tid 140155956442880] [client 172.30.0.30:64432] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.bangladeshtoday.net/%e0%a7%aa%e0%a7%a6-%e0%a6%ac%e0%a6%9b%e0%a6%b0-%e0%a6%86%e0%a6%97%e0%a7%87%e0%a6%b0-%e0%a6%89%e0%a6%aa%e0%a6%a8%e0%a7%8d%e0%a6%af%e0%a6%be%e0%a6%b8%e0%a7%87-%e0%a6%89%e0%a6%b9%e0%a6%be%e0%a6%a8/?fbclid=IwAR2nUsERG2CG7uuHAYkqnmBDYbIqFLwtaZf0NNcotAoqkhE9nNlXfL4gvf4
[Thu Feb 20 00:21:18.010906 2020] [reqtimeout:info] [pid 21732:tid 140156181567232] [client 172.30.1.233:57388] AH01382: Request body read timeout
[Thu Feb 20 00:21:18.010923 2020] [wsgi:error] [pid 21732:tid 140156181567232] [client 172.30.1.233:57388] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://androidwaves.com/download-whatsapp-plus-apk/
[Thu Feb 20 00:21:21.934906 2020] [reqtimeout:info] [pid 21574:tid 140156031977216] [client 172.30.2.15:39980] AH01382: Request body read timeout
[Thu Feb 20 00:21:21.934925 2020] [wsgi:error] [pid 21574:tid 140156031977216] [client 172.30.2.15:39980] mod_wsgi (pid=21574): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.megaatualizado.com/thammy-se-cansa-resolve-desabafar-e-caso-de-traicao-vem-a-tona-foi-na-minha-cama/?slot=ab7bcf0d86&utm_campaign=67677576616A&utm_medium=ab7bcf0d86&insta=true
[Thu Feb 20 00:21:22.970277 2020] [wsgi:error] [pid 21842:tid 140155948050176] [client 172.30.2.15:39986] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.apost.com/th/blog/elvis-thaaaihephlngnepnthnym-aetem-andrea-bocelli-rngmn-naataataanghydlngmaacchaakaibhnaakhngthkkhn/1737/?fbclid=IwAR2DmYGgWFE-5lpNDNJeu0BktsOad47YUxa8z-ceLaK6VFycQ5MXA96f0RI
[Thu Feb 20 00:21:32.514047 2020] [wsgi:error] [pid 21732:tid 140155990013696] [client 172.30.1.233:58082] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://romsmania.cc/download/roms/playstation-portable/tekken-6-275407
[Thu Feb 20 00:21:35.530925 2020] [reqtimeout:info] [pid 21780:tid 140155956442880] [client 172.30.2.15:40126] AH01382: Request body read timeout
[Thu Feb 20 00:21:35.530946 2020] [wsgi:error] [pid 21780:tid 140155956442880] [client 172.30.2.15:40126] mod_wsgi (pid=21780): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://senhorfinancas.com/content/m2-m/os-30-brasileiros-mais-ricos-do-mundo/16/?utm_source=fb&utm_campaign=23844152138870012&utm_content=23844152138920012&utm_medium=Facebook_Mobile_Feed
[Thu Feb 20 00:21:38.132611 2020] [wsgi:error] [pid 21839:tid 140155973228288] [client 172.30.0.30:1294] mod_wsgi (pid=21839): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://romsmania.cc/roms/playstation-portable
[Thu Feb 20 00:21:38.515295 2020] [wsgi:error] [pid 21842:tid 140155914479360] [client 172.30.0.30:64912] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://community.cartalk.com/
[Thu Feb 20 00:21:40.827123 2020] [wsgi:error] [pid 21842:tid 140155914479360] [client 172.30.1.233:58378] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.jugantor.com/todays-paper/first-page/242303/%E0%A6%AA%E0%A6%AC%E0%A6%BF%E0%A6%A4%E0%A7%8D%E0%A6%B0-%E0%A6%88%E0%A6%A6%E0%A7%87-%E0%A6%AE%E0%A6%BF%E0%A6%B2%E0%A6%BE%E0%A6%A6%E0%A7%81%E0%A6%A8%E0%A7%8D%E0%A6%A8%E0%A6%AC%E0%A7%80-(%E0%A6%B8%E0%A6%BE.)-%E0%A6%86%E0%A6%9C?fbclid=IwAR1IqqhUMBHXpZWES_UR0hsZZIDbS3ifzvvgkJvOmSdloxtq9hbQKnSuufA
[Thu Feb 20 00:21:44.610921 2020] [reqtimeout:info] [pid 21842:tid 140156181567232] [client 172.30.1.233:58200] AH01382: Request body read timeout
[Thu Feb 20 00:21:44.610939 2020] [wsgi:error] [pid 21842:tid 140156181567232] [client 172.30.1.233:58200] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.pixilart.com/
[Thu Feb 20 00:21:45.603464 2020] [wsgi:error] [pid 21842:tid 140156173174528] [client 172.30.0.30:1426] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.matematicagenial.com/2017/12/teste-de-visao-quantos-rolos-de-papel-tem-nessa-foto.html?m=1
[Thu Feb 20 00:21:45.858035 2020] [wsgi:error] [pid 21842:tid 140156057155328] [client 172.30.1.233:58424] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://diariodopoder.com.br/sponholz-347/?fbclid=IwAR30j_5K8xX_kkb2QMtHovhkoHp1Tc6EGnOCUOrS8tWCouFlv3eoiIrmVt8
[Thu Feb 20 00:21:46.191051 2020] [wsgi:error] [pid 21842:tid 140156189959936] [client 172.30.1.233:58434] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://saaa25.org/%d9%81%d9%88%d8%a7%d8%a6%d8%af-%d8%a7%d9%84%d8%ae%d8%b2%d8%a7%d9%85%d9%89-%d8%a7%d9%84%d9%85%d9%86%d8%b9%d8%b4%d8%a9-%d9%84%d9%84%d8%a8%d8%b4%d8%b1%d8%a9-%d9%88%d8%a7%d9%84%d8%b4%d8%b9%d8%b1/?fbclid=IwAR22J-NkYHOL90UeXv2wFNm0BzscyJ5O32pu1I7QSz6BqUqhtc60jFMMrAk
[Thu Feb 20 00:21:53.865271 2020] [wsgi:error] [pid 21842:tid 140155922872064] [client 172.30.2.15:41220] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://mdundo.com/a/35252?fbclid=IwAR2PONsdaHFkTtYYPQCk5V4me_UnDqlIt-Xm2xUGMKRxMGz3XWaOrMSgQZI
[Thu Feb 20 00:21:55.506500 2020] [wsgi:error] [pid 21732:tid 140156164781824] [client 172.30.2.15:41176] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py
[Thu Feb 20 00:21:57.002434 2020] [wsgi:error] [pid 21574:tid 140155939657472] [client 172.30.0.30:1744] mod_wsgi (pid=21574): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://bustednewspaper.com/john-w-hesler-2020-02-19-134100-arrest-booking-mugshot-mason-county-kentucky/
[Thu Feb 20 00:21:59.472819 2020] [wsgi:error] [pid 21732:tid 140156139603712] [client 172.30.0.30:1928] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py
[Thu Feb 20 00:22:00.730918 2020] [reqtimeout:info] [pid 21842:tid 140155939657472] [client 172.30.0.30:1740] AH01382: Request body read timeout
[Thu Feb 20 00:22:00.730933 2020] [wsgi:error] [pid 21842:tid 140155939657472] [client 172.30.0.30:1740] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.dicascaseiras.net/como-fazer-rabanada-assada-confira-passo-a-passo-facil-de-receita-saborosa-com-pao/
[Thu Feb 20 00:22:05.391873 2020] [wsgi:error] [pid 21732:tid 140156031977216] [client 172.30.1.233:59212] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.bibliaonline.com.br/acf/sl/143/6
[Thu Feb 20 00:22:07.816169 2020] [wsgi:error] [pid 21732:tid 140155998406400] [client 172.30.1.233:58842] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.megaatualizado.com/bbb-em-luto/?slot=abb8425ad284b37&utm_campaign=796670736374&utm_medium=abb8425ad284b37&insta=true
[Thu Feb 20 00:22:09.392777 2020] [wsgi:error] [pid 21842:tid 140155922872064] [client 172.30.0.30:2350] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://punchnew.com/senator-dino-melaye-acquires-two-bulletproof-cars-worth-n726-million-watch-videos/?fbclid=IwAR0umJ3fs8dssOBsuXGE_rWNILXvKHi0yR9Mfkrq4FX_woP3I3abyi6YhMQ
[Thu Feb 20 00:22:11.970898 2020] [reqtimeout:info] [pid 21732:tid 140156139603712] [client 172.30.1.233:58906] AH01382: Request body read timeout
[Thu Feb 20 00:22:11.970910 2020] [wsgi:error] [pid 21732:tid 140156139603712] [client 172.30.1.233:58906] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.megaatualizado.com/bbb20-manu-desvenda-indireta-de-leifert-com-a-saida-de-lucas-e-conquista-ainda-mais-fas/?slot=abae6a196602aad&utm_campaign=687061707379&utm_medium=abae6a196602aad&insta=true
[Thu Feb 20 00:22:12.081645 2020] [wsgi:error] [pid 21732:tid 140156189959936] [client 172.30.2.15:41760] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://diariooculto.com/teste/em-cada-um-destes-aneis-revela/3/
[Thu Feb 20 00:22:18.166898 2020] [reqtimeout:info] [pid 21732:tid 140156057155328] [client 172.30.1.233:59244] AH01382: Request body read timeout
[Thu Feb 20 00:22:18.166911 2020] [wsgi:error] [pid 21732:tid 140156057155328] [client 172.30.1.233:59244] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.house-tricks.com/es/14211-v2/1/?utm_source=taboola&utm_campaign=3674827&utm_content=2875743795&utm_medium=tiempodesanjuan
[Thu Feb 20 00:22:18.587547 2020] [wsgi:error] [pid 21732:tid 140156139603712] [client 172.30.0.30:2584] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://androidwaves.com/download-whatsapp-plus/
[Thu Feb 20 00:22:20.286903 2020] [reqtimeout:info] [pid 21574:tid 140155939657472] [client 172.30.0.30:2374] AH01382: Request body read timeout
[Thu Feb 20 00:22:20.286920 2020] [wsgi:error] [pid 21574:tid 140155939657472] [client 172.30.0.30:2374] mod_wsgi (pid=21574): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://bustednewspaper.com/category/nd-mugshots/nd-burleigh/page/5/
[Thu Feb 20 00:22:22.494916 2020] [reqtimeout:info] [pid 21842:tid 140155998406400] [client 172.30.2.15:41740] AH01382: Request body read timeout
[Thu Feb 20 00:22:22.494933 2020] [wsgi:error] [pid 21842:tid 140155998406400] [client 172.30.2.15:41740] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://meaww.com/my-600-lb-life-758-lbs-joyce-weight-loss-surgery-weight-gain-new-episode-dr-nowzaradan-heart-attack?utm_source=taboola&utm_medium=referral&dc_data=1381653_vivo-browser-chrome-english
[Thu Feb 20 00:22:24.889229 2020] [wsgi:error] [pid 21732:tid 140155914479360] [client 172.30.0.30:2550] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.neogaf.com/threads/final-fantasy-7-remake-producer-says-game-has-exceeded-his-expectations.1526763/?fbclid=IwAR1OhwMPBkOQcQmpXTiAJVPHPnVuyjhCM_SbhM0HjLIs9ZuM-fZsvDl9YAI
[Thu Feb 20 00:22:27.231122 2020] [wsgi:error] [pid 21732:tid 140155914479360] [client 172.30.1.233:59844] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://plox.com.br/noticia/19/02/2020/novo-meme-em-ipatinga-viraliza-no-brasil-e-faz-sucesso-nas-redes-sociais?fbclid=IwAR1Ht9y5RD_uNhVxP90utib05o-b4x5gUcIbiPJb8LRzA61W9Vb-lD7c7ZI
[Thu Feb 20 00:22:27.642040 2020] [wsgi:error] [pid 21574:tid 140156057155328] [client 172.30.0.30:2708] mod_wsgi (pid=21574): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://androidwaves.com/download-gbwhatsapp-pro-apk/
[Thu Feb 20 00:22:29.382473 2020] [wsgi:error] [pid 21732:tid 140155922872064] [client 172.30.2.15:42232] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://unum.com.br/solicite/como-solicitar-emprestimo-caixa/
[Thu Feb 20 00:22:30.080351 2020] [wsgi:error] [pid 21732:tid 140155990013696] [client 172.30.0.30:3004] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.kikiers.com/signup?code=JCRtnJvnD&fbclid=IwAR2D80dbucL3ljWawQILpMZaZMqzg_1GLB4-mJot9XFwd31zVGPxA26_obo
[Thu Feb 20 00:22:31.105435 2020] [wsgi:error] [pid 21842:tid 140155964835584] [client 172.30.1.233:59786] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py
[Thu Feb 20 00:22:31.614918 2020] [reqtimeout:info] [pid 21732:tid 140156040369920] [client 172.30.0.30:2720] AH01382: Request body read timeout
[Thu Feb 20 00:22:31.614938 2020] [wsgi:error] [pid 21732:tid 140156040369920] [client 172.30.0.30:2720] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://bustednewspaper.com/category/nd-mugshots/nd-burleigh/page/6/
[Thu Feb 20 00:22:36.045115 2020] [wsgi:error] [pid 21781:tid 140156057155328] [client 172.30.0.30:3138] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.conservativeinstitute.org/conservative-news/pence-tweets-ad-bloomberg.htm?fbclid=IwAR2M1rg-jV3LZXYIv22puIfayorATiReRWWYEht1VQ7AE1tN0YPwxzVXvdc
[Thu Feb 20 00:22:37.814896 2020] [reqtimeout:info] [pid 21842:tid 140156181567232] [client 172.30.0.30:2890] AH01382: Request body read timeout
[Thu Feb 20 00:22:37.814912 2020] [wsgi:error] [pid 21842:tid 140156181567232] [client 172.30.0.30:2890] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://mobatualizado.com/marido-da-atriz-nathalia-dill-aparece-na-tv-pela-primeira-vez-e-publico-fica-incredulo-e-isso-mesmo/?slot=ab2629909aab16c&utm_campaign=6D6E73646362&utm_medium=ab2629909aab16c&insta=true
[Thu Feb 20 00:22:45.119887 2020] [wsgi:error] [pid 21732:tid 140155948050176] [client 172.30.0.30:3178] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.jugantor.com/everyday/268795/%E0%A6%AA%E0%A7%8D%E0%A6%B0%E0%A7%9F%E0%A6%BE%E0%A6%A4-%E0%A6%B0%E0%A6%BE%E0%A6%B7%E0%A7%8D%E0%A6%9F%E0%A7%8D%E0%A6%B0%E0%A6%AA%E0%A6%A4%E0%A6%BF-%E0%A6%9C%E0%A6%BF%E0%A7%9F%E0%A6%BE%E0%A6%89%E0%A6%B0-%E0%A6%B0%E0%A6%B9%E0%A6%AE%E0%A6%BE%E0%A6%A8%E0%A7%87%E0%A6%B0-%E0%A6%9C%E0%A6%A8%E0%A7%8D%E0%A6%AE%E0%A6%A6%E0%A6%BF%E0%A6%A8-%E0%A6%86%E0%A6%9C?fbclid=IwAR27UQtG9M8EzrFH3qMEshSQjIk1N4Wt-BppWuQT7fKsRd-p6G5W7pinTSQ
[Thu Feb 20 00:22:48.685885 2020] [wsgi:error] [pid 21781:tid 140156040369920] [client 172.30.2.15:42724] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: http://www.dailynayadiganta.com/first-page/481118/%E0%A6%AC%E0%A6%BF%E0%A6%B8%E0%A6%BF%E0%A6%8F%E0%A6%B8%E0%A7%87-%E0%A6%AA%E0%A7%8D%E0%A6%B0%E0%A6%AC%E0%A7%87%E0%A6%B6%E0%A7%87-%E0%A6%AC%E0%A7%9F%E0%A6%B8%E0%A6%B8%E0%A7%80%E0%A6%AE%E0%A6%BE-%E0%A6%A8%E0%A6%BF%E0%A7%9F%E0%A7%87-%E0%A6%B9%E0%A6%BE%E0%A6%87%E0%A6%95%E0%A7%8B%E0%A6%B0%E0%A7%8D%E0%A6%9F%E0%A7%87%E0%A6%B0-%E0%A6%B0%E0%A7%81%E0%A6%B2?fbclid=IwAR3-_GtGJ7yYn9WZ0Nb_FK0TZ_pYJBriZ81yW6Iu5uhHtjn4lnpuclkAIPs
[Thu Feb 20 00:22:50.030899 2020] [reqtimeout:info] [pid 21842:tid 140156164781824] [client 172.30.2.15:42316] AH01382: Request body read timeout
[Thu Feb 20 00:22:50.030915 2020] [wsgi:error] [pid 21842:tid 140156164781824] [client 172.30.2.15:42316] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://panlasangpinoy.com/pork-afritada-recipe/
[Thu Feb 20 00:22:50.210915 2020] [reqtimeout:info] [pid 21574:tid 140155939657472] [client 172.30.0.30:3208] AH01382: Request body read timeout
[Thu Feb 20 00:22:50.210935 2020] [wsgi:error] [pid 21574:tid 140155939657472] [client 172.30.0.30:3208] mod_wsgi (pid=21574): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.apost.com/es/blog/un-juez-no-tolera-a-quienes-abusan-de-los-animales-asi-que-les-da-una-dosis-de-su-propia-medicina/22335/?un_id=1582095882583&utm_source=fb_buffer&utm_medium=fb_1638431039720157_nt_es&utm_term=SPAIN_es&utm_campaign=blog_22335&utm_content=858
[Thu Feb 20 00:22:57.009612 2020] [wsgi:error] [pid 21732:tid 140156006799104] [client 172.30.0.30:3246] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.pixilart.com/kirishimashark
[Thu Feb 20 00:22:57.214910 2020] [reqtimeout:info] [pid 21842:tid 140155990013696] [client 172.30.1.233:60320] AH01382: Request body read timeout
[Thu Feb 20 00:22:57.214927 2020] [wsgi:error] [pid 21842:tid 140155990013696] [client 172.30.1.233:60320] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: http://allmedicinedata.info/tag/female-heart-attack-signs/
[Thu Feb 20 00:22:57.375522 2020] [wsgi:error] [pid 21842:tid 140156189959936] [client 172.30.2.15:42864] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.bibliaonline.com.br/acf/busca?q=Falsos+profetas
[Thu Feb 20 00:22:58.675726 2020] [wsgi:error] [pid 21574:tid 140156164781824] [client 172.30.2.15:43082] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.ittefaq.com.bd/sports/131705/%E0%A6%86%E0%A6%B2-%E0%A6%86%E0%A6%AE%E0%A6%BF%E0%A6%A8-%E0%A6%A4%E0%A6%BE%E0%A6%AE%E0%A6%BF%E0%A6%AE%E0%A7%87%E0%A6%B0-%E0%A6%AC%E0%A7%8D%E0%A6%AF%E0%A6%BE%E0%A6%9F%E0%A6%BF%E0%A6%82-%E0%A6%A4%E0%A6%BE%E0%A6%A3%E0%A7%8D%E0%A6%A1%E0%A6%AC%E0%A7%87-%E0%A6%AA%E0%A7%8D%E0%A6%B0%E0%A6%B8%E0%A7%8D%E0%A6%A4%E0%A7%81%E0%A6%A4%E0%A6%BF-%E0%A6%AE%E0%A7%8D%E0%A6%AF%E0%A6%BE%E0%A6%9A-%E0%A6%A1%E0%A7%8D%E0%A6%B0?fbclid=IwAR2jjOymjZV8UzHMKr1Z_-jUXVdO9MSMVBKmD-CRHE_3w8X9cwoRPDnp3-s
[Thu Feb 20 00:23:00.313779 2020] [reqtimeout:info] [pid 21781:tid 140156048762624] [client 172.30.2.15:42676] AH01382: Request body read timeout
[Thu Feb 20 00:23:00.313812 2020] [wsgi:error] [pid 21781:tid 140156048762624] [client 172.30.2.15:42676] mod_wsgi (pid=21781): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.conservativeinstitute.org/
[Thu Feb 20 00:23:00.730854 2020] [wsgi:error] [pid 21842:tid 140156006799104] [client 172.30.0.30:3610] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.apost.com/pt/blog/jimmy-pede-para-celine-imitar-michael-jackson-mas-precisa-parar-o-show-quando-ela-comeca-a-dancar/15753/?fbclid=IwAR2IhhGK2AHvObc1agCLyJao0LtZ2gW7TAS4zguiAENbaT7kyG91gSwql10
[Thu Feb 20 00:23:01.506147 2020] [wsgi:error] [pid 21732:tid 140155990013696] [client 172.30.0.30:3682] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.dnevno.hr/vijesti/zar-se-za-ovo-ginulo-milanovic-nas-je-vec-sad-uvrijedio-psihijatar-ljutit-zbog-inauguracije-1433125/?utm_source=Midas&utm_medium=Widget&utm_campaign=Content%2bexchange
[Thu Feb 20 00:23:04.295404 2020] [wsgi:error] [pid 21842:tid 140156156389120] [client 172.30.1.233:60830] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://bustednewspaper.com/david-ray-winemiller-2020-02-18-120200-arrest-booking-mugshot-hendricks-county-indiana/?fbclid=IwAR3eRd_YJSSqvp8AQ8hHzd5F7un1NxFrU3EUoQCy-PkSQcGq1abFvxYUHRg
[Thu Feb 20 00:23:04.443823 2020] [wsgi:error] [pid 21781:tid 140155990013696] [client 172.30.0.30:4046] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://bustednewspaper.com/david-ray-winemiller-2020-02-18-120200-arrest-booking-mugshot-hendricks-county-indiana/?fbclid=IwAR3eRd_YJSSqvp8AQ8hHzd5F7un1NxFrU3EUoQCy-PkSQcGq1abFvxYUHRg
[Thu Feb 20 00:23:06.014906 2020] [reqtimeout:info] [pid 21842:tid 140156147996416] [client 172.30.1.233:60446] AH01382: Request body read timeout
[Thu Feb 20 00:23:06.014936 2020] [wsgi:error] [pid 21842:tid 140156147996416] [client 172.30.1.233:60446] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://conceptodefinicion.de/tarjeta-madre/
[Thu Feb 20 00:23:13.033934 2020] [wsgi:error] [pid 21574:tid 140155914479360] [client 172.30.1.233:61000] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://sportscastr.com/stream/36553110?fbclid=IwAR2GDJhTzXSsTEsCK0KTFnP4b4OHAKOPkT3GyRqNfvZ2V1ro41L0HjfgKRk
[Thu Feb 20 00:23:13.928368 2020] [wsgi:error] [pid 21732:tid 140155931264768] [client 172.30.1.233:60848] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://dailyhealthpost.com/health-benefits-beets/?fbclid=IwAR302MI5pb8U3MaOYfZ4x7kMYOlzlcnwGyQvD2dqY1ujacVDuUkyhzHIL_w
[Thu Feb 20 00:23:14.264929 2020] [wsgi:error] [pid 21781:tid 140155973228288] [client 172.30.2.15:43684] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://sportscastr.com/stream/36553110?fbclid=IwAR09Ogv7Aj5iZZl60_bOeSldOlgtAv7lQbxNL8LyOGgsUD_2cpK6tBo7O8E
[Thu Feb 20 00:23:15.358840 2020] [wsgi:error] [pid 21781:tid 140156006799104] [client 172.30.2.15:43160] mod_wsgi (pid=21781): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.investorvillage.com/smbd.asp?mb=8852
[Thu Feb 20 00:23:15.790931 2020] [reqtimeout:info] [pid 21842:tid 140156156389120] [client 172.30.1.233:60656] AH01382: Request body read timeout
[Thu Feb 20 00:23:15.790948 2020] [wsgi:error] [pid 21842:tid 140156156389120] [client 172.30.1.233:60656] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://conceptodefinicion.de/sorpresa/
[Thu Feb 20 00:23:16.756194 2020] [wsgi:error] [pid 21574:tid 140156139603712] [client 172.30.0.30:4438] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.demilked.com/110-year-old-tree-library/?fbclid=IwAR3Rt2qy3_TtSO_oT1d2VCa1v657zT2dd4XQEo46A-_2WXcDmtY8zD6VY3s
[Thu Feb 20 00:23:18.714914 2020] [reqtimeout:info] [pid 21732:tid 140155990013696] [client 172.30.0.30:3882] AH01382: Request body read timeout
[Thu Feb 20 00:23:18.714932 2020] [wsgi:error] [pid 21732:tid 140155990013696] [client 172.30.0.30:3882] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.jogalo.com/jogos-de-terror/hello-neighbor.html
[Thu Feb 20 00:23:18.866909 2020] [reqtimeout:info] [pid 21732:tid 140156156389120] [client 172.30.2.15:43292] AH01382: Request body read timeout
[Thu Feb 20 00:23:18.866926 2020] [wsgi:error] [pid 21732:tid 140156156389120] [client 172.30.2.15:43292] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://portalowl.com/como-solicitar-o-cartao-de-credito-credicard-black-zero-anuidade/
[Thu Feb 20 00:23:19.760165 2020] [wsgi:error] [pid 21781:tid 140156006799104] [client 172.30.1.233:61352] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.apost.com/fr/blog/une-femme-ecrit-sa-propre-necrologie-avant-de-mourir-avec-des-conseils-pour-ceux-quelle-a-laisses-derriere-elle/22927/?fbclid=IwAR2eo_l5hxP9nI1xvFB7qdIm5GfTK4zBxDVG2vtPoC5iC-XcS3zVl_PmXJ4
[Thu Feb 20 00:23:20.098946 2020] [wsgi:error] [pid 21781:tid 140156057155328] [client 172.30.1.233:61354] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://doyouremember.com/105394/elderly-playgrounds-boost-activity?fbclid=IwAR0Ob2CpAJ2jJKqNnp48Zkr9SfTD5RwG8OTvDvL201WAVi441eeCMBq5usI
[Thu Feb 20 00:23:20.510913 2020] [reqtimeout:info] [pid 21842:tid 140155981620992] [client 172.30.0.30:3340] AH01382: Request body read timeout
[Thu Feb 20 00:23:20.510930 2020] [wsgi:error] [pid 21842:tid 140155981620992] [client 172.30.0.30:3340] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://mobatualizado.com/marido-da-atriz-nathalia-dill-aparece-na-tv-pela-primeira-vez-e-publico-fica-incredulo-e-isso-mesmo/2/
[Thu Feb 20 00:23:20.766920 2020] [reqtimeout:info] [pid 21781:tid 140155939657472] [client 172.30.1.233:61068] AH01382: Request body read timeout
[Thu Feb 20 00:23:20.766938 2020] [wsgi:error] [pid 21781:tid 140155939657472] [client 172.30.1.233:61068] mod_wsgi (pid=21781): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.boredpanda.com/teen-makes-dress-prom-date-expensive-dream-gown/?isFollow=0&utm_source=newsbreakapp&utm_medium=referral&utm_campaign=organic
[Thu Feb 20 00:23:21.982907 2020] [reqtimeout:info] [pid 21780:tid 140156189959936] [client 172.30.0.30:4198] AH01382: Request body read timeout
[Thu Feb 20 00:23:21.982940 2020] [wsgi:error] [pid 21780:tid 140156189959936] [client 172.30.0.30:4198] mod_wsgi (pid=21780): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.boredpanda.com/teen-makes-dress-prom-date-expensive-dream-gown/?isFollow=0&utm_source=newsbreakapp&utm_medium=referral&utm_campaign=organic
[Thu Feb 20 00:23:22.923314 2020] [wsgi:error] [pid 21574:tid 140156147996416] [client 172.30.0.30:4264] mod_wsgi (pid=21574): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://leismunicipais.com.br/codigo-de-obras-araucaria-pr
[Thu Feb 20 00:23:23.773410 2020] [wsgi:error] [pid 21732:tid 140155939657472] [client 172.30.2.15:43680] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://unum.com.br/cartao-saraiva-internacional-sem-anuidade/?utm_source=activecampaign&utm_medium=email&utm_campaign=newsletter&utm_content=email-396&utm_term=b1&cp=3018
[Thu Feb 20 00:23:28.918926 2020] [reqtimeout:info] [pid 21575:tid 140156015191808] [client 172.30.2.15:43428] AH01382: Request body read timeout
[Thu Feb 20 00:23:28.918952 2020] [wsgi:error] [pid 21575:tid 140156015191808] [client 172.30.2.15:43428] mod_wsgi (pid=21575): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.rotamapas.com.br/como-ir-de-belo-horizonte-a-mato-grosso
[Thu Feb 20 00:23:33.931140 2020] [wsgi:error] [pid 21781:tid 140155948050176] [client 172.30.0.30:5086] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.megaatualizado.com/a-turma-da-monica-tera-conteudo-lgbt-criado-pelo-filho-e-genro-de-mauricio-de-sousa-confira/?fbclid=IwAR1QMUIIMZTkZxQLRtsKZF04J6RfKRiwBvD-BT1soap-VEghH9ntEl1KwwI
[Thu Feb 20 00:23:34.232669 2020] [wsgi:error] [pid 21732:tid 140155931264768] [client 172.30.0.30:5090] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.meups4.com.br/noticias/netflix-resident-evil-em-junho/?fbclid=IwAR3Z0TDeo2VjiYhstt3c127rFAwufVJNzuQhNlg6SApj3gfcBaSVzCQxqkU
[Thu Feb 20 00:23:34.746901 2020] [reqtimeout:info] [pid 21732:tid 140156006799104] [client 172.30.1.233:61374] AH01382: Request body read timeout
[Thu Feb 20 00:23:34.746914 2020] [wsgi:error] [pid 21732:tid 140156006799104] [client 172.30.1.233:61374] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://folhabv.com.br/
[Thu Feb 20 00:23:35.944331 2020] [wsgi:error] [pid 21842:tid 140156181567232] [client 172.30.0.30:5066] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.allgoodwedding.com/pt/14292-fotografia-de-familia/18/?utm_source=taboola&utm_campaign=3356924&utm_medium=segredosdomundo-segredosdomundo&utm_content=2856720182
[Thu Feb 20 00:23:36.282916 2020] [reqtimeout:info] [pid 21574:tid 140156040369920] [client 172.30.2.15:44026] AH01382: Request body read timeout
[Thu Feb 20 00:23:36.282937 2020] [wsgi:error] [pid 21574:tid 140156040369920] [client 172.30.2.15:44026] mod_wsgi (pid=21574): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.factahead.com/pt/15120-papa-beijo/8/?utm_source=taboola&utm_campaign=3815502&utm_medium=portaldoholanda-portaldoholanda&utm_content=2881672882
[Thu Feb 20 00:23:36.313809 2020] [wsgi:error] [pid 21732:tid 140155948050176] [client 172.30.1.233:61686] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: http://mzamin.com/article.php?mzamin=213142&cat=5&fbclid=IwAR0i0cxb-jvZ-FEvn_lvtl0oCOP1HH-56hM6nvVTdUKTAw4VqNosQIrtUTA
[Thu Feb 20 00:23:37.110504 2020] [wsgi:error] [pid 21842:tid 140156173174528] [client 172.30.2.15:44356] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.megaatualizado.com/andressa-suita-reage-apos-estado-de-saude-da-ex-bbb-aline-ser-revelado-chorando/?slot=abba621d4b5da4c&utm_campaign=6A70736B6275&utm_medium=abba621d4b5da4c&insta=true
[Thu Feb 20 00:23:37.558906 2020] [reqtimeout:info] [pid 21842:tid 140156015191808] [client 172.30.0.30:4782] AH01382: Request body read timeout
[Thu Feb 20 00:23:37.558920 2020] [wsgi:error] [pid 21842:tid 140156015191808] [client 172.30.0.30:4782] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://bustednewspaper.com/evette-marie-seewalker-2020-02-17-232500-arrest-booking-mugshot-burleigh-county-north-dakota/
[Thu Feb 20 00:23:39.314894 2020] [reqtimeout:info] [pid 21842:tid 140156040369920] [client 172.30.0.30:4846] AH01382: Request body read timeout
[Thu Feb 20 00:23:39.314909 2020] [wsgi:error] [pid 21842:tid 140156040369920] [client 172.30.0.30:4846] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.factahead.com/pt/13624-miss-belvedere/2/?utm_source=taboola&utm_campaign=3815525&utm_medium=movimentocountry-movimentocountry&utm_content=2881680495
[Thu Feb 20 00:23:45.512219 2020] [wsgi:error] [pid 21781:tid 140155956442880] [client 172.30.0.30:5236] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.apost.com/sv/blog/kvinnans-funktionshindrade-hund-som-stals-med-hennes-bil-upphittades-dod/23736/?fbclid=IwAR019hAS5zXnf5w5jZQzXKOdcDJ0I7GfAWhg6EHiKUIFY-wY_s__FodGA2w
[Thu Feb 20 00:23:46.923959 2020] [wsgi:error] [pid 21842:tid 140156189959936] [client 172.30.0.30:5492] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.bitelists.com/es/10-senales-del-cuerpo-que-nunca-deberias-ignorar/3/
[Thu Feb 20 00:23:50.213908 2020] [wsgi:error] [pid 21732:tid 140156023584512] [client 172.30.1.233:62216] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://www.lifehack.org/articles/money/how-maximize-your-return-freelance-work.html?fbclid=IwAR02Z20pPIWrNcCIv1QDPR0cJElsOCAbjcOlDpb8-6iHgHdX8sJmu4UN_H4
[Thu Feb 20 00:23:50.814011 2020] [wsgi:error] [pid 21732:tid 140155939657472] [client 172.30.2.15:44472] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://unum.com.br/cartao-saraiva-internacional-sem-anuidade/?utm_source=activecampaign&utm_medium=email&utm_campaign=newsletter&utm_content=email-396&utm_term=b1&cp=3018
[Thu Feb 20 00:23:51.846907 2020] [reqtimeout:info] [pid 21842:tid 140155998406400] [client 172.30.1.233:61668] AH01382: Request body read timeout
[Thu Feb 20 00:23:51.846924 2020] [wsgi:error] [pid 21842:tid 140155998406400] [client 172.30.1.233:61668] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.megaatualizado.com/bbb20-lucas-diz-que-deixou-surpresa-para-brothers-e-revela-a-casa-vai-feder/?slot=abae6a196602aad&utm_campaign=79626175716F&utm_medium=abae6a196602aad&insta=true
[Thu Feb 20 00:23:54.064660 2020] [wsgi:error] [pid 21732:tid 140156156389120] [client 172.30.2.15:44684] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://buhamster.com/br/20-pontes-de-tirar-o-folego-que-irao-fazer-o-seu-queixo-cair-.html/page4
[Thu Feb 20 00:23:57.323169 2020] [wsgi:error] [pid 21574:tid 140156139603712] [client 172.30.0.30:5742] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: https://m.rtvonline.com/bangladesh?fbclid=IwAR1_UNmqZJxWxCbUQQOba-lODPV5gU_wG24B-xrkcBDr3CXIbWSPX8OUOgM
[Thu Feb 20 00:23:59.290906 2020] [reqtimeout:info] [pid 21732:tid 140155981620992] [client 172.30.0.30:5556] AH01382: Request body read timeout
[Thu Feb 20 00:23:59.290922 2020] [wsgi:error] [pid 21732:tid 140155981620992] [client 172.30.0.30:5556] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://plox.com.br/noticia/19/02/2020/luciano-camargo-comemora-melhora-do-pai-estava-com-medo-e-pronto-para-me
[Thu Feb 20 00:23:59.785563 2020] [wsgi:error] [pid 21574:tid 140156147996416] [client 172.30.2.15:44520] mod_wsgi (pid=21574): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://buzzdrives.com/50-autos-que-estan-cayendo-en-valor/8/
[Thu Feb 20 00:24:01.674549 2020] [wsgi:error] [pid 21842:tid 140156006799104] [client 172.30.1.233:62568] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.boredpanda.com/
[Thu Feb 20 00:24:04.208000 2020] [wsgi:error] [pid 21732:tid 140155998406400] [client 172.30.2.15:44590] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://diariooculto.com/beleza/oleo/5/
[Thu Feb 20 00:24:07.869366 2020] [reqtimeout:info] [pid 21732:tid 140156147996416] [client 172.30.1.233:62304] AH01382: Request body read timeout
[Thu Feb 20 00:24:07.869385 2020] [wsgi:error] [pid 21732:tid 140156147996416] [client 172.30.1.233:62304] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://conceptodefinicion.de/
[Thu Feb 20 00:24:08.030898 2020] [reqtimeout:info] [pid 21842:tid 140156156389120] [client 172.30.1.233:62378] AH01382: Request body read timeout
[Thu Feb 20 00:24:08.030915 2020] [wsgi:error] [pid 21842:tid 140156156389120] [client 172.30.1.233:62378] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://descomundo.com.br/bbb-em-luto/2/
[Thu Feb 20 00:24:10.392844 2020] [wsgi:error] [pid 21574:tid 140155964835584] [client 172.30.0.30:6198] Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.: /home/ml/serve-ml-dev/serve-ml/serve_ml/serve_ml/wsgi.py, referer: http://www.dainikamadershomoy.com/post/242562
[Thu Feb 20 00:24:12.539566 2020] [wsgi:error] [pid 21732:tid 140156147996416] [client 172.30.0.30:6244] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://www.laprensa.com.ni/2020/02/18/hoy/2641893-hombre-mata-a-su-madre-a-pedradas-en-matagalpa-porque-ella-se-nego-a-darle-dinero-para-seguir-tomando-licor
[Thu Feb 20 00:24:20.601149 2020] [reqtimeout:info] [pid 21575:tid 140155922872064] [client 172.30.2.15:45356] AH01382: Request body read timeout
[Thu Feb 20 00:24:20.601182 2020] [wsgi:error] [pid 21575:tid 140155922872064] [client 172.30.2.15:45356] mod_wsgi (pid=21575): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.megaatualizado.com/bbb20-participante-chora-de-fome-e-globo-e-acusada-de-todas-as-formas-desumano/?slot=abcf071f05f7dda&utm_campaign=726966656875&utm_medium=abcf071f05f7dda&insta=true
[Thu Feb 20 00:24:28.290921 2020] [reqtimeout:info] [pid 21842:tid 140155964835584] [client 172.30.2.15:45602] AH01382: Request body read timeout
[Thu Feb 20 00:24:28.290937 2020] [wsgi:error] [pid 21842:tid 140155964835584] [client 172.30.2.15:45602] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.boredpanda.com/worst-photoshop-fails/
[Thu Feb 20 00:24:34.234914 2020] [reqtimeout:info] [pid 21732:tid 140156164781824] [client 172.30.0.30:6470] AH01382: Request body read timeout
[Thu Feb 20 00:24:34.234928 2020] [wsgi:error] [pid 21732:tid 140156164781824] [client 172.30.0.30:6470] mod_wsgi (pid=21732): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.bright-side.life/pt/13888/?utm_source=taboola&utm_campaign=3406448&utm_medium=brasil247&utm_content=2859408621
[Thu Feb 20 00:24:44.279376 2020] [wsgi:error] [pid 21781:tid 140156057155328] [client 172.30.0.30:7154] mod_wsgi (pid=21781): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: https://senhorfinancas.com/content/m2-m/os-30-brasileiros-mais-ricos-do-mundo/16/?utm_source=fb&utm_campaign=23844152138870012&utm_content=23844152138920012&utm_medium=Facebook_Mobile_Feed
[Thu Feb 20 00:24:48.366901 2020] [reqtimeout:info] [pid 21575:tid 140156057155328] [client 172.30.0.30:7042] AH01382: Request body read timeout
[Thu Feb 20 00:24:48.366916 2020] [wsgi:error] [pid 21575:tid 140156057155328] [client 172.30.0.30:7042] mod_wsgi (pid=21575): Request data read error when proxying data to daemon process: The timeout specified has expired., referer: https://www.carnotion.com/50-cars-that-are-plummeting-in-value/38/?tict=1

[Graham Dumpleton]

One comment about the errors:

    Request data read error when proxying data to daemon process: The timeout specified has expired

in case you don't understand the likely source.

This can be caused by slow HTTP clients or clients coming over a bad network where the connection hangs or is lost. This is especially relevant to traffic from mobile phones.

The consequences of this can be a problem if it occurs with high incidence as it robs you of server capacity. The timeout message indicates that the request handler thread was occupied (60 seconds based on your config) waiting for request content that never ended up getting delivered and so the request ended up being discarded. Obviously if you have a lot of these occurring, means you end up having to set capacity higher.

Playing with socket-timeout and setting it lower might aid in releasing the request handler thread sooner by timing out sooner, but you would need to be careful in how far you reduce this as it could have implications on the response side, as it sets timeout for both read and write on socket between Apache child processes and daemon process. You may want to at the same time set response-socket-timeout so you can control them separately. So set response-socket-timeout to 60, but reduce socket-timeout to 30. I will need to go look at the code to remind myself exactly how these work to confirm though.

An alternative to dealing with slow clients or clients that hang may be to use the mod_reqtimeout module instead.

    https://httpd.apache.org/docs/2.4/mod/mod_reqtimeout.html

This allows you to better control the timeout and also look at rate of transfer as trigger for failing when reading request content.

For example, in mod_wsgi-express, I set this up by default as:

    RequestReadTimeout header=15-30,MinRate=500 body=15,MinRate=500

How you configure this really depends on what sort of post/upload your application handles. You may want to even set this differently based on URL patterns.

Anyway, the aim here is to try and identify stuck requests which will eventually just fail and kick them out sooner, and so reduce their impact on capacity available.

A second thing to have a think about is what I talk about in:

    http://blog.dscpl.com.au/2014/02/vertically-partitioning-python-web.html

The idea in that is that a web application has a diverse set of URLs and don't all have the same performance characteristics or memory requirements.

Taking an example. Imagine that 50% of your requests are to your home page, that it has a quick response time, and doesn't load that much data or does anything that causes lots of memory to be used.

What you can do is use multiple mod_wsgi daemon process groups, with separate instances of your application. You would direct all requests for your home page to the instance in one mod_wsgi daemon process group. All other requests would go to the other.

If the instance which handled just requests for the home page only required 150MB of memory instead of 400MB, then you can tune how many process required in that daemon process group separately. Thus, drop the number of process of the more memory hungry processes handling all the other requests, and beef up the number for handling the home page. Overall the memory usage could be less, as you don't have memory sitting there used in a process, just in case a memory hungry URL may be hit.

At the other end of the spectrum, if you had a particular set of URLs (eg admin pages), which were rarely used by chewed up lots of memory, you could separate them into its own daemon process group, use just a single process, and also set up the process to be recycled every so often so the memory is released.

Anyway, have a read of the article and see if this idea of splitting your application across multiple daemon process groups, where each is tuned separately for that set of URLs, makes any sense in your case. If you can, then careful attention to this can be a good way of reducing memory usage for what you have, allowing you to then spin up more processes to handle just the URLs which see high traffic but may use less memory by themselves.

I'll be interested to hear if you think you may be able to start partitioning up things based on URLs allowing you to tune for things in this way.

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/3fd25b45-c464-4e0b-a179-a2756b939e1f%40googlegroups.com.

[Jason Garber]

Another point here is we always run nginx in front of mod_wsgi to handle slow clients.  Nginx buffers requests and responses up to 1MB which dramatically reduces number of wsgi threads needed.

Nginx is stellar at handling tons of  concurrent clients. 

So... nginx owns port 80 and 443

Apache on 127.0.0.1:45667 or whatever

Nginx served all statics directly

And uses proxy_pass to send dynamic requests to mod_wsgi. 

It works really well.

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/C3D28645-2CA9-4C8E-961E-B8194FADAF98%40gmail.com.

[Andrew Charles]

We don't have any static files besides robots.txt as it's only an API, I've considered using nginx proxy but I didn't want to add another dependency to manage yet.

Splitting the processes by url does make sense, we do have one endpoint that requires 5x memory as the others. I'm hesitant as it adds another thing to monitor but I'll think about it.

New memory numbers after procs/threads 10/25: wsgi ~2400M VIRT ~420M RES apache ~2000M VIRT ~10000 RES system at 4.3G/7.5G Could possibly add more procs and use less threads

I've pared down the apache modules:

alias
authz_core
authz_host
deflate
dir
filter
mime
mpm_event
reqtimeout
setenvif
status
wsgi

and updated the .conf:

WSGIRestrictEmbedded On
WSGIPythonOptimize 1

<VirtualHost *:80>
    ServerName name.com
    ServerAlias name2.com
    ServerAdmin email

    LogFormat "%h %t \"%r\" %>s \"%{Referer}i\"" short
    CustomLog /var/log/apache2/other_vhosts_access.log short

    # AWS ELB Target Group Health Checker
    <Location /hc>
        Require all granted
    </Location>

 

    SetEnvIfNoCase Host name\.com VALID_AHOST
    <Location /server-status>
        SetHandler server-status
        Require env VALID_AHOST
    </Location>

    # for Require host
    SetEnvIfNoCase Host name(2)?\.com VALID_HOST

    Alias /robots.txt ...robots.txt
    <Location /robots.txt>
        Require env VALID_HOST
    </Location>

    RequestReadTimeout header=15-30,MinRate=500 body=15,MinRate=500

    WSGIScriptAlias / ...wsgi.py
    WSGIDaemonProcess name \
        display-name='%{GROUP}' \
        processes=10 \
        threads=25 \
        queue-timeout=45 \
        socket-timeout=30 \
        response-socket-timeout=60 \

        request-timeout=60 \
        inactivity-timeout=0 \
        startup-timeout=45 \
        deadlock-timeout=60 \
        graceful-timeout=15 \

        eviction-timeout=0 \
        python-path=...dir \
        python-home=...venv
    WSGIProcessGroup name
    WSGIApplicationGroup %{GLOBAL}
    <Directory ...dir/dir>
        <Files wsgi.py>
            Require env VALID_HOST
        </Files>
    </Directory>
</VirtualHost>

I also took out some superfluous packages from requirements.txt as well as removed some background workers from django-q

On Wednesday, February 19, 2020 at 6:14:05 PM UTC-7, Graham Dumpleton wrote:

One comment about the errors:

    Request data read error when proxying data to daemon process: The timeout specified has expired

in case you don't understand the likely source.

This can be caused by slow HTTP clients or clients coming over a bad network where the connection hangs or is lost. This is especially relevant to traffic from mobile phones.

The consequences of this can be a problem if it occurs with high incidence as it robs you of server capacity. The timeout message indicates that the request handler thread was occupied (60 seconds based on your config) waiting for request content that never ended up getting delivered and so the request ended up being discarded. Obviously if you have a lot of these occurring, means you end up having to set capacity higher.

Playing with socket-timeout and setting it lower might aid in releasing the request handler thread sooner by timing out sooner, but you would need to be careful in how far you reduce this as it could have implications on the response side, as it sets timeout for both read and write on socket between Apache child processes and daemon process. You may want to at the same time set response-socket-timeout so you can control them separately. So set response-socket-timeout to 60, but reduce socket-timeout to 30. I will need to go look at the code to remind myself exactly how these work to confirm though.

An alternative to dealing with slow clients or clients that hang may be to use the mod_reqtimeout module instead.

    https://httpd.apache.org/docs/2.4/mod/mod_reqtimeout.html

This allows you to better control the timeout and also look at rate of transfer as trigger for failing when reading request content.

For example, in mod_wsgi-express, I set this up by default as:

    RequestReadTimeout header=15-30,MinRate=500 body=15,MinRate=500

How you configure this really depends on what sort of post/upload your application handles. You may want to even set this differently based on URL patterns.

Anyway, the aim here is to try and identify stuck requests which will eventually just fail and kick them out sooner, and so reduce their impact on capacity available.

A second thing to have a think about is what I talk about in:

    http://blog.dscpl.com.au/2014/02/vertically-partitioning-python-web.html

The idea in that is that a web application has a diverse set of URLs and don't all have the same performance characteristics or memory requirements.

Taking an example. Imagine that 50% of your requests are to your home page, that it has a quick response time, and doesn't load that much data or does anything that causes lots of memory to be used.

What you can do is use multiple mod_wsgi daemon process groups, with separate instances of your application. You would direct all requests for your home page to the instance in one mod_wsgi daemon process group. All other requests would go to the other.

If the instance which handled just requests for the home page only required 150MB of memory instead of 400MB, then you can tune how many process required in that daemon process group separately. Thus, drop the number of process of the more memory hungry processes handling all the other requests, and beef up the number for handling the home page. Overall the memory usage could be less, as you don't have memory sitting there used in a process, just in case a memory hungry URL may be hit.

At the other end of the spectrum, if you had a particular set of URLs (eg admin pages), which were rarely used by chewed up lots of memory, you could separate them into its own daemon process group, use just a single process, and also set up the process to be recycled every so often so the memory is released.

Anyway, have a read of the article and see if this idea of splitting your application across multiple daemon process groups, where each is tuned separately for that set of URLs, makes any sense in your case. If you can, then careful attention to this can be a good way of reducing memory usage for what you have, allowing you to then spin up more processes to handle just the URLs which see high traffic but may use less memory by themselves.

I'll be interested to hear if you think you may be able to start partitioning up things based on URLs allowing you to tune for things in this way.

Graham

...

[Jason Garber]

I'd run nginx for the buffering aspects unless you already have that in place.

Let me know if you want a full config.

[Thu Feb 20 00:21:10.561797 2020] [wsgi:error] [pid 21842:tid 140155956442880] [client 172.30.1.233:57310] mod_wsgi (pid=21842): Request data read error when proxying data to daemon process: Partial results are valid but processing is incomplete., referer: <a href="https://www.megaatualizado.com/apos-relacionamento-com-o-jogador-ex-de-neymar-surge-gravida-e-anuncia-a-chegada-do-herdeiro/2/" target="_blank" rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.megaatualizado.com%2Fapos-relacionamento-com-o-jogador-ex-de-neymar-surge-gravida-e-anuncia-a-chegada-do-herdeiro%2F2%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGl

[Andrew Charles]

That would be appreciated.

On Friday, February 21, 2020 at 8:06:02 AM UTC-7, Jason Garber wrote:

I'd run nginx for the buffering aspects unless you already have that in place.

Let me know if you want a full config.

[Andrew Charles]

Attempted to split urls via processes, however I need to send an environment variable from apache to django to disable the code utilising unneeded memory. In this post https://stackoverflow.com/questions/19754834/access-apache-setenv-variable-from-django-wsgi-py-file you say not to do this however I need to pass something.

On Wednesday, February 19, 2020 at 6:14:05 PM UTC-7, Graham Dumpleton wrote:

One comment about the errors:

    Request data read error when proxying data to daemon process: The timeout specified has expired

in case you don't understand the likely source.

This can be caused by slow HTTP clients or clients coming over a bad network where the connection hangs or is lost. This is especially relevant to traffic from mobile phones.

The consequences of this can be a problem if it occurs with high incidence as it robs you of server capacity. The timeout message indicates that the request handler thread was occupied (60 seconds based on your config) waiting for request content that never ended up getting delivered and so the request ended up being discarded. Obviously if you have a lot of these occurring, means you end up having to set capacity higher.

Playing with socket-timeout and setting it lower might aid in releasing the request handler thread sooner by timing out sooner, but you would need to be careful in how far you reduce this as it could have implications on the response side, as it sets timeout for both read and write on socket between Apache child processes and daemon process. You may want to at the same time set response-socket-timeout so you can control them separately. So set response-socket-timeout to 60, but reduce socket-timeout to 30. I will need to go look at the code to remind myself exactly how these work to confirm though.

An alternative to dealing with slow clients or clients that hang may be to use the mod_reqtimeout module instead.

    https://httpd.apache.org/docs/2.4/mod/mod_reqtimeout.html

This allows you to better control the timeout and also look at rate of transfer as trigger for failing when reading request content.

For example, in mod_wsgi-express, I set this up by default as:

    RequestReadTimeout header=15-30,MinRate=500 body=15,MinRate=500

How you configure this really depends on what sort of post/upload your application handles. You may want to even set this differently based on URL patterns.

Anyway, the aim here is to try and identify stuck requests which will eventually just fail and kick them out sooner, and so reduce their impact on capacity available.

A second thing to have a think about is what I talk about in:

    http://blog.dscpl.com.au/2014/02/vertically-partitioning-python-web.html

The idea in that is that a web application has a diverse set of URLs and don't all have the same performance characteristics or memory requirements.

Taking an example. Imagine that 50% of your requests are to your home page, that it has a quick response time, and doesn't load that much data or does anything that causes lots of memory to be used.

What you can do is use multiple mod_wsgi daemon process groups, with separate instances of your application. You would direct all requests for your home page to the instance in one mod_wsgi daemon process group. All other requests would go to the other.

If the instance which handled just requests for the home page only required 150MB of memory instead of 400MB, then you can tune how many process required in that daemon process group separately. Thus, drop the number of process of the more memory hungry processes handling all the other requests, and beef up the number for handling the home page. Overall the memory usage could be less, as you don't have memory sitting there used in a process, just in case a memory hungry URL may be hit.

At the other end of the spectrum, if you had a particular set of URLs (eg admin pages), which were rarely used by chewed up lots of memory, you could separate them into its own daemon process group, use just a single process, and also set up the process to be recycled every so often so the memory is released.

Anyway, have a read of the article and see if this idea of splitting your application across multiple daemon process groups, where each is tuned separately for that set of URLs, makes any sense in your case. If you can, then careful attention to this can be a good way of reducing memory usage for what you have, allowing you to then spin up more processes to handle just the URLs which see high traffic but may use less memory by themselves.

I'll be interested to hear if you think you may be able to start partitioning up things based on URLs allowing you to tune for things in this way.

Graham

...

[Graham Dumpleton]

You can do something like the following at global scope in module code:

    import mod_wsgi

    if mod_wsgi.process_group == 'xxx':

        ...

In other words, you could work it out based on the name of the mod_wsgi daemon process group.

That is the easiest thing to do.

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/5cbd894c-78bb-4c4a-ae50-58036451e6cc%40googlegroups.com.

[Andrew Charles]

In my project's wsgi.py I added the following:

mod_wsgi = None
try:
    import mod_wsgi
except ImportError:
    pass  # local dev, ignore
if mod_wsgi:
    os.environ['DISABLE'] = 'True'
    if mod_wsgi.process_group == 'no-disable':
        os.environ['DISABLE'] = 'False'

I forgot I had done something similar on another project.

On Friday, February 21, 2020 at 12:55:44 PM UTC-7, Graham Dumpleton wrote:

You can do something like the following at global scope in module code:

    import mod_wsgi

    if mod_wsgi.process_group == 'xxx':

        ...

In other words, you could work it out based on the name of the mod_wsgi daemon process group.

That is the easiest thing to do.

Graham

...

[Andrew Charles]

It's working well, I've been able to up the processes by a lot due to the reduced memory usage. procs/threads disable 14/20 and not-disable 6/20 using 4.5GB/7.5GB memory.

 

WSGIRestrictEmbedded On
WSGIPythonOptimize 1

<VirtualHost *:80>
    ServerName name

    LogFormat "%h %t \"%r\" %>s \"%{Referer}i\" %{ms}T" short

    CustomLog /var/log/apache2/other_vhosts_access.log short

    # AWS ELB Target Group Health Checker
    <Location /hc>
        Require all granted
    </Location>

    SetEnvIfNoCase Host name\.com VALID_AHOST
    <Location /server-status>
        SetHandler server-status
        Require env VALID_AHOST
    </Location>

    # for Require host
    SetEnvIfNoCase Host name(2)?\.com VALID_HOST

    Alias /robots.txt ...robots.txt
    <Location /robots.txt>
        Require env VALID_HOST
    </Location>

    RequestReadTimeout header=15-30,MinRate=500 body=15,MinRate=500

    WSGIScriptAlias / ...wsgi.py

    WSGIDaemonProcess disable \
        display-name='%{GROUP}' \
        processes=14 \
        threads=20 \
        queue-timeout=45 \
        socket-timeout=30 \
        response-socket-timeout=60 \

        request-timeout=60 \
        inactivity-timeout=0 \
        startup-timeout=45 \
        deadlock-timeout=60 \
        graceful-timeout=15 \

        eviction-timeout=0 \
        python-path=...dir \
        python-home=...venv
    WSGIDaemonProcess not-disable \
        display-name='%{GROUP}' \
        processes=6 \
        threads=20 \
        queue-timeout=45 \
        socket-timeout=30 \
        response-socket-timeout=60 \

        request-timeout=60 \
        inactivity-timeout=0 \
        startup-timeout=45 \
        deadlock-timeout=60 \
        graceful-timeout=15 \

        eviction-timeout=0 \
        python-path=...dir \
        python-home=...venv
    WSGIProcessGroup disable

    WSGIApplicationGroup %{GLOBAL}
    <Directory ...dir/dir>
        <Files wsgi.py>
            Require env VALID_HOST
        </Files>
    </Directory>

    <Location /not-disable>
        WSGIProcessGroup not-disable
    </Location>
</VirtualHost>

On Friday, February 21, 2020 at 12:55:44 PM UTC-7, Graham Dumpleton wrote:

You can do something like the following at global scope in module code:

    import mod_wsgi

    if mod_wsgi.process_group == 'xxx':

        ...

In other words, you could work it out based on the name of the mod_wsgi daemon process group.

That is the easiest thing to do.

Graham

...

[Graham Dumpleton]

Sounds great.

Next thing would like to investigate if interested to go further, is to try and get better visibility into how things are performing. This would be by making use of the mod_wsgi events/metrics to try and track server capacity utilisation, cpu load etc. This would be dependent on working out how we can generate the information in a form that is consumable by you in how your system is setup and with whatever monitoring systems you may have. Also working out what will be worthwhile to capture and would provide some value.

Anyway, have a read of:

http://blog.dscpl.com.au/2015/06/implementing-request-monitoring-within.html

so you can see what the events/metrics stuff is I am talking about. No need to try anything with your actual application. I will have to track down some extra code I had which did some extra processing of the raw data to work out stuff like server capacity utilisation. I hope I haven't lost the code in various laptop swaps.

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/8be4c9d1-e960-4a7f-8725-adddb21303dd%40googlegroups.com.

[Andrew Charles]

I would very much like to implement that. I'd probably setup one of the instances to track metrics to a file and do analysis on the file after a few hours (or a day) of gathering. Where would I place the event handler?

On Monday, February 24, 2020 at 7:56:35 PM UTC-7, Graham Dumpleton wrote:

Sounds great.

Next thing would like to investigate if interested to go further, is to try and get better visibility into how things are performing. This would be by making use of the mod_wsgi events/metrics to try and track server capacity utilisation, cpu load etc. This would be dependent on working out how we can generate the information in a form that is consumable by you in how your system is setup and with whatever monitoring systems you may have. Also working out what will be worthwhile to capture and would provide some value.

Anyway, have a read of:

http://blog.dscpl.com.au/2015/06/implementing-request-monitoring-within.html

so you can see what the events/metrics stuff is I am talking about. No need to try anything with your actual application. I will have to track down some extra code I had which did some extra processing of the raw data to work out stuff like server capacity utilisation. I hope I haven't lost the code in various laptop swaps.

Graham

...

[Jason Garber]

Hi Andrew,

I've included below a full nginx + apache config for a development site (which aside from NOT caching static assets because it's a dev site) is similar to a production config.

A couple things to note which you may not use:

• .well-known/acme-challenge/ has to do with Lets Encrypt SSL verification
• All http requests are redirected to https
• All requests with an extension (e.g. .png, .jpg, etc...) are served directly by nginx
  
• All other requests (e.g. /user/detail) are proxied to apache
• Apache assumes all requests are targeted for mod_wsgi

Let me know if you have any questions.

Other options in nginx can be used to tune buffering of request/response bodies.

#######################################################################################################################################################

# Instance GiftHopper-2-dev-jason Main_nginx_conf Config

# Redirect http traffic to https://
server
{
  listen 192.168.2.241:80;
  server_name gifthopper.jason.dev.appcove.net;
  access_log /home/jason/DevLevel.2/GiftHopper/Run/GiftHopper-2-dev-jason.nginx-access.log main;
 
  location /.well-known/acme-challenge/ {root /home/deploy/ServerDocumentRoot;}
 
  location /
  {
    return 302 https://$host$request_uri;
  }
}

server
{
  listen 192.168.2.241:443 ssl;
 
  server_name gifthopper.jason.dev.appcove.net;

  error_page 503 @maintenance;
  location @maintenance {
    root  /home/jason/DevLevel.2/GiftHopper/Web/Main;
    rewrite ^(.*)$ /ACRM2/503.html break;
  }

  ssl_certificate     /home/jason/DevLevel.2/GiftHopper/Run/file-hash-c7a8ce6fcedae1c5c2cc27b38d94e5e6589871e2.crt.pem;
  ssl_certificate_key /home/jason/DevLevel.2/GiftHopper/Run/file-hash-f853a504461333a646ef9b263e8b6d05dc8d3f0d.key.pem;

  location /.well-known/acme-challenge/ {root /home/deploy/ServerDocumentRoot;}

  access_log /home/jason/DevLevel.2/GiftHopper/Run/GiftHopper-2-dev-jason.nginx-access.log main;

  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-ConfKey 'Main_Domain';
  proxy_set_header Scheme $scheme;

  client_max_body_size 128m;
   
  root  /home/jason/DevLevel.2/GiftHopper/Web/Main;
 
  location ~ \.(py|pyc|pyo|wsgi)$
  {
    return 403;
  }   

  location ~ \.([a-zA-Z0-9])+$
  {    
    add_header Cache-Control 'no-cache, no-store, max-age=0, must-revalidate';
    add_header Expires 'Thu, 01 Jan 1970 00:00:01 GMT';    
  }
 
  # Handle .well-known stuff directly in nginx
  location /.well-known
  {
    add_header Cache-Control 'no-cache, no-store, max-age=0, must-revalidate';
    add_header Expires 'Thu, 01 Jan 1970 00:00:01 GMT';
  }

  location /
  {
    # Never cache dynamic requests.
    add_header Cache-Control 'no-cache, no-store, max-age=0, must-revalidate';
    add_header Expires 'Thu, 01 Jan 1970 00:00:01 GMT';
    proxy_pass http://127.0.0.1:60734;
  }

}

#######################################################################################################################################################

# Instance GiftHopper-2-dev-jason Async_httpd_conf Config
Listen 127.0.0.1:60735

# Async background process

<VirtualHost 127.0.0.1:60735>
  ServerName _default_;
  WSGIDaemonProcess async-60735 processes=1 threads=1 python-path=/home/jason/DevLevel.2/GiftHopper/Python:/home/jason/DevLevel.2/GiftHopper/PythonUserBase/lib/python3.6/site-packages display-name=async-60735
  WSGIImportScript /home/jason/DevLevel.2/GiftHopper/Async/wsgi_import_script.wsgi process-group=async-60735 application-group=%{GLOBAL}
  ErrorLog /home/jason/DevLevel.2/GiftHopper/Run/GiftHopper-2-dev-jason.Async.log
</VirtualHost>

# Instance GiftHopper-2-dev-jason WebProcess_httpd_conf Config

# Main WSGI Process
WSGIDaemonProcess GiftHopper-2-dev-jason-WebProcess processes=2 threads=2 python-path=/home/jason/DevLevel.2/GiftHopper/Python:/home/jason/DevLevel.2/GiftHopper/PythonUserBase/lib/python3.6/site-packages display-name=wsgi-GiftHopper-2-dev-jason-WebProcess

# Instance GiftHopper-2-dev-jason Main_httpd_conf Config
<Directory "/home/jason/DevLevel.2/GiftHopper/Web/Main">
  AllowOverride None
  Require all granted
</Directory>

Listen 127.0.0.1:60734

<VirtualHost 127.0.0.1:60734>
  ServerName _default_
  DocumentRoot /home/jason/DevLevel.2/GiftHopper/Web/Main
  AddDefaultCharset UTF-8

  RewriteEngine on
  RewriteOptions inherit

  # Forbid any python source files from being served.
  RewriteRule \.(py|pyc|pyo|wsgi)$  -  [F]

  WSGIScriptAlias /API/V1 /home/jason/DevLevel.2/GiftHopper/Web/Main/API/V1/__init__.wsgi
  WSGIScriptAlias / /home/jason/DevLevel.2/GiftHopper/Web/Main/__init__.wsgi
  WSGIProcessGroup GiftHopper-2-dev-jason-WebProcess

  LogLevel info
  ErrorLog /home/jason/DevLevel.2/GiftHopper/Run/GiftHopper-2-dev-jason.WebProcess.log
</VirtualHost>