Django/uWSGI/nginx under load

[Andy Robinson]

Hi all,

We're maintaining an application that is hitting scaling problems.   It helps run grass-roots sporting events all over Europe, so Saturdays can produce large and unpredictable loads, with both logged-in users and public viewers. Despite a powerful server seemingly being lightly loaded (htop suggests just 3-6 out of24 cores typically busy, and free memory available), it often "hangs" for long periods - simple requests return after 10sec, and at times we have had to restart nginx.   

If anyone here has experience of this configuration under high loads, we'd be keen to hire an experienced expert to review and check we are not doing anything stupid with our nginx or uwsgi configuration.  

Likewise, if you really know how to cache Django with nginx, and work around the various cookies and language settings so that logged-in pages still work but anonymous ones can be cached, we'd love to get a bit of help  

Architecture is ubuntu 18.04, Django 3.0.12, with MariaDB and Mongo back ends used for slightly different things.

Feel free to contact me by private email if you,

Otherwise, if people have suggestions and questions here, I'll try to share what I can and am very grateful for any help...

Best Regards

Andy Robinson

 

[Aldian Fazrihady]

Check the number of open file descriptor when you experienced the slow response.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/4fcc006e-37bb-4497-a692-45dfb99bfad1n%40googlegroups.com.

--

Regards,

Aldian Fazrihady
http://aldianfazrihady.com

[John McClain]

are you using django directly to run the queries or are you querying the db. I had a similar problem a while back with a project and the load was resolved by letting python do the work.

--

You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/4fcc006e-37bb-4497-a692-45dfb99bfad1n%40googlegroups.com.

--

John McClain

Cell: 085-1977-823

Skype: jmcclain0129

Email: Jmccla...@gmail.com

[Andy Robinson]

Thanks for this.  27k file descriptors out of about 370k allowed.  I presume that means this isn't a problem?

(We run a lot of different virtual hosts on this box, each with a few Django processes)

- Andy

[Andy Robinson]

I'm unsure what you mean - do you mean "are we executing raw SQL queries and a Python db api, or using Django's ORM?"   If so, usually the latter but this is definitely not a problem of poorly written queries, we've profiled them.

- Andy