Website slowed down drasticaly
43 views
Skip to first unread message
Saurabh Adhikary
May 2, 2019, 7:40:20 AM5/2/19
to Django users
Hello ,
I'm using Django version 1.8.1 and Python version 2.7.12 . Suddenly since Feb '19 there has been a drastic decrease in my website's response rate.
For sure , there has been some minor increase in the no of hits, but the performance is too bad.
Initially , the thought came that the hardware of the server was old , so a new server with high configuration was bought.
We have done the new indexing also on it.
Still the sought for a higher performance is awaited.
Is it that the Django support for 1.8.1 or Python support for 2.7.12 has reduced and that is casing the website to slow down or I am missing out on something ?
Kindly help.
Chetan Ganji
May 2, 2019, 8:22:11 AM5/2/19
to django...@googlegroups.com
Not sure about whats the issue. You could do couple of things to understand whats the root cause of the problem.
I know they are generic guidelines. Anyone couldnt be more specific than this.
- Benchmark the time required to process each request. You could write a middleware to track this time.
Attach starttime to each request object and read that at the time of returning the response. - Try using the django debug toolbar to see how much time it is taking to execute the sql queries.
There might be some room for improvement as most developers dont practice sql regularly.
https://django-debug-toolbar.readthedocs.io/en/latest/ - Maybe try using a different database - PostgreSQL, MySQL, etc.
- But if you have time for R&D, you could try using the different python implementation, it is said to be faster in many cases than cpython.
It's not 100% compliant i.e. some packages might not work with pypy. So please do your research before walking down this road.
https://pypy.org/ - Try using a different web server. If you are using Apache, try using nginx. Also use a different wsgi server. If you are using gunicorn, try with waitress or others.
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/238a6da2-8f34-4b8b-939c-e20d4306545b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
אורי
May 2, 2019, 8:41:05 AM5/2/19
to django...@googlegroups.com
Why don't you upgrade Django at least to 1.11? Django 1.8 extended support ended last year.
--
nm
May 2, 2019, 9:30:05 AM5/2/19
to Django users
Django Debug Toolbar is a useful tool, it'll tell you what takes the biggest amount of time while the website loads.
One caveat regarding performance: I once developed a project (running Django 1.8 and Python 3.4), and the Toolbar slowed down our back end tremendously. We tried to figure out the reason why some pages took as much as 20 seconds to load, and discovered by chance that if you disabled the toolbar, performance improved considerably (down to 3-4 seconds, or so). Obviously, you don't use it in production, so it's only an issue in development, but it may be worth knowing if you decide to use DDT.
Granted, I'm not sure if this happens always, because it was the only project where I used DDT.
On Thursday, 2 May 2019 14:22:11 UTC+2, Chetan Ganji wrote:
Not sure about whats the issue. You could do couple of things to understand whats the root cause of the problem.I know they are generic guidelines. Anyone couldnt be more specific than this.
- Benchmark the time required to process each request. You could write a middleware to track this time.
Attach starttime to each request object and read that at the time of returning the response.- Try using the django debug toolbar to see how much time it is taking to execute the sql queries.
There might be some room for improvement as most developers dont practice sql regularly.
https://django-debug-toolbar.readthedocs.io/en/latest/- Maybe try using a different database - PostgreSQL, MySQL, etc.
- But if you have time for R&D, you could try using the different python implementation, it is said to be faster in many cases than cpython.
It's not 100% compliant i.e. some packages might not work with pypy. So please do your research before walking down this road.
https://pypy.org/- Try using a different web server. If you are using Apache, try using nginx. Also use a different wsgi server. If you are using gunicorn, try with waitress or others.
On Thu, May 2, 2019 at 5:09 PM Saurabh Adhikary <adhikar...@gmail.com> wrote:
Hello ,--I'm using Django version 1.8.1 and Python version 2.7.12 . Suddenly since Feb '19 there has been a drastic decrease in my website's response rate.For sure , there has been some minor increase in the no of hits, but the performance is too bad.Initially , the thought came that the hardware of the server was old , so a new server with high configuration was bought.We have done the new indexing also on it.Still the sought for a higher performance is awaited.Is it that the Django support for 1.8.1 or Python support for 2.7.12 has reduced and that is casing the website to slow down or I am missing out on something ?Kindly help.
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django...@googlegroups.com.
Saurabh Adhikary
May 3, 2019, 1:26:24 AM5/3/19
to Django users
Hello Uri,
The project I handle is quite big , so before taking it up as a major task, I wish to get it from the experts that is the django version the issue or python version or am I missing out on something major.
The issue started to escalate from Feb '19 , after which even a new powerful server could'nt give us good performance.
Now after adding extra indexes , at least the website is loading.
We are still not sure if we can handle around 5K - 10K concurrent request with a 128GB SSD server.
On Thursday, 2 May 2019 18:11:05 UTC+5:30, Uri Even-Chen wrote:
Why don't you upgrade Django at least to 1.11? Django 1.8 extended support ended last year.
On Thu, May 2, 2019 at 2:39 PM Saurabh Adhikary <adhikar...@gmail.com> wrote:
Hello ,--I'm using Django version 1.8.1 and Python version 2.7.12 . Suddenly since Feb '19 there has been a drastic decrease in my website's response rate.For sure , there has been some minor increase in the no of hits, but the performance is too bad.Initially , the thought came that the hardware of the server was old , so a new server with high configuration was bought.We have done the new indexing also on it.Still the sought for a higher performance is awaited.Is it that the Django support for 1.8.1 or Python support for 2.7.12 has reduced and that is casing the website to slow down or I am missing out on something ?Kindly help.
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django...@googlegroups.com.
Saurabh Adhikary
May 3, 2019, 1:29:51 AM5/3/19
to Django users
Hi Chetan,
1) Yes . We are infact trying this. Thank You.
2) Taken
2) Taken
3) We are on MySql , migration will cost a lot of man hours, which can be invested only if the result is concrete.
4) Up-gradation, that's our last resort.
5) Tried almost all. All in vain.
Thanks for your inputs. :)
On Thursday, 2 May 2019 17:52:11 UTC+5:30, Chetan Ganji wrote:
Not sure about whats the issue. You could do couple of things to understand whats the root cause of the problem.I know they are generic guidelines. Anyone couldnt be more specific than this.
- Benchmark the time required to process each request. You could write a middleware to track this time.
Attach starttime to each request object and read that at the time of returning the response.- Try using the django debug toolbar to see how much time it is taking to execute the sql queries.
There might be some room for improvement as most developers dont practice sql regularly.
https://django-debug-toolbar.readthedocs.io/en/latest/- Maybe try using a different database - PostgreSQL, MySQL, etc.
- But if you have time for R&D, you could try using the different python implementation, it is said to be faster in many cases than cpython.
It's not 100% compliant i.e. some packages might not work with pypy. So please do your research before walking down this road.
https://pypy.org/- Try using a different web server. If you are using Apache, try using nginx. Also use a different wsgi server. If you are using gunicorn, try with waitress or others.
On Thu, May 2, 2019 at 5:09 PM Saurabh Adhikary <adhikar...@gmail.com> wrote:
Hello ,--I'm using Django version 1.8.1 and Python version 2.7.12 . Suddenly since Feb '19 there has been a drastic decrease in my website's response rate.For sure , there has been some minor increase in the no of hits, but the performance is too bad.Initially , the thought came that the hardware of the server was old , so a new server with high configuration was bought.We have done the new indexing also on it.Still the sought for a higher performance is awaited.Is it that the Django support for 1.8.1 or Python support for 2.7.12 has reduced and that is casing the website to slow down or I am missing out on something ?Kindly help.
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django...@googlegroups.com.
Chetan Ganji
May 3, 2019, 5:18:41 AM5/3/19
to django...@googlegroups.com
Hi Saurabh,
You are welcome :)
No 1 will give you insights about which endpoints are taking the most time to load. Then, you can drill down the specific endpoints for bottlenecks.
Two more things you could try.
1. Remove unused/unnecessary middlewares from the middlewares list in the settings.py of the project.
As they are executed before and after every single request, they could add hugh unnecessary overhead easily.
2. Normally requests are processed in synchronous manner. Gunicorn has aysnc worker types.
I did not have a need to look into it yet, but they could help speed up things.
I would not suggest it for payment gateway endpoints. If your project is a ERP/CRM type project, it could help you a little to look into it.
Nginx already handles requests async manner. Try combining Nginx + Async worker in gunicorn.
You could get some boost.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/64e51c12-1ff3-4e9f-a45b-e0d32a6551b3%40googlegroups.com.
Chetan Ganji
May 3, 2019, 5:21:55 AM5/3/19
to django...@googlegroups.com
One more thing -
All the third party resources i.e. js and css files e.g. bootstrap, jquery, etc; don't fetch them from your server.
Use CDN for those. This will help the load times a lot.
Nick Sarbicki
May 3, 2019, 6:01:50 AM5/3/19
to Django users
Saurabh what do you use for the WSGI HTTP Server (gunicorn etc.)? Worth checking that as well to make sure it has enough workers.
In general performance problems are rarely version related. Usually it's either a bad setup or inefficient code.
- Nick
Django toolbar and most of the other suggestions will help you profile the code but worth seeing how it is served as well.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAMKMUjsHZRHj-j10Bf_5DxapcHu1w2tYBt2vLSFEYpojBHTckA%40mail.gmail.com.
Saurabh Adhikary
May 3, 2019, 7:14:30 AM5/3/19
to Django users
Hi Nick,
We are using uwsgi . Uwsgi is installed via pip.
Bad setup :
is my area of interest as in some senior professionals can highlight the probable area of mistakes while shifting website from one server to another (literal hardware shift).
Although we have some good people around, but there can a slip here and there.
Inefficient coding:
Not my area of concern as of now, it was working fair in our old server, so should work fine here too.
Another area:
Django & Python support end-life issue ?
I'm yet to here anything concrete on this.
On Friday, 3 May 2019 15:31:50 UTC+5:30, Nick Sarbicki wrote:
Saurabh what do you use for the WSGI HTTP Server (gunicorn etc.)? Worth checking that as well to make sure it has enough workers.In general performance problems are rarely version related. Usually it's either a bad setup or inefficient code.Django toolbar and most of the other suggestions will help you profile the code but worth seeing how it is served as well.- Nick
On Fri, May 3, 2019 at 10:21 AM Chetan Ganji <ganji...@gmail.com> wrote:
One more thing -All the third party resources i.e. js and css files e.g. bootstrap, jquery, etc; don't fetch them from your server.Use CDN for those. This will help the load times a lot.
On Fri, May 3, 2019 at 2:47 PM Chetan Ganji <ganji...@gmail.com> wrote:
Hi Saurabh,You are welcome :)No 1 will give you insights about which endpoints are taking the most time to load. Then, you can drill down the specific endpoints for bottlenecks.Two more things you could try.1. Remove unused/unnecessary middlewares from the middlewares list in the settings.py of the project.As they are executed before and after every single request, they could add hugh unnecessary overhead easily.2. Normally requests are processed in synchronous manner. Gunicorn has aysnc worker types.I did not have a need to look into it yet, but they could help speed up things.I would not suggest it for payment gateway endpoints. If your project is a ERP/CRM type project, it could help you a little to look into it.Nginx already handles requests async manner. Try combining Nginx + Async worker in gunicorn.You could get some boost.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/64e51c12-1ff3-4e9f-a45b-e0d32a6551b3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
opistola
May 3, 2019, 7:30:24 AM5/3/19
to django...@googlegroups.com
my thoughts... check dns resolution time
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/e24a0cf4-28ad-482c-b14c-888ed2f242a6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Cordiais cumprimentos,
Sérgio Pinto
tf: +351 937641545
tw: http://twitter.com/opistola
ld: http://pt.linkedin.com/pub/sergio-pinto/11/b4b/33a
| tf=Telemóvel | tw=Twitter | ld=LinkedIn
'' Imagine your life without internet!!! ''
Sérgio Pinto
tf: +351 937641545
tw: http://twitter.com/opistola
ld: http://pt.linkedin.com/pub/sergio-pinto/11/b4b/33a
| tf=Telemóvel | tw=Twitter | ld=LinkedIn
'' Imagine your life without internet!!! ''
Scott Miesbauer
May 3, 2019, 12:00:29 PM5/3/19
to django...@googlegroups.com
I apologize if I missed it but did someone find out what started happening in Feb that might have some impact on the performance lag?
The very first thing I ask when troubleshooting is “What changed?”
So what changed in your environment in February? Other teams changes? Application? Network?
If you find what changed then the solution might end up being right there in front of you.
Good luck!
Sent from my iPhone
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/08bc1409-322e-41ce-a94b-5dd9db888e86%40googlegroups.com.
Alex Heyden
May 3, 2019, 12:17:03 PM5/3/19
to django...@googlegroups.com
Definitely echoing Scott's sentiment here. Start with what changed.
In my personal experience, if the answer is "nothing," you can narrow it down very quickly. Start by reloading with the browser network monitor in the foreground. If the first response comes back very quickly, Django itself is off the hook and you proceed with front-end optimizations. If it doesn't, and you're really sure that no one changed anything (check top or the process manager to make sure there's nothing new running!), it's your database. You need your data set to be shrunk, indices to be improved, or queries to be optimized. Django Debug Toolbar will help you see which of these apply to you. If you see hundreds of queries where you expected just one or two, look for any model reference in a loop, including template code.
If you'd like extra reassurance that that's the problem, point your config to an empty database, run the migrations, and see the difference.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/05C52FE4-C1E8-45F1-8146-A9C6D3154AC2%40gmail.com.
Sally Middleton
May 5, 2019, 10:06:25 AM5/5/19
to Django users
Hi
From a generic debugging point of view, you need to find out the root cause of the performance decline. Ideally you need to isolate and test each part of the system in turn and look for the one which is causing the problem.
1. If it can be reproduced on a prod-like environment then that's really great as you can make changes easily to test. Always keep a note of EXACTLY what you've changed and put it back the way it was previously before your next test. If you cannot reproduce that also tells you something, for example it's due to data quantities or prod-hardware.
2. Put tracing/logging on your prod environment database and check for SQL for degradation (maybe compared to your development env). Maybe all is OK or one query is slow or everything is slow. Any performance issues will need tuning, but it sounds like you are looking for something quite fundamental.
3. What happened in February? Did you perform a release to Production? What was in it? If you roll-back (in test environment) does it fix the problem? Maybe it is a component that has been updated. Any configuration changes to your web server?
4. If it isn't the database then it's something else. You need to isolate each part of the system in turn and put on logging then check the results.
Best regards
Sally
Saurabh Adhikary
May 6, 2019, 3:20:41 AM5/6/19
to Django users
Hey Sally,Alex,Scott,
I'm real glad to see that , all of you are exactly pointing out the area of my concern. Thank you so much.
The point is , there are no major change that was deployed on to the code since Dec'18.
As I did mention that there has been some minor load increase of our users in the new year since Jan, but the change is not significant.
(Originally we had around 2million unique people , ~5million approx approx users in total, and around 50K new users during this time)
We are on MySQL 14.14 Distrib 5.1.73. My worry is if Django support(for 1.8.1) or Python support (for 2.7.12) diminishing is the main reason for slowing down.
Rest, the debugging has been done, wherin we could'nt find a fault as of yet.
My project is really big and I hence wish to seek some experience or concrete knowledge before investing time for upgrading Django & Python.
Regards,
Saurabh
Nick Sarbicki
May 6, 2019, 3:47:58 AM5/6/19
to Django users
Hi Saurabh,
If the diminishing support was a cause of the dramatic slowdown then this would have had to be intentionally put into the codebase. I'm almost certain this isn't the case.
All losing support really means is that the developers no longer provide updates and security patches. There would not be a slowdown as nothing actually changes in your dependencies.
Sorry to say that it's almost certainly caused by a change in your code, configuration, state, or limited by your hardware.
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/85afc1b7-f8bb-468d-8759-c41bf3645dce%40googlegroups.com.
Tim Chase
May 6, 2019, 6:49:49 AM5/6/19
to django...@googlegroups.com
On 2019-05-06 00:20, Saurabh Adhikary wrote:
> The point is , there are no major change that was deployed on to
> the code since Dec'18.
> As I did mention that there has been some minor load increase of
> our users in the new year since Jan, but the change is not
> significant. (Originally we had around 2million unique people ,
> ~5million approx approx users in total, and around 50K new users
> during this time)
I imagine that, if nothing changed in the code, and nothing changed
> The point is , there are no major change that was deployed on to
> the code since Dec'18.
> As I did mention that there has been some minor load increase of
> our users in the new year since Jan, but the change is not
> significant. (Originally we had around 2million unique people ,
> ~5million approx approx users in total, and around 50K new users
> during this time)
in the back-end server infrastructure (no changes to your DB
settings, server hardware, etc), but there was an increase in
traffic, then I imagine the issue stems from some unfortunate scaling
issues. Often this is either related to poor choices for indexing in
the database or some algorithm that used to work fine with smaller
datasets but fails to scale to larger numbers of users.
-tim
Reply all
Reply to author
Forward
0 new messages