awx-web Memory Usage Always Up

[stan.v...@exlhub.com]

I run 1.0.6.41 in an OpenShift 3.9 semi-prod install, where I execute occasional jobs and sometimes shift components around when redeploying oc nodes, so that the awx pod gets periodically recreated. Each time it starts up with a low memory consumption, but as I run jobs via the web console - I see that the awx-web memory usage in the oc metrics weekly graphs consistently goes up and never recedes. Is this a known issue? Looks like Django never releases some resources that it grabs. What is the command to restart the Django process on it, by the way?   

[Christopher Meyers]

Uwsgi, which houses the Django environment would be the most likey culprit. If you have DEBUG set to true Django will record all SQL queries in memory. To check, connect to the container and run awx-manage shell. Then

from django.conf import settings

settings.DEBUG

If this setting is false let us know. We'd be interested in tracking this down.

On Mon, Aug 27, 2018, 11:17 PM <stan.v...@exlhub.com> wrote:

I run 1.0.6.41 in an OpenShift 3.9 semi-prod install, where I execute occasional jobs and sometimes shift components around when redeploying oc nodes, so that the awx pod gets periodically recreated. Each time it starts up with a low memory consumption, but as I run jobs via the web console - I see that the awx-web memory usage in the oc metrics weekly graphs consistently goes up and never recedes. Is this a known issue? Looks like Django never releases some resources that it grabs. What is the command to restart the Django process on it, by the way?   

--
You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.
To post to this group, send email to awx-p...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/02390e3f-d51f-47d9-a031-b36ee89e5451%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[stan.v...@exlhub.com]

In awx-web container, the DEBUG setting checked per below is False. So, after the last restart a few days ago, the awx-web was sitting idle at about 512Mb. After updating a couple of templates and running a few jobs, the memory usage went up to 850 and then, after running another series of jobs, progressively to over 1,600. I closed the browser and logged out, but the memory didn't move down from the highest value it's gotten to. The memcached memory followed a similar pattern (from almost zero to 60Mb) - which generally makes sense, although all the executed jobs where from the same group of templates, so the memcached appears to be filling up with job run details (more jobs - more memory). The celery was quite stable at around 800Mb - in either idle or working states. From my earlier experience, the web's memory keeps on creeping - I have it limited at 2560 - till another restart. The job cleanup schedule is set to 30 days, and I usually reset the container prior to that. If I go and manually delete a bunch of completed jobs - the awx-web memory goes slightly up (like with any activity in the console).