Received: by 10.52.23.146 with SMTP id m18mr16062612vdf.7.1336264636969;
        Sat, 05 May 2012 17:37:16 -0700 (PDT)
X-BeenThere: pylons-discuss@googlegroups.com
Received: by 10.220.140.194 with SMTP id j2ls154734vcu.5.gmail; Sat, 05 May
 2012 17:37:12 -0700 (PDT)
Received: by 10.52.32.66 with SMTP id g2mr16075912vdi.5.1336264632967;
        Sat, 05 May 2012 17:37:12 -0700 (PDT)
Received: by 10.52.32.66 with SMTP id g2mr16075911vdi.5.1336264632956;
        Sat, 05 May 2012 17:37:12 -0700 (PDT)
Return-Path: <chr...@plope.com>
Received: from cabana.palladion.com (cabana.palladion.com. [64.34.177.88])
        by gmr-mx.google.com with ESMTPS id jy3si6300201vdb.1.2012.05.05.17.37.12
        (version=TLSv1/SSLv3 cipher=OTHER);
        Sat, 05 May 2012 17:37:12 -0700 (PDT)
Received-SPF: neutral (google.com: 64.34.177.88 is neither permitted nor denied by best guess record for domain of chr...@plope.com) client-ip=64.34.177.88;
Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 64.34.177.88 is neither permitted nor denied by best guess record for domain of chr...@plope.com) smtp.mail=chr...@plope.com
Received: from [192.168.1.145] (ip72-209-213-54.dc.dc.cox.net [72.209.213.54])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by cabana.palladion.com (Postfix) with ESMTP id 9AB511C8056
	for <pylons-discuss@googlegroups.com>; Sat,  5 May 2012 18:20:14 -0400 (EDT)
Message-ID: <4FA5C7B7.5050702@plope.com>
Date: Sat, 05 May 2012 20:37:11 -0400
From: Chris McDonough <chr...@plope.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:12.0) Gecko/20120430 Thunderbird/12.0.1
MIME-Version: 1.0
To: pylons-discuss@googlegroups.com
Subject: Re: Python's memory hogging
References: <4FA5BAEA.8050807@haronmedia.com>
In-Reply-To: <4FA5BAEA.8050807@haronmedia.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 05/05/2012 07:42 PM, Vlad K. wrote:
>
> Hi all.
>
>
> As I understand it Python won't release internally released memory back
> to OS. So it happens in my Pyramid app that has to process/construct
> rather largish XML files occasionally, the memory consumption jumps to
> several hundred MB and stays there for good or until I recycle the
> processes (uwsgi reload). This is pretty bad in my case because I'm
> forced to have a rather large memory headroom on the server just to
> handle those peaks which comprise less than 1% of total requests
> throughout the day. Hell, it's way less than 1%. A single request for
> such XML file takes anywhere from 1 to up to 10 seconds on this
> particular VPS "hardware", and there are maybe a few dozen such requests
> throughout the day (out of several thousand daily requests). So what
> should really be a transient memory consumption peak lasting up to 10
> seconds becomes permanent memory requirement (or until the uwsgi
> processes are reloaded).
>
> Now I know I can set max-requests for the uwsgi processes and it will
> recycle automatically, but is there another solution? I find this
> reluctance of Python to return memory to OS very annoying, right next to
> the GIL.

Commenting on how Python does or doesn't return memory to the OS is 
above my current pay grade.  Because I don't understand it, and have no 
competence to try to "fix" it, I have to work around it.  I usually try 
to do that by writing code doesn't ask for hundreds of megabytes all in 
one shot from the OS.

For example, maybe you can construct the XML in portions instead of 
constructing a huge collection of strings that all reside in memory at once.

If there's a memory leak somewhere outside of my control, what I usually 
do to work around it is to use supervisor + memmon to automatically 
restart my processes when they use more than some amount of memory:

http://www.plope.com/Members/chrism/memmon_sample

This isn't immediately useful if your processes are spawned by uwsgi, 
but might be useful if you choose to use a different frontend server 
like Apache or nginx to proxy to a number of backend supervisor-managed 
Pyramid processes.

- C