Putting Limits on Memory and CPU Usage in python ?

[SHINTO PETER]

how to limit memory utilization for a particular python script ?

[François Schiettecatte]

You don’t say what platform you are running on, but on linux/unix you can use ulimit, see http://ss64.com/bash/ulimit.html

François

> On Apr 20, 2015, at 1:09 PM, SHINTO PETER <shint...@gmail.com> wrote:
>
> how to limit memory utilization for a particular python script ?
>

> --
> You received this message because you are subscribed to the Google Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
> To post to this group, send email to django...@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/8bed3762-1dde-45e2-8573-4be58ab4d090%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[SHINTO PETER]

Working with linux/unix

[Tim Chase]

On 2015-04-20 13:13, François Schiettecatte wrote:
> You don’t say what platform you are running on, but on linux/unix
> you can use ulimit, see http://ss64.com/bash/ulimit.html

I've wanted this occasionally, but "ulimit -m" doesn't appear to work
on most modern Linux installations. See

http://unix.stackexchange.com/questions/129587/does-ulimit-m-not-work-on-modern-linux

for more info.

I'm not sure *why* it was removed, as there have been times I wanted
this functionality (I'm looking at you, Firefox, and your
memory-gorging habits) so it would be awfully helpful to have
around. I also just tested "ulimit -m" on OpenBSD and it too seems
to let the RSS of the process grow beyond the specified limit. I've
got a FreeBSD laptop around here that I could test as well if I can
find the power adaptor.

-tkc

[François Schiettecatte]

Bummer, shows how long it has been since I have used that.

Peter, why do you want to limit memory and CPU usage? To prevent someone from killing the machine ?

F.

> --
> You received this message because you are subscribed to the Google Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
> To post to this group, send email to django...@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.

> To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/20150420124711.0f49d04d%40bigbox.christie.dr.

[Tim Chase]

On 2015-04-20 13:54, François Schiettecatte wrote:
> Peter, why do you want to limit memory and CPU usage? To prevent
> someone from killing the machine ?

I know I've wanted it to help prevent malicious data that could suck
up my RAM and push things into swap. Absent "ulimit -a", I've had to
work around it with extra care to ensure that huge files or pessimal
instructions don't drag my machine to the ground.

-tkc

[François Schiettecatte]

I did a little research on this and found cpulimit, you can also use nice and renice, and (on fedora core 21 at least) you can do 'cat /proc/###/status’ where ### is the process ID which will give you memory stats on a process. Not ideal but you can wrap all of these in some python to make it easier to manage.

François

> --
> You received this message because you are subscribed to the Google Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
> To post to this group, send email to django...@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.

> To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/20150420125834.3faf7102%40bigbox.christie.dr.

[Tim Chase]

On 2015-04-20 16:01, François Schiettecatte wrote:
> I did a little research on this and found cpulimit, you can also
> use nice and renice,

Yes, nice/renice and their I/O friend ionice (in my Debian
installation, it's part of the "util-linux" package) work quite
nicely. It's the memory-limiting that I wish would work.

> and (on fedora core 21 at least) you can do
> 'cat /proc/###/status’ where ### is the process ID which will give
> you memory stats on a process. Not ideal but you can wrap all of
> these in some python to make it easier to manage.

Yes, I suppose you could have a watch-dog that will monitor memory
usage and then kill the process. I'd rather the process get an
out-of-memory error if it tried surpassing a set threshold (what
ulimit is *supposed* to do).

-tkc

[Erik Cederstrand]

> Den 20/04/2015 kl. 19.46 skrev SHINTO PETER <shint...@gmail.com>:
>
> Working with linux/unix

If unix includes FreeBSD, check out https://www.freebsd.org/doc/en/books/handbook/security-resourcelimits.html

Erik

[SHINTO PETER]

Hi

François Schiettecatte , limit memory and CPU usage for python socket client service

You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/LE6J4sRujbQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/E8425E9D-C9B9-4474-B6BC-3B719C599D23%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

SHINTO PETER
NIRAPPIL
09591040010

[David Palao]

Hi,
Have you had a look at "psutil"? It is a python module to manage processes.

Best

> https://groups.google.com/d/msgid/django-users/CAKggVnC%3DJGo%2BKxfE21%3DGAJY%2BTS%3DhTCJ8Ba1ZAVupLAHFanA1Yw%40mail.gmail.com.

[Erik Cederstrand]

> Den 21/04/2015 kl. 09.20 skrev SHINTO PETER <shint...@gmail.com>:
>
> Hi
> François Schiettecatte , limit memory and CPU usage for python socket client service

Really, if you want qualified help, you need to be more verbose.

Do you want to kill a process violating your limits? Or can the process be expected to cooperate, so you can communicate to a process that it needs to cut down on memory usage / CPU? Are these hard limits, or can the process use available resources if they are available? Do you want to build some sort of accounting, like AWS pay-per-use?

Anyway, this is not really a Django or Python problem. It's a hard problem to solve in most operating systems, so you may have better luck asking this in a forum related to your operating system.

Erik

[SHINTO PETER]

Hi ,

Erik Cederstrand , i just want to limit the memory (maximum heap size for your process) with in the python application/Service ie in it client Socket Program.

I found a solution like below

import resource
rsrc = resource.RLIMIT_DATA
soft, hard = resource.getrlimit(rsrc)
resource.setrlimit(rsrc, (1024, hard))

Link : http://stackoverflow.com/questions/2308091/how-to-limit-python-heap-size

--
You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/LE6J4sRujbQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/63B8E65D-E694-4F22-AA89-4732430FD840%40cederstrand.dk.

For more options, visit https://groups.google.com/d/optout.