How to limit CPU usage in Python

[Rolando Cañer Roblejo]

Hi all,

Is it possible for me to put a limit in the amount of processor usage (%
CPU) that my current python script is using? Is there any module useful
for this task? I saw Resource module but I think it is not the module I
am looking for. Some people recommend to use nice and cpulimit unix
tools, but those are external to python and I prefer a python solution.
I am working with Linux (Ubuntu 10.04).

Best regards.

[Terry Reedy]

Call the external tools with subprocess.open.

--
Terry Jan Reedy

[Terry Reedy]

I meant to end that with ? as I don't know how easy it is to get the
external id of the calling process that is to be limited. I presume that
can be done by first calling ps (with subprocess) and searching the
piped-back output.


--
Terry Jan Reedy

[Jerry Hill]

On Thu, Sep 20, 2012 at 11:12 AM, Rolando Cañer Roblejo
<roland...@cnic.edu.cu> wrote:
> Hi all,
>
> Is it possible for me to put a limit in the amount of processor usage (%
> CPU) that my current python script is using? Is there any module useful for
> this task? I saw Resource module but I think it is not the module I am
> looking for. Some people recommend to use nice and cpulimit unix tools, but
> those are external to python and I prefer a python solution. I am working
> with Linux (Ubuntu 10.04).

Maximum percentage of CPU used isn't normally something you control.
The only way I know of to do it involves having another process
monitor the thing you want to control and sending signals to stop and
start it (e.g., http://cpulimit.sourceforge.net/).

Typically, you instead want to control the priority (so that higher
priority apps can easily take more CPU time). That's what nice is for
(http://docs.python.org/library/os.html#os.nice). If you want to
limit a process in the same way that ulimit does, then the resources
module is what you want
(http://docs.python.org/library/resource.html#resource.setrlimit).

Is there a particular reason that you'd rather have your CPU sitting
idle, rather than continuing with whatever code is waiting to be run?
I'm having a hard time understanding what problem you might be having
that some combination of setting the nice level and imposing resource
limits won't handle.

--
Jerry

[Christian Heimes]

Am 20.09.2012 17:12, schrieb Rolando Ca�er Roblejo:
> Hi all,
>
> Is it possible for me to put a limit in the amount of processor usage (%
> CPU) that my current python script is using? Is there any module useful
> for this task? I saw Resource module but I think it is not the module I
> am looking for. Some people recommend to use nice and cpulimit unix
> tools, but those are external to python and I prefer a python solution.
> I am working with Linux (Ubuntu 10.04).

Hello,

you have two options here. You can either limit the total amount of CPU
seconds with the resource module or reduce the priority and scheduling
priority of the process.

The resource module is a wrapper around the setrlimit and getrlimit
feature as described in http://linux.die.net/man/2/setrlimit .

The scheduling priority can be altered with nice, get/setpriority or io
priority. The psutil package http://code.google.com/p/psutil/ wraps all
functions in a nice Python API.

Regards
Christian

[Cameron Simpson]

On 20Sep2012 12:53, Terry Reedy <tjr...@udel.edu> wrote:
| On 9/20/2012 12:46 PM, Terry Reedy wrote:

| > On 9/20/2012 11:12 AM, Rolando Cañer Roblejo wrote:
| >> Is it possible for me to put a limit in the amount of processor usage (%
| >> CPU) that my current python script is using? Is there any module useful
| >> for this task? I saw Resource module but I think it is not the module I
| >> am looking for. Some people recommend to use nice and cpulimit unix
| >> tools, but those are external to python and I prefer a python solution.
| >> I am working with Linux (Ubuntu 10.04).
| >

| > Call the external tools with subprocess.open.
|
| I meant to end that with ? as I don't know how easy it is to get the
| external id of the calling process that is to be limited. I presume that
| can be done by first calling ps (with subprocess) and searching the
| piped-back output.

If you're limiting yourself, os.getpid().
--
Cameron Simpson <c...@zip.com.au>

[Ramchandra Apte]

you

You could use os.times to compute the CPU usage and then stop the process when that happens and then start it after some time using signals.

[Ramchandra Apte]

you

On Saturday, 22 September 2012 05:14:15 UTC+5:30, Cameron Simpson wrote:

[Tim Roberts]

Rolando Cañer Roblejo <roland...@cnic.edu.cu> wrote:
>

>Is it possible for me to put a limit in the amount of processor usage (%
>CPU) that my current python script is using?

Why? That's an odd request. It's natural to want to reduce your priority
if you want other processes handled first, but an idle CPU is a wasted
resource. You want it to be busy all of the time.

>Some people recommend to use nice and cpulimit unix
>tools, but those are external to python and I prefer a python solution.

Scheduling and CPU priority are, by their very nature, operating system
concepts. You will not find generic mechanisms wrapping them.
--
Tim Roberts, ti...@probo.com
Providenza & Boekelheide, Inc.

[Paul Rubin]

Rolando Cañer Roblejo <roland...@cnic.edu.cu> writes:
> Is it possible for me to put a limit in the amount of processor usage
> (% CPU) that my current python script is using? Is there any module
> useful for this task?

One way is check your cpu usage once in a while, compare with elapsed
time, and if your % usage is above what you want, sleep for a suitable
interval before proceeding.

Tim Roberts: reasons to want to do this might involve a shared host
where excessive cpu usage affects other users; or a computer with
limited power consumption, where prolonged high cpu activity causes
thermal or other problems.

[Dwight Hutto]

rites:
>> Is it possible for me to put a limit in the amount of processor usage
>> (% CPU) that my current python script is using? Is there any module
>> useful for this task?
>
> One way is check your cpu usage once in a while, compare with elapsed
> time, and if your % usage is above what you want, sleep for a suitable
> interval before proceeding.
>

The script in constant runtime, unless it's in relation to other
processes, could be put on a % based sleep constant variable.

If the script is constantly running the same processes, and the OP
wants to limit it statistically, then at a critical portion in the
script sleep for a constant, or maybe, dynamic variable.

The only other is to create an app, disassemble it, and then refine
the asm instructions being used at the assembly level, but I'm just
scratching the surface of those enhancements.



--
Best Regards,
David Hutto
CEO: http://www.hitwebdevelopment.com

[Dwight Hutto]

Now also, just thinking theoretically with the knowledge I have,
you could underclock(as opposed to overclocking, which is what gamers
do), but have never seen that option in BIOS.

And maybe there is an option in your OS, google search term 'limiting
processes activity cpu usage':

https://www.google.com/search?client=ubuntu&channel=fs&q=limiting+processes+activity+cpu+usage&ie=utf-8&oe=utf-8

This seemed good for what you want from a brief overview:

http://www.cyberciti.biz/faq/cpu-usage-limiter-for-linux/

[Tim Roberts]

Paul Rubin <no.e...@nospam.invalid> wrote:
>
>Tim Roberts: reasons to want to do this might involve a shared host
>where excessive cpu usage affects other users;

That's what priorities are for.

>...or a computer with

>limited power consumption, where prolonged high cpu activity causes
>thermal or other problems.

OK, I grant that. However, statistically speaking, it is much more likely
that the OP merely has a misunderstanding.

[DPalao]

On Jueves septiembre 20 2012 11:12:44 Rolando Cañer Roblejo escribió:
> Hi all,

>
> Is it possible for me to put a limit in the amount of processor usage (%
> CPU) that my current python script is using? Is there any module useful

> for this task? I saw Resource module but I think it is not the module I

> am looking for. Some people recommend to use nice and cpulimit unix

> tools, but those are external to python and I prefer a python solution.

> I am working with Linux (Ubuntu 10.04).
>

> Best regards.

Hola,
Sometimes a stupid solution like the following does the trick:

> import time
> for t in tasks:
> do_something(t)
> time.sleep(some_seconds)

where "some_seconds" is a number related to the typical time-scale of the
tasks you are doing.

Hope it helps,

Regards


--
Miller's Slogan:
Lose a few, lose a few.

[88888 Dihedral]

DPalao於 2012年9月25日星期二UTC+8下午11時13分54秒寫道：

I think I'll prefer to use a generator of my object in python
to replace the sleep from the unix world. The reason is that I am not paid
from selling or buying work-stations in some business unit directly and immediately.

[88888 Dihedral]

DPalao於 2012年9月25日星期二UTC+8下午11時13分54秒寫道：

[Prasad, Ramit]

The problem is that checking the CPU usage is fairly misleading
if you are worried about contention. If your process takes up
100% of CPU and nothing else needs the resource, does it matter?
I would not want to sleep *unless* something else needs the
resource. Of course, there might be a good/easy way of checking
usage + contention, but I am unaware of any off the top of my
head.

On *nix you should just set the appropriate nice-ness and then
let the OS handle CPU scheduling. Not sure what you would do
for Windows--I assume OS X is the same as *nix for this context.



This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.

[Jerry Hill]

On Thu, Sep 27, 2012 at 12:58 PM, Prasad, Ramit
<ramit....@jpmorgan.com> wrote:
> On *nix you should just set the appropriate nice-ness and then
> let the OS handle CPU scheduling. Not sure what you would do
> for Windows--I assume OS X is the same as *nix for this context.

On windows, you can also set the priority of a process, though it's a
little different from the *nix niceness level. See
http://code.activestate.com/recipes/496767/ for a recipe using
pywin32. I believe the psutil module handles this too, but I don't
think it manages to abstract away the platform differences.

--
Jerry