High cpu usage of the top clone

[Dimitris Diamantis (ftso)]

Hello folks!
First of all i would like to congratulate the development team of this
great project :)
Many thanks guys!

I am trying to develop a cross-platform system monitor with python2.x
and qt4.
For getting the process informations for my application i am using the
psutil like the top.py in examples does.
(http://code.google.com/p/psutil/source/browse/trunk/examples/top.py)

My problem is the high cpu usage of the script.
The top.py cpu usage in my system (intel core 2 duo) is about 12% .
The number of process is 200-250.

Can you fix this performance issue or give me an idea to fix this
except the increasing of the interval time?

Thanks

--
Sorry for my bad English

[Giampaolo Rodolà]

2011/11/15 Dimitris Diamantis (ftso) <kot...@gmail.com>:

> Hello folks!
> First of all i would like to congratulate the development team of this
> great project :)
> Many thanks guys!

Thanks =)

> My problem is the high cpu usage of the script.
> The top.py cpu usage in my system (intel core 2 duo) is about 12% .
> The number of process is 200-250.
>
> Can you fix this performance issue or give me an idea to fix this
> except the increasing of the interval time?

I get roughly 6-7 % on my intel core 2 on Linux.
I guess this is normal and I doubt you can do much about that.
I took a look at gnome-system-monitor, which I suppose it uses an
implementation similar to top.py/psutil's, and there I get roughly
16%, so I think the problem is physiological.

Currently what we do on Linux to retrieve processes information is
reading and parsing the files in /proc filesystem in pure python.
This could be improved by rewriting those functions in C, but it's a
lot of work and I doubt it actually worths the effort.
Out of curiosity I tried to do some profiling against top.py script.
It turned out the most time consuming calls are get_cpu_percent(),
get_system_cpu_times() and get_memory_percent().
Not that this makes any different though, as I don't see any
particular bottleneck with them which can be fixed or improved.

73495 function calls (73486 primitive calls) in 1.178 seconds

Ordered by: cumulative time

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.001 0.001 1.180 1.180 top.py:13(<module>)
1 0.000 0.000 1.158 1.158 top.py:198(main)
2 0.005 0.002 1.150 0.575 top.py:79(poll)
2 1.001 0.501 1.001 0.501 {time.sleep}
3426 0.004 0.000 0.094 0.000 _pslinux.py:333(wrapper)
380 0.002 0.000 0.047 0.000 __init__.py:308(get_cpu_percent)
384 0.000 0.000 0.034 0.000 __init__.py:573(cpu_times)
381 0.003 0.000 0.033 0.000
_pslinux.py:163(get_system_cpu_times)
760 0.019 0.000 0.031 0.000 _pslinux.py:475(get_memory_info)
384 0.025 0.000 0.025 0.000 {method 'readline' of
'file' objects}
380 0.001 0.000 0.021 0.000 __init__.py:228(username)
760 0.004 0.000 0.020 0.000 _pslinux.py:437(get_cpu_times)
3438 0.018 0.000 0.018 0.000 {open}
380 0.001 0.000 0.018 0.000 __init__.py:152(name)
1 0.001 0.001 0.017 0.017 __init__.py:12(<module>)
380 0.001 0.000 0.016 0.000
__init__.py:377(get_memory_percent)
380 0.000 0.000 0.016 0.000 __init__.py:366(get_memory_info)
380 0.000 0.000 0.014 0.000 __init__.py:207(uids)
380 0.008 0.000 0.013 0.000 _pslinux.py:729(get_process_uids)
1520 0.013 0.000 0.013 0.000 {method 'read' of 'file' objects}
386 0.000 0.000 0.011 0.000 __init__.py:190(status)
380 0.000 0.000 0.010 0.000 __init__.py:360(get_cpu_times)
3438 0.010 0.000 0.010 0.000 {method 'close' of 'file' objects}
386 0.007 0.000 0.010 0.000
_pslinux.py:583(get_process_status)
1 0.000 0.000 0.009 0.009 _common.py:9(<module>)
380 0.001 0.000 0.009 0.000 _pslinux.py:357(get_process_name)
2 0.004 0.002 0.008 0.004 top.py:164(refresh_window)
18 0.006 0.000 0.008 0.000 collections.py:237(namedtuple)
380 0.000 0.000 0.007 0.000 __init__.py:185(cmdline)
380 0.001 0.000 0.007 0.000
_pslinux.py:399(get_process_cmdline)
380 0.006 0.000 0.006 0.000 {pwd.getpwuid}
29029 0.005 0.000 0.005 0.000 {method 'startswith' of
'str' objects}
573 0.000 0.000 0.004 0.000 __init__.py:556(process_iter)
3848 0.004 0.000 0.004 0.000 {method 'split' of 'str' objects}
1 0.001 0.001 0.003 0.003 _pslinux.py:9(<module>)
760 0.003 0.000 0.003 0.000 top.py:60(bytes2human)
570 0.001 0.000 0.003 0.000 __init__.py:98(__init__)
1 0.001 0.001 0.002 0.002 subprocess.py:389(<module>)
1 0.002 0.002 0.002 0.002 socket.py:45(<module>)
570 0.000 0.000 0.002 0.000 _pslinux.py:247(pid_exists)
1 0.000 0.000 0.002 0.002 _psposix.py:9(<module>)
1 0.000 0.000 0.001 0.001 _compat.py:9(<module>)
570 0.001 0.000 0.001 0.000 _psposix.py:27(pid_exists)

[Dimitris Diamantis (ftso)]

On Nov 16, 8:12 pm, Giampaolo Rodolà <g.rod...@gmail.com> wrote:
> 2011/11/15 Dimitris Diamantis (ftso) <kots...@gmail.com>:

>
> > Hello folks!
> > First of all i would like to congratulate the development team of this
> > great project :)
> > Many thanks guys!
>
> Thanks =)
>
> > My problem is the high cpu usage of the script.
> > The top.py cpu usage in my system (intel core 2 duo) is about 12% .
> > The number of process is 200-250.
>
> > Can you fix this performance issue or give me an idea to fix this
> > except the increasing of the interval time?
>
> I get roughly 6-7 % on my intel core 2 on Linux.

Yes but this depends on the number of the processes i guess.

> I guess this is normal and I doubt you can do much about that.
> I took a look at gnome-system-monitor, which I suppose it uses an
> implementation similar to top.py/psutil's, and there I get roughly
> 16%, so I think the problem is physiological.

Ok, i see.
I set the interval to 3 sec like the default interval time of the real
top and the cpu usage decreased
to 4-5% while the real top has 0.5-1% usage at the same time.
Thats not good, but not so bad!

> Currently what we do on Linux to retrieve processes information is
> reading and parsing the files in /proc filesystem in pure python.
> This could be improved by rewriting those functions in C, but it's a
> lot of work and I doubt it actually worths the effort.

I think that rewriting only the most time consuming calls in C in the
future will make the difference.

> Out of curiosity I tried to do some profiling against top.py script.
> It turned out the most time consuming calls are get_cpu_percent(),
> get_system_cpu_times() and get_memory_percent().
> Not that this makes any different though, as I don't see any
> particular bottleneck with them which can be fixed or improved.

>     ...

Can you tell me which software you are using for profiling?
Thanks for your answer :)

[Giampaolo Rodolà]

2011/11/16 Dimitris Diamantis (ftso) <kot...@gmail.com>:

>
>
> On Nov 16, 8:12 pm, Giampaolo Rodolà <g.rod...@gmail.com> wrote:
>> 2011/11/15 Dimitris Diamantis (ftso) <kots...@gmail.com>:
>>
>> > Hello folks!
>> > First of all i would like to congratulate the development team of this
>> > great project :)
>> > Many thanks guys!
>>
>> Thanks =)
>>
>> > My problem is the high cpu usage of the script.
>> > The top.py cpu usage in my system (intel core 2 duo) is about 12% .
>> > The number of process is 200-250.
>>
>> > Can you fix this performance issue or give me an idea to fix this
>> > except the increasing of the interval time?
>>
>> I get roughly 6-7 % on my intel core 2 on Linux.
>
> Yes but this depends on the number of the processes i guess.

I don't think so. You said you had 200-250 processes. Same for me here (204).

>> I guess this is normal and I doubt you can do much about that.
>> I took a look at gnome-system-monitor, which I suppose it uses an
>> implementation similar to top.py/psutil's, and there I get roughly
>> 16%, so I think the problem is physiological.
>
> Ok, i see.
> I set the interval to 3 sec like the default interval time of the real
> top and the cpu usage decreased
> to 4-5% while the real top has 0.5-1% usage at the same time.
> Thats not good, but not so bad!

Well, I'd say 4-5% is acceptable for such an app.
Out of curiosity, I've tried top.py on FreeBSD, which implementation
is written in C, and there I get a constant 1%, so the difference
might actually be the Linux implementation which relies on /proc.

>> Currently what we do on Linux to retrieve processes information is
>> reading and parsing the files in /proc filesystem in pure python.
>> This could be improved by rewriting those functions in C, but it's a
>> lot of work and I doubt it actually worths the effort.
>
> I think that rewriting only the most  time consuming calls in C in the
> future will make the difference.

It's a matter of figuring out whether the benefit justifies the cost.
As of now I don't think the cost is justiified.
I'll try to look deeper into cpu/memory related functions and figure
out whether they're significantly slower compared to others though
(but I doubt it), in which case it might make sense to rewrite those
in C.

>> Out of curiosity I tried to do some profiling against top.py script.
>> It turned out the most time consuming calls are get_cpu_percent(),
>> get_system_cpu_times() and get_memory_percent().
>> Not that this makes any different though, as I don't see any
>> particular bottleneck with them which can be fixed or improved.
>>     ...
>
> Can you tell me which software you are using for profiling?
> Thanks for your answer :)

I used the cProfile module.
I first removed the ncurses-related lines from top.py to avoid messing
with the standard output (patch below) and then run:
python -m cProfile -s cumulative examples/top.py

Index: examples/top.py
===================================================================
--- examples/top.py (revisione 1224)
+++ examples/top.py (copia locale)
@@ -26,6 +26,7 @@

# --- curses stuff
def tear_down():
+ return
win.keypad(0)
curses.nocbreak()
curses.echo()
@@ -38,6 +39,8 @@

def print_line(line, highlight=False):
"""A thin wrapper around curses's addstr()."""
+# print line
+ return
global lineno
try:
if highlight:
@@ -162,7 +165,7 @@
"""Print results on screen by using curses."""
curses.endwin()
templ = "%-6s %-8s %4s %5s %5s %6s %4s %9s %2s"
- win.erase()
+# win.erase()
header = templ % ("PID", "USER", "NI", "VIRT", "RES", "CPU%", "MEM%",
"TIME+", "NAME")
print_header(procs_status)
@@ -189,13 +192,13 @@
print_line(line)
except curses.error:
break
- win.refresh()
+# win.refresh()

def main():
try:
interval = 0
- while 1:
+ for x in range(2):
args = poll(interval)
refresh_window(*args)
interval = 1

[Jay Loden]

On Wed, Nov 16, 2011 at 3:48 PM, Giampaolo Rodolà <g.ro...@gmail.com> wrote:

Well, I'd say 4-5% is acceptable for such an app.
Out of curiosity, I've tried top.py on FreeBSD, which implementation
is written in C, and there I get a constant 1%, so the difference
might actually be the Linux implementation which relies on /proc.

>> Currently what we do on Linux to retrieve processes information is
>> reading and parsing the files in /proc filesystem in pure python.
>> This could be improved by rewriting those functions in C, but it's a
>> lot of work and I doubt it actually worths the effort.
>
> I think that rewriting only the most  time consuming calls in C in the
> future will make the difference.

It's a matter of figuring out whether the benefit justifies the cost.
As of now I don't think the cost is justiified.
I'll try to look deeper into cpu/memory related functions and figure
out whether they're significantly slower compared to others though
(but I doubt it), in which case it might make sense to rewrite those
in C.

Agreed on both counts. I think the utilization is reasonably, considering we're comparing an interpreted application to an app written in pure C, optimized over years of development :) 

In most use cases, psutil is not a high performance library where it would be worth the effort to implement in C when we don't have to. That may change if we discover a truly compelling use case in the future, but at the moment it's a bigger benefit to have the ease of maintenance and code readability and easier portability of Python than the lower overhead of C. 

-Jay

[Giampaolo Rodolà]

> Agreed on both counts. I think the utilization is reasonably, considering
> we're comparing an interpreted application to an app written in pure C

I don't think the reason is the interpreter overhead.
An hypothetical C implementation parsing /proc wouldn't be
significantly faster (gnome-system-monitor is an example).

2011/11/16 Jay Loden <jlo...@gmail.com>:

> --
> You received this message because you are subscribed to the "Python process
> utilities (psutil)" project group:
> http://code.google.com/p/psutil
> To post to this group, send email to psu...@googlegroups.com
> To unsubscribe from this group, send email to
> psutil-un...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/psutil

[Jay Loden]

On Wed, Nov 16, 2011 at 5:04 PM, Giampaolo Rodolà <g.ro...@gmail.com> wrote:

> Agreed on both counts. I think the utilization is reasonably, considering
> we're comparing an interpreted application to an app written in pure C

I don't think the reason is the interpreter overhead.
An hypothetical C implementation parsing /proc wouldn't be
significantly faster (gnome-system-monitor is an example).

Right, I meant comparing the C version of the unix top util to the Python top implementation using psutil, I would inherently expect to see more CPU time used by Python.