Idle time percent.

[Pepe Aracil]

Hi All.

How can I calculate the amount idle time of gevent main loop in last
minute?

It will be useful to calculate de load of a python process and
redistribute load
to other process.


Thanks.

[Bob Van Zant]

When gevent's event loop is idle the whole process is blocked on a
call to epoll() (assuming linux). In that case the entire process is
idle. So just look at the process CPU usage?

-Bob

[Pepe Aracil]

Method to get process CPU usage is SO dependent.

A simple time accumulator would be better portable solution.

startIdle = time.time()
epoll.poll() # any eventlib poll method
idleAcumulator = time.time() - startIdle

Simply checking idleAcumulator is easy to calculate process load
average on last milliseconds, seconds, minutes , ...

Pepe.

On 22 abr, 22:24, Bob Van Zant <b...@veznat.com> wrote:
> When gevent's event loop is idle the whole process is blocked on a
> call to epoll() (assuming linux). In that case the entire process is
> idle. So just look at the process CPU usage?
>
> -Bob
>
> On Fri, Apr 22, 2011 at 11:39 AM, Pepe Aracil
>

[Pepe Aracil]

idleAcumulator += time.time() - startIdle

:)

[Andy]

On Apr 22, 4:24 pm, Bob Van Zant <b...@veznat.com> wrote:
> When gevent's event loop is idle the whole process is blocked on a
> call to epoll() (assuming linux). In that case the entire process is
> idle. So just look at the process CPU usage?

I thought gevent is non-blocking. So why would a gevent process block
on a call to epoll()? Can you explain?

[Bob Van Zant]

On a linux machine, at the end of the day, gevent is an event
processor around epoll() (let's ignore libevent for a moment).

It looks a bit like this (a python-like pseudo code):

while 1:
events_to_process = epoll()
for event in events_to_process:
greenlet = find_greenlet_for_event(event)
run_greenlet(greenlet)

When there is nothing to do we sit blocked on epoll() waiting for an
I/O event to occur.

The easy thing to screw up with gevent is what we could call a
gevent-unsafe operation. Imagine that you call into some C library
(one of the MySQL drivers for example) and it creates a connection to
some server. All of the socket operations for that connection
(connect(), send(), recv()) are done in C and thus monkey patching
does nothing. All of those operations are blocking operations and when
they block, they block the whole darn process. The epoll() call in the
event loop never gets called and all of the other greenlets are
starved of events to process. On a local network where MySQL is fast
these blocking calls may only take a few milliseconds each and the
overall effect on the gevent application may be negligible in the
common case. One bad query that takes too long, however, might
introduce unacceptable latency.

Well written gevent applications that can handle these sorts of
scenarios are hard to write correctly. You can goof around with
threads and multiple event loops. You can wrap MySQL in some sort of
an application layer proxy and talk to it using gevent-safe socket
operations. Or apply the ostrich algorithm: stick your head in the
sand and hope nothing ever goes really wrong.

-Bob