Is it possible to track "load average" (waiting greenlets) in gevent?

[Shahaf Abileah]

I would like to measure something like "load average" for gevent: http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages

In other words, I'd like to know how many greenlets are ready to run but are waiting for their turn (i.e. waiting for the greenlets ahead of them to yield).

My process is running on Heroku, and Heroku has a way to log the standard unix load average: https://devcenter.heroku.com/articles/log-runtime-metrics.

But I expect that the standard unix load average won't help me.  If my gevent process is perfectly loaded I would expect to see a load average of 1.  And if my process is heavily overloaded I would still expect to see a unix load average of 1.  As far as unix is concerned, there's just one process to run, and it's getting CPU, so there's no problem.

So, is there a way for me to tap into gevent's internals in order to periodically log the size of the ready queue?

Note, this is related to a discussion in another thread, but I didn't see this specific question answered: https://groups.google.com/forum/#!searchin/gevent/%22load$20average%22/gevent/aCnd7qY5iiQ/bp05rbW_o2oJ

thanks!

--Shahaf

[Matt Billenstein]

I've been curious about this - from looking through the source, it seems like
the hub loop has a couple interesting properties, activecnt and pending cnt.
Spawning 100 greenlets that just sleep every 100ms, and then polling these
values gives:

(venv)mattb@mattb-mbp:~ $ ./g.py
1434065031.584 99 1
1434065032.586 10 90
1434065033.589 100 0
1434065034.589 95 5
1434065035.590 100 0
1434065036.591 100 0
...

The comments in the source are sparse, but it appears pending means pending
events waiting to be handled which is perhaps something like a run queue?

The script:

#!/usr/bin/env python

import gevent.monkey
gevent.monkey.patch_all()

import gevent
import gevent.hub
import time

def g():
while 1:
time.sleep(0.01)

greenlets = [gevent.spawn(g) for _ in xrange(100)]

hub = gevent.hub.get_hub()
loop = hub.loop
while True:
time.sleep(1)
print '%.3f %d %d' % (time.time(), loop.activecnt, loop.pendingcnt)

gevent.joinall(greenlets)
print 'done'

m

> --
> You received this message because you are subscribed to the Google Groups
> "gevent: coroutine-based Python network library" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to gevent+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.


--
Matt Billenstein
ma...@vazor.com
http://www.vazor.com/

[Shahaf Abileah]

Thanks Matt!

I ran a couple more little experiments to check whether one of these values (pendingcnt or activecnt) gives me the value I expect.

EXPERIMENT 1: EMPTY RUN QUEUE

This script simulates 5 greenlets making (yielding) API calls in a staggered fashion, such that we never have two greenlets that want to run at the same time.  I would expect the run queue to be empty at all times.

The code:

import gevent.monkey

gevent.monkey.patch_all()

import gevent

import gevent.hub

import time

hub = gevent.hub.get_hub()

loop = hub.loop

def worker(worker_id):

    for iteration in xrange(3):

        print 'worker: %d, pendingcnt: %d, activecnt: %d' % (worker_id, loop.pendingcnt, loop.activecnt)

        # Simulate making a yielding call to some API.  The call takes 3 seconds.

        time.sleep(3)

# Spawn 5 greenlets in a staggered fashion, kicking them of 0.5 seconds

# apart.  This ensure that each greenlet will make its fake API call at

# a different time, which means that the API call will also finish at

# a different time.  Bottom line: each greenlet will want to run at a

# different time (no waiting).

greenlets = []

for worker_id in xrange(5):

    greenlet = gevent.spawn(worker, worker_id)

    greenlets.append(greenlet)

    time.sleep(.5)

gevent.joinall(greenlets)

print 'done'

Here's what I get when I run it:

(venv)~/work> python g1.py 

worker: 0, pendingcnt: 0, activecnt: 1

worker: 1, pendingcnt: 0, activecnt: 2

worker: 2, pendingcnt: 0, activecnt: 3

worker: 3, pendingcnt: 0, activecnt: 4

worker: 4, pendingcnt: 0, activecnt: 5

worker: 0, pendingcnt: 0, activecnt: 4

worker: 1, pendingcnt: 0, activecnt: 4

worker: 2, pendingcnt: 0, activecnt: 4

worker: 3, pendingcnt: 0, activecnt: 4

worker: 4, pendingcnt: 0, activecnt: 4

worker: 0, pendingcnt: 0, activecnt: 4

worker: 1, pendingcnt: 0, activecnt: 4

worker: 2, pendingcnt: 0, activecnt: 4

worker: 3, pendingcnt: 0, activecnt: 4

worker: 4, pendingcnt: 0, activecnt: 4

done

So far, pendingcnt gives exactly the result I would expect to see for the size of the run queue.

EXPERIMENT 2: FULL RUN QUEUE

This time I spawn 5 greenlets at the same time.  Each of them simulates hogging the CPU for 0.5 seconds and then yielding.  I would expect the run queue to always be full (it should always have the other 4 greenlets).

The code:

import gevent.monkey

gevent.monkey.patch_all()

import gevent

import gevent.hub

import time

hub = gevent.hub.get_hub()

loop = hub.loop

def worker(worker_id):

    # First yield so we'll get a chance to spawn all the greenlets.

    time.sleep(0.001)

    for iteration in xrange(3):

        print 'worker: %d, pendingcnt: %d, activecnt: %d' % (worker_id, loop.pendingcnt, loop.activecnt)

        # Simulate actively using the CPU for 0.5 seconds.

        start_time = time.time()

        target_time = start_time + 0.5

        while time.time() < target_time:

            pass

        # Now yield so others can run.

        time.sleep(0.001)

# Kick off all the greenlets at once.  Each one will perform busy work

# so we should expect to see some contention for CPU.

greenlets = [gevent.spawn(worker, worker_id) for worker_id in xrange(5)]

gevent.joinall(greenlets)

print 'done'

Here's what I get when I run it:

(venv)~/work> python g2.py 

worker: 0, pendingcnt: 4, activecnt: 0

worker: 1, pendingcnt: 3, activecnt: 1

worker: 2, pendingcnt: 2, activecnt: 2

worker: 3, pendingcnt: 1, activecnt: 3

worker: 4, pendingcnt: 0, activecnt: 4

worker: 0, pendingcnt: 3, activecnt: 1

worker: 1, pendingcnt: 2, activecnt: 2

worker: 2, pendingcnt: 1, activecnt: 3

worker: 3, pendingcnt: 0, activecnt: 4

worker: 4, pendingcnt: 3, activecnt: 1

worker: 0, pendingcnt: 2, activecnt: 2

worker: 1, pendingcnt: 1, activecnt: 3

worker: 2, pendingcnt: 0, activecnt: 4

worker: 3, pendingcnt: 3, activecnt: 1

worker: 4, pendingcnt: 2, activecnt: 2

done

So, this is directionally good but it's not exactly the result I would expect.  Can anyone explain why pendingcnt has this repeating pattern of 3, 2, 1, 0?  Do you agree with my claim that the run queue should be pegged at 4?

thanks!

--S

[Matt Billenstein]

On Fri, Jun 12, 2015 at 09:54:52AM -0700, Shahaf Abileah wrote:
> EXPERIMENT 2: FULL RUN QUEUE
> This time I spawn 5 greenlets at the same time. Each of them simulates
> hogging the CPU for 0.5 seconds and then yielding. I would expect the run
> queue to always be full (it should always have the other 4 greenlets).

Only one at a time can be on the cpu, so I think you're seeing them basically
interleave on the time.sleep at the end of the loop. So as they switch after
the print, they go from pending to active one at a time until they all run one
pass, then they start over again. I think.

This isn't a typically good thing to be doing in gevent anyway since you're
blocking everything else while one greenlet gets to run.

m