How to detect slow-running greenlets?

[Oliver Wilkie]

I have a suspicion that one greenlet is hogging CPU and not yielding to allow other, nimbler greenlets to run.

Does anyone have any good tips on how to add some profiling which will reveal the slow running greenlet?

I would typically use something like New Relic to monitor slow transactions but New Relic transaction tracing does not support Greenlets.

Any advice is welcomed!

Oli

[Shahaf Abileah]

I ran into a similar problem and used the following code...

    # Log how much time each greenlet runs before it yields control back to the gevent loop.

    #

    # This can help diagnose issues where it appears that some greenlets are running for too

    # long before yielding, and thus causing other greenlets to starve.

    #

    # More info:

    # https://groups.google.com/forum/#!topic/gevent/pfy-jx3O_yM

    # http://www.rfk.id.au/blog/entry/detect-gevent-blocking-with-greenlet-settrace/

    if environ.get('LOG_GREENLET_RUN_DURATIONS') and environ.get('LOG_GREENLET_RUN_DURATIONS').lower() == 'true':

        import time, greenlet, gevent.hub, threading

        MIN_DURATION_TO_LOG = float(environ.get('MIN_GREENLET_RUN_DURATION_TO_LOG', 0.5)) # seconds

        def log_greenlet_run_duration(what, (origin, target)):

            global _last_switch_time

            then = _last_switch_time

            now = _last_switch_time = time.time()

            if then is not None:

                blocking_time = now - then

                if origin is not gevent.hub.get_hub() and blocking_time > MIN_DURATION_TO_LOG:

                    msg = "Greenlet ran for %.4f seconds (%s from %s %s to %s %s).\n"

                    msg = msg % (blocking_time, what, type(origin), id(origin), type(target), id(target))

                    print msg

        greenlet.settrace(log_greenlet_run_duration)

--S

--
You received this message because you are subscribed to the Google Groups "gevent: coroutine-based Python network library" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gevent+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

http://twitter.com/shahafabileah

[Oliver Wilkie]

Many thanks, that settrace is very helpful

greenlet.settrace(switch_time_tracer)

In the comments section of that blog post I found the link to some Mozilla source code to be very helpful too

https://github.com/mozilla-services/mozservices/blob/master/mozsvc/gunicorn_worker.py

​