greenlet stack

[Tarek Ziadé]

Hey

I have this piece of code I use to dump a stack for each running
thread :

<<<
import threading,
sys

threads = dict([(th.ident, th.name) for th in
threading.enumerate()])

dump = []

for thread, frame in
sys._current_frames().items():.
dump.append('Thread 0x%x (%s)' % (thread, threads[thread]))
dump.append(''.join(traceback.format_stack(frame)))
<<<<

This is useful to watch a process and know what code is being
executed.

The problem is it cease to work with gevent activate (e.g. that has
patched the std lib)

I am able to get the frame of the main greenlet (the hub) via
gevent.getcurrent().gr_frame but of course I don't get much
information since the code that's being executed is on other
greenlets.

So I was wondering: how can I enumerate all "active" frames in order
to print their stacks like I would do with threads ? If this feature
does not exists, how hard would it be to add a registry to track all
greenlets ?

Thanks!

Tarek

[Denis Bilenko]

On Tue, May 15, 2012 at 11:33 PM, Tarek Ziadé <ziade...@gmail.com> wrote:
> So I was wondering: how can I enumerate all "active" frames in order
> to print their stacks like I would do with threads ? If this feature
> does not exists, how hard would it be to add a registry to track all
> greenlets ?

I don't think threading.enumerate() is supposed to work with
greenlets. It only enumerates Thread() objects. It does not even work
with threads created by other means, e.g. thread.start_new_thread.

There won't be a registry of all greenlets in gevent as that will come
with overhead.

Maybe it could be possible to modify greenlet to enumerate greenlets,
perhaps it's worth looking into exposing stack_prev member
(https://github.com/python-greenlet/greenlet/blob/master/greenlet.h#L21)

We also could walk over all watchers from libev and that can be used
to get all currently active greenlets.

[Tarek Ziadé]

Thanks for the tips, I'll investigate

On May 22, 10:02 am, Denis Bilenko <denis.bile...@gmail.com> wrote:

[Tarek Ziadé]

On a side note, not being able to know exactly what's going on in
gevent at a given instant, makes my life miserable to fix a bug right
now.

Maybe that could be a debug-only feature ?

On May 22, 10:02 am, Denis Bilenko <denis.bile...@gmail.com> wrote:

[Floris Bruynooghe]

On 22 May 2012 09:02, Denis Bilenko <denis....@gmail.com> wrote:
> On Tue, May 15, 2012 at 11:33 PM, Tarek Ziadé <ziade...@gmail.com> wrote:
>> So I was wondering: how can I enumerate all "active" frames in order
>> to print their stacks like I would do with threads ? If this feature
>> does not exists, how hard would it be to add a registry to track all
>> greenlets ?
>
> I don't think threading.enumerate() is supposed to work with
> greenlets. It only enumerates Thread() objects. It does not even work
> with threads created by other means, e.g. thread.start_new_thread.
>
> There won't be a registry of all greenlets in gevent as that will come
> with overhead.
>
> Maybe it could be possible to modify greenlet to enumerate greenlets,
> perhaps it's worth looking into exposing stack_prev member
> (https://github.com/python-greenlet/greenlet/blob/master/greenlet.h#L21)

I just remembered that Alexey Borzenkov has recently been working on a
tracing callback for debugging purposes of greenlets:
https://github.com/python-greenlet/greenlet/issues/5 There is even a
branch with working code. I don't think it's very hard to write such
a callback which keeps track of all greenlets in a thread-local dict,
which would give you what you need.


Regards,
Floris

--
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

[Tarek Ziadé]

On May 23, 10:36 am, Floris Bruynooghe <f...@devork.be> wrote:
> On 22 May 2012 09:02, Denis Bilenko <denis.bile...@gmail.com> wrote:

>
>
>
>
>
>
>
>
>
> > On Tue, May 15, 2012 at 11:33 PM, Tarek Ziadé <ziade.ta...@gmail.com> wrote:
> >> So I was wondering: how can I enumerate all "active" frames in order
> >> to print their stacks like I would do with threads ? If this feature
> >> does not exists, how hard would it be to add a registry to track all
> >> greenlets ?
>
> > I don't think threading.enumerate() is supposed to work with
> > greenlets. It only enumerates Thread() objects. It does not even work
> > with threads created by other means, e.g. thread.start_new_thread.
>
> > There won't be a registry of all greenlets in gevent as that will come
> > with overhead.
>
> > Maybe it could be possible to modify greenlet to enumerate greenlets,
> > perhaps it's worth looking into exposing stack_prev member
> > (https://github.com/python-greenlet/greenlet/blob/master/greenlet.h#L21)
>
> I just remembered that Alexey Borzenkov has recently been working on a
> tracing callback for debugging purposes of greenlets:https://github.com/python-greenlet/greenlet/issues/5 There is even a
> branch with working code.  I don't think it's very hard to write such
> a callback which keeps track of all greenlets in a thread-local dict,
> which would give you what you need.
>

Thanks for the tip, that leaded me to another tip: I used gc

Just for the record if someone else wants to do this:

https://github.com/mozilla-services/powerhose/blob/master/powerhose/util.py#L200