Re: Tracebacks when using gevent and psycopg2 go "missing?"

[Oliver Beattie]

I've done some digging, and managed to figure out what's happening. Specifically, it seems as if under some condition that gevent deliberately clears the traceback information. Specifically, this commit from way back in 2010:

https://bitbucket.org/denis/gevent/changeset/51f0fddafa9868841fb77d8232d14861a7e0cf92

I've confirmed that this is what's happening by commenting out the lines to exc_clear() and core.set_exc_info in my local version of hub.py, and the traceback is preserved.

Is there any way around this? It would be great if someone could shed more light on why this is being done, it's not clear to me why you'd deliberately throw away traceback objects? I assume perhaps a memory leak issue?

On Friday, 7 September 2012 10:22:00 UTC+1, Oliver Beattie wrote:

Hi,

I'm having a bit of a frustrating time trying to track down an issue I'm seeing with Python/gevent in production, which is making things extremely difficult for us to debug our application. It seems that some of the time our tracebacks can "go missing". By this I mean that the traceback part of sys.exc_info() is None.

As mentioned in this GitHub issue a while back, it is caused by a piece of code to patch psycopg2 to be green. Given that when using an eventlet worker under Gunicorn, this does not happen, I'm inclined to think the source of this must be gevent… but looking at the source I'm at a loss as to where I might begin to try tracking something like this down.

I was wondering if anyone might be able to offer any advice on how to track this down?

Many thanks

[Denis Bilenko]

It's a well-known (or perhaps just "known" :) ) issue with gevent:

try:
  1/0
except:
  # here the traceback is available: raise would work as expected
  gevent.sleep(0)
  # here the traceback is gone: raise would work and do raise ZeroDivisionError but the original stacktrace is lost

I can only suggest a work around: either print the stacktrace immediately or save it in text form for later.

On Fri, Sep 7, 2012 at 2:00 PM, Oliver Beattie <oli...@obeattie.com> wrote:

I've done some digging, and managed to figure out what's happening. Specifically, it seems as if under some condition that gevent deliberately clears the traceback information. Specifically, this commit from way back in 2010:

https://bitbucket.org/denis/gevent/changeset/51f0fddafa9868841fb77d8232d14861a7e0cf92

I've confirmed that this is what's happening by commenting out the lines to exc_clear() and core.set_exc_info in my local version of hub.py, and the traceback is preserved.

You now have exc_info() raised in one greenlet accessible by all other greenlets (and they are able to override it too). Much worse bug.

Is there any way around this? It would be great if someone could shed more light on why this is being done, it's not clear to me why you'd deliberately throw away traceback objects? I assume perhaps a memory leak issue?

Yes, there were indeed a problem (leak) when trying to save/restore the traceback part of exc_info like we do with exc_type and exc_value. I'm not sure if was just a bug or fundamental issue of how greenlet interacts with PyThreadState. If someone wants to look at it, any insight into what's going on would much appreciated.

eventlet seemingly does not have it because it completely ignores the fact that all greenlets share PyThreadState (and thus exc_info()) and leaks sys.exc_info() into unrelated contexts.

 

[Denis Bilenko]

On Fri, Sep 7, 2012 at 2:20 PM, Denis Bilenko <denis....@gmail.com> wrote:

eventlet seemingly does not have it because it completely ignores the fact that all greenlets share PyThreadState (and thus exc_info()) and leaks sys.exc_info() into unrelated contexts.

I was wrong about eventlet - it does clean up exc_info. It never restores it, so you get:

denis@ubuntu:~/test_greenlet$ cat eventlet_test.py
import eventlet
try:
    1/0
except:
    eventlet.sleep()
    raise

denis@ubuntu:~/test_greenlet$ bin/python eventlet_test.py
Traceback (most recent call last):
  File "eventlet_test.py", line 7, in <module>
    raise
TypeError: exceptions must be old-style classes or derived from BaseException, not NoneType

[Oliver Beattie]

Is there any way this can be the behaviour in Gevent? How is it that the memory leaks can occur? I'd be willing to work on this…

[Denis Bilenko]

On Thu, Sep 27, 2012 at 11:28 AM, Oliver Beattie <oli...@obeattie.com> wrote:
> Is there any way this can be the behaviour in Gevent? How is it that the
> memory leaks can occur? I'd be willing to work on this…

Apparently, the issue is fixed in greenlet since 0.3.2. So I just
removed restore/cleanup of exception information.

https://github.com/SiteSupport/gevent/commit/554ed446db75ed6b15f277ccc0094140d2b03e88

[Oliver Beattie]

On 27 October 2012 21:57, Denis Bilenko <denis....@gmail.com> wrote:
>
> On Thu, Sep 27, 2012 at 11:28 AM, Oliver Beattie <oli...@obeattie.com> wrote:
> > Is there any way this can be the behaviour in Gevent? How is it that the
> > memory leaks can occur? I'd be willing to work on this…
>
> Apparently, the issue is fixed in greenlet since 0.3.2. So I just
> removed restore/cleanup of exception information.
>
> https://github.com/SiteSupport/gevent/commit/554ed446db75ed6b15f277ccc0094140d2b03e88

Amazing. Thanks so much for all your work on gevent, it's brilliant.