Tornado memory leak

[David Koblas]

Found this on StackOverflow

   http://stackoverflow.com/questions/13403760/memory-leak-using-tornados-gen-engine

from tornado import gen, httpclient, ioloop

io_loop = ioloop.
 IOLoop.instance()
client = httpclient.AsyncH
TTPClient(io_loop=io_l
 oop)

@gen.engine
def go_for_it():
    while True:
        r = yield ge
 n<
span class="pun" style="margin: 0px; padding: 0px; border: 0px; font-size: 14px; vertical-align: baseline; background-color: transparent; color: rgb(0, 0, 0); background-position: initial initial; background-repeat: initial initial;">.Task(fetch)

@gen.engine
def fetch(callback):
    response = yield
  gen.Task(client.fetch, 'http://localhost:8888/')
    callback(response)

io_loop.add_callback(g
 o_for_it<
/span>)
io_loop.start()

Leaks memory - looking deeper it appears that ExceptionStackContexts are not being cleaned up.  Though I can't quite figure out why since everything appears to remove all the references.

--koblas

[Serge S. Koval]

Here's simplified test case: https://gist.github.com/4086529 - it is reproducible with any asynchronous operation, even add_callback.

I have suspicion it has something to do with recursion.

Here's supposed workflow:

1. go_for_it() calls fetch()

2. fetch() creates nested ExceptionStackContext

3. fetch() calls add_callback() and it will wrap() callback to maintain current ExceptionStackContext

4. on next tick, IOLoop will call wrapped callback function

5. wrap() will explicitly restore chain of the ExceptionStackContext by calling __enter__ for each item in stack and then continue fetch() execution

6. fetch() will call callback() with restored stack context

7. callback() will return control to go_for_it() with restored stack context

8. go_for_it() will start next 'while' iteration by calling go_for_it() while in restored stack context

9. jump to #2

I found only one workaround: wrap loop code in go_for_it() with NullContext.

Unfortunately, using add_callback to create endless loop won't work either - ExceptionStackContext stack will continue to grow, as it will run in the fetch() context.

Not sure how to fix it, to be honest, as it is working "as expected", though it took me hour to figure out what's going on.

P.S. enter/leave flow looks like this:

Enter 0

Enter 1

! fetch()

Leave 1

Leave 0

Enter 0

Enter 1

Enter 2

! fetch()

Leave 2

Leave 1

Leave 0

Enter 0

Enter 1

Enter 2

Enter 3

! fetch()

Leave 3

Leave 2

Leave 1

Leave 0

... and so on. Number next to enter/leave = len(_state.contexts)

Serge.

[Serge S. Koval]

On a side note, with the approach I proposed some time before (implicit callbacks, code here: https://gist.github.com/3891601, line 104), this is no longer the problem, as continuation will happen outside of the ExceptionStackContext.

Serge.

[Ben Darnell]

I think I have a fix:

https://github.com/facebook/tornado/commit/bff07405549a6eb173a4cfc9bbc3fc7c6da5cdd7

-Ben

[Serge S. Koval]

Not sure it is proper fix - this fixed leak if exception happens in inner function. However, original question was related to inner function making callback function  call.

Is there a way to fix it without using NullContext before calling callback function?

Serge.

[Ben Darnell]

On Sat, Nov 17, 2012 at 7:59 PM, Serge S. Koval <serge...@gmail.com> wrote:

Not sure it is proper fix - this fixed leak if exception happens in inner function. However, original question was related to inner function making callback function  call.

As far as I can tell this change works whether there was an exception or not - I tested with your gist https://gist.github.com/4086529, which doesn't have exceptions.  Is there another case I'm missing?  (the test case I added in this commit was actually unrelated; I wanted to see whether the fact that the "except Exception" clause in gen.py didn't call deactivate_stack_context was a problem).

-Ben

[Serge S. Koval]

Ah, you're right. I saw unit test and thought that it is testing my test case.

Now I understand why it was fixed - StackExceptionContext will forget about old contexts and wrap() is smart enough to ignore disabled ExceptionContext even when there's recursion.

Thanks,

Serge.