Tornado memory leak - Tornado Web Server

The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Message from discussion Tornado memory leak
Received-SPF: pass (google.com: domain of serge.ko...@gmail.com designates 74.125.82.179 as permitted sender) client-ip=74.125.82.179;
MIME-Version: 1.0
In-Reply-To: <50A5D8AD.2030...@koblas.com>
References: <50A5D8AD.2030...@koblas.com>
Date: Fri, 16 Nov 2012 14:06:33 +0200
Message-ID: <CAJZrG8jeiPniXb0Xhw6b8OO4wgsucG6+y_xPEyuTmuxyN7c...@mail.gmail.com>
Subject: Re: [tornado] Tornado memory leak
From: "Serge S. Koval" <serge.ko...@gmail.com>
To: python-tornado@googlegroups.com
Content-Type: multipart/alternative; boundary=0016e6db2b03444b3b04ce9b9995

--0016e6db2b03444b3b04ce9b9995
Content-Type: text/plain; charset=ISO-8859-1

Here's simplified test case: https://gist.github.com/4086529 - it is
reproducible with any asynchronous operation, even add_callback.

I have suspicion it has something to do with recursion.

Here's supposed workflow:
1. go_for_it() calls fetch()
2. fetch() creates nested ExceptionStackContext
3. fetch() calls add_callback() and it will wrap() callback to maintain
current ExceptionStackContext
4. on next tick, IOLoop will call wrapped callback function
5. wrap() will explicitly restore chain of the ExceptionStackContext by
calling __enter__ for each item in stack and then continue fetch() execution
6. fetch() will call callback() with restored stack context
7. callback() will return control to go_for_it() with restored stack context
8. go_for_it() will start next 'while' iteration by calling go_for_it()
while in restored stack context
9. jump to #2

I found only one workaround: wrap loop code in go_for_it() with NullContext.

Unfortunately, using add_callback to create endless loop won't work either
- ExceptionStackContext stack will continue to grow, as it will run in the
fetch() context.

Not sure how to fix it, to be honest, as it is working "as expected",
though it took me hour to figure out what's going on.

P.S. enter/leave flow looks like this:

Enter 0
Enter 1
! fetch()
Leave 1
Leave 0
Enter 0
Enter 1
Enter 2
! fetch()
Leave 2
Leave 1
Leave 0
Enter 0
Enter 1
Enter 2
Enter 3
! fetch()
Leave 3
Leave 2
Leave 1
Leave 0

... and so on. Number next to enter/leave = len(_state.contexts)

Serge.

On Fri, Nov 16, 2012 at 8:09 AM, David Koblas <da...@koblas.com> wrote:

>  Found this on StackOverflow
>
>
> http://stackoverflow.com/questions/13403760/memory-leak-using-tornados-gen-engine
>
> from tornado import gen, httpclient, ioloop
>
> io_loop = ioloop.
>  IOLoop.instance()
> client = httpclient.AsyncH
> TTPClient(io_loop=io_l
>  oop)
> @gen.enginedef go_for_it():
>     while True:
>         r = yield ge
>  n<
> span class="pun" style="margin: 0px; padding: 0px; border: 0px; font-size: 14px; vertical-align: baseline; background-color: transparent; color: rgb(0, 0, 0); background-position: initial initial; background-repeat: initial initial;">.Task(fetch)
> @gen.enginedef fetch(callback****):
>     response = yield
>   gen**.Task(client**.fetch, 'http://localhost:8888/')
>     callback(response)
>
> io_loop.add_callback(g
>  o_for_it<
> /span>)
> io_loop.start()
>
>
> Leaks memory - looking deeper it appears that ExceptionStackContexts are
> not being cleaned up.  Though I can't quite figure out why since everything
> appears to remove all the references.
>
> --koblas
>

--0016e6db2b03444b3b04ce9b9995
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Here&#39;s simplified test case:=A0<a href=3D"https://gist.github.com/40865=
29">https://gist.github.com/4086529</a> - it is reproducible with any async=
hronous operation, even add_callback.<br><div><br></div><div>I have suspici=
on it has something to do with recursion.<div>
<br></div></div><div>Here&#39;s supposed workflow:</div><div>1. go_for_it()=
 calls fetch()</div><div>2. fetch() creates nested ExceptionStackContext</d=
iv><div>3. fetch() calls add_callback() and it will wrap() callback to main=
tain current ExceptionStackContext</div>
<div>4. on next tick, IOLoop will call wrapped callback function</div><div>=
5. wrap() will explicitly restore chain of the ExceptionStackContext by cal=
ling __enter__ for each item in stack and then continue fetch() execution</=
div>
<div>6. fetch() will call callback() with restored stack context</div><div>=
7. callback() will return control to go_for_it() with restored stack contex=
t</div><div>8. go_for_it() will start next &#39;while&#39; iteration by cal=
ling go_for_it() while in restored stack context</div>
<div>9. jump to #2</div><div><br></div><div>I found only one workaround: wr=
ap loop code in go_for_it() with NullContext.</div><div><br></div><div>Unfo=
rtunately, using add_callback to create endless loop won&#39;t work either =
- ExceptionStackContext stack will continue to grow, as it will run in the =
fetch() context.</div>
<div><br></div><div><div>Not sure how to fix it, to be honest, as it is wor=
king &quot;as expected&quot;, though it took me hour to figure out what&#39=
;s going on.</div></div><div><br></div><div>P.S. enter/leave flow looks lik=
e this:</div>
<div><br></div><div><div><div>Enter 0</div></div><div><div>Enter 1</div></d=
iv><div>! fetch()</div><div>Leave 1</div><div>Leave 0</div><div>Enter 0</di=
v><div>Enter 1</div><div>Enter 2</div><div>! fetch()</div><div>Leave 2</div=
>
<div>Leave 1</div><div>Leave 0</div><div>Enter 0</div><div>Enter 1</div><di=
v>Enter 2</div><div>Enter 3<br><div>! fetch()</div><div>Leave 3</div><div>L=
eave 2</div><div>Leave 1</div><div>Leave 0</div><div><br></div><div>... and=
 so on. Number next to enter/leave =3D len(_state.contexts)</div>
<div><br></div><div>Serge.<br><br><div class=3D"gmail_quote">On Fri, Nov 16=
, 2012 at 8:09 AM, David Koblas <span dir=3D"ltr">&lt;<a href=3D"mailto:dav=
i...@koblas.com" target=3D"_blank">da...@koblas.com</a>&gt;</span> wrote:<br>=
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">

 =20

   =20
 =20
  <div bgcolor=3D"#FFFFFF" text=3D"#000000">
    Found this on StackOverflow<br>
    <br>
    =A0=A0
<a href=3D"http://stackoverflow.com/questions/13403760/memory-leak-using-to=
rnados-gen-engine" target=3D"_blank">http://stackoverflow.com/questions/134=
03760/memory-leak-using-tornados-gen-engine</a><br>
    <br>
   =20
    <pre style=3D"max-height:600px;text-indent:0px;letter-spacing:normal;te=
xt-align:left;font-variant:normal;width:auto;font-style:normal;background-c=
olor:rgb(238,238,238);font-weight:normal;padding:5px;vertical-align:baselin=
e;line-height:18px;text-transform:none;font-size:14px;overflow:auto;margin:=
0px 0px 10px;font-family:Consolas,Menlo,Monaco,&#39;Lucida Console&#39;,&#3=
9;Liberation Mono&#39;,&#39;DejaVu Sans Mono&#39;,&#39;Bitstream Vera Sans =
Mono&#39;,&#39;Courier New&#39;,monospace,serif;border:0px;background-repea=
t:initial initial;word-spacing:0px">
<code><span style=3D"margin:0px;padding:0px;border:0px;font-size:14px;verti=
cal-align:baseline;background-color:transparent;color:rgb(0,0,139);backgrou=
nd-repeat:initial initial">from</span><span style=3D"vertical-align:baselin=
e;font-size:14px;background-color:transparent;margin:0px;border:0px;padding=
:0px;background-repeat:initial initial"> tornado </span><span style=3D"marg=
in:0px;padding:0px;border:0px;font-size:14px;vertical-align:baseline;backgr=
ound-color:transparent;color:rgb(0,0,139);background-repeat:initial initial=
">import</span><span> gen</span><span style=3D"vertical-align:baseline;font=
-size:14px;background-color:transparent;margin:0px;border:0px;padding:0px;b=
ackground-repeat:initial initial">,</span><span style=3D"vertical-align:bas=
eline;font-size:14px;background-color:transparent;margin:0px;border:0px;pad=
ding:0px;background-repeat:initial initial"> httpclient</span><span style=
=3D"vertical-align:baseline;font-size:14px;background-color:transparent;mar=
gin:0px;border:0px;padding:0px;background-repeat:initial initial">,</span><=
span> ioloop

io_loop </span><span style=3D"vertical-align:baseline;font-size:14px;backgr=
ound-color:transparent;margin:0px;border:0px;padding:0px;background-repeat:=
initial initial">=3D</span><span style=3D"vertical-align:baseline;font-size=
:14px;background-color:transparent;margin:0px;border:0px;padding:0px;backgr=
ound-repeat:initial initial"> ioloop</span><span style=3D"vertical-align:ba=
seline;font-size:14px;background-color:transparent;margin:0px;border:0px;pa=
dding:0px;background-repeat:initial initial">.</span><span style=3D"margin:=
0px;padding:0px;border:0px;font-size:14px;vertical-align:baseline;backgroun=
d-color:transparent;color:rgb(43,145,175);background-repeat:initial initial=
">
 IOLoop<span style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vert=
ical-align:baseline;background-color:transparent;color:rgb(0,0,0);backgroun=
d-repeat:initial initial">.</span><span style=3D"margin:0px;padding:0px;bor=
der:0px;font-size:14px;vertical-align:baseline;background-color:transparent=
;color:rgb(0,0,0);background-repeat:initial initial">instance</span><span s=
tyle=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertical-align:bas=
eline;background-color:transparent;color:rgb(0,0,0);background-repeat:initi=
al initial">()</span><span style=3D"margin:0px;padding:0px;border:0px;font-=
size:14px;vertical-align:baseline;background-color:transparent;color:rgb(0,=
0,0);background-repeat:initial initial">
client </span><span style=3D"margin:0px;padding:0px;border:0px;font-size:14=
px;vertical-align:baseline;background-color:transparent;color:rgb(0,0,0);ba=
ckground-repeat:initial initial">=3D</span><span style=3D"margin:0px;paddin=
g:0px;border:0px;font-size:14px;vertical-align:baseline;background-color:tr=
ansparent;color:rgb(0,0,0);background-repeat:initial initial"> httpclient</=
span><span style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertic=
al-align:baseline;background-color:transparent;color:rgb(0,0,0);background-=
repeat:initial initial">.</span><span style=3D"margin:0px;padding:0px;borde=
r:0px;font-size:14px;vertical-align:baseline;background-color:transparent;c=
olor:rgb(43,145,175);background-repeat:initial initial">AsyncH
TTPClient</span><span style=3D"margin:0px;padding:0px;border:0px;font-size:=
14px;vertical-align:baseline;background-color:transparent;color:rgb(0,0,0);=
background-repeat:initial initial">(</span><span style=3D"margin:0px;paddin=
g:0px;border:0px;font-size:14px;vertical-align:baseline;background-color:tr=
ansparent;color:rgb(0,0,0);background-repeat:initial initial">io_loop</span=
><span style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertical-a=
lign:baseline;background-color:transparent;color:rgb(0,0,0);background-repe=
at:initial initial">=3D</span><span style=3D"margin:0px;padding:0px;border:=
0px;font-size:14px;vertical-align:baseline;background-color:transparent;col=
or:rgb(0,0,0);background-repeat:initial initial">io_l
 oop</span><span style=3D"margin:0px;padding:0px;border:0px;font-size:14px;=
vertical-align:baseline;background-color:transparent;color:rgb(0,0,0);backg=
round-repeat:initial initial">)</span><span style=3D"margin:0px;padding:0px=
;border:0px;font-size:14px;vertical-align:baseline;background-color:transpa=
rent;color:rgb(0,0,0);background-repeat:initial initial">

</span><span style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vert=
ical-align:baseline;background-color:transparent;color:rgb(128,0,0);backgro=
und-repeat:initial initial">@gen</span><span style=3D"margin:0px;padding:0p=
x;border:0px;font-size:14px;vertical-align:baseline;background-color:transp=
arent;color:rgb(0,0,0);background-repeat:initial initial">.</span><span sty=
le=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertical-align:basel=
ine;background-color:transparent;color:rgb(0,0,0);background-repeat:initial=
 initial">engine
</span><span style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vert=
ical-align:baseline;background-color:transparent;color:rgb(0,0,139);backgro=
und-repeat:initial initial">def</span><span style=3D"margin:0px;padding:0px=
;border:0px;font-size:14px;vertical-align:baseline;background-color:transpa=
rent;color:rgb(0,0,0);background-repeat:initial initial"> go_for_it</span><=
span style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertical-ali=
gn:baseline;background-color:transparent;color:rgb(0,0,0);background-repeat=
:initial initial">():</span><span style=3D"margin:0px;padding:0px;border:0p=
x;font-size:14px;vertical-align:baseline;background-color:transparent;color=
:rgb(0,0,0);background-repeat:initial initial">
    </span><span style=3D"margin:0px;padding:0px;border:0px;font-size:14px;=
vertical-align:baseline;background-color:transparent;color:rgb(0,0,139);bac=
kground-repeat:initial initial">while</span><span style=3D"margin:0px;paddi=
ng:0px;border:0px;font-size:14px;vertical-align:baseline;background-color:t=
ransparent;color:rgb(0,0,0);background-repeat:initial initial"> </span><spa=
n style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertical-align:=
baseline;background-color:transparent;color:rgb(0,0,139);background-repeat:=
initial initial">True</span><span style=3D"margin:0px;padding:0px;border:0p=
x;font-size:14px;vertical-align:baseline;background-color:transparent;color=
:rgb(0,0,0);background-repeat:initial initial">:<span style=3D"margin:0px;p=
adding:0px;border:0px;font-size:14px;vertical-align:baseline;background-col=
or:transparent;color:rgb(0,0,0);background-repeat:initial initial">
        r </span><span style=3D"margin:0px;padding:0px;border:0px;font-size=
:14px;vertical-align:baseline;background-color:transparent;color:rgb(0,0,0)=
;background-repeat:initial initial">=3D</span><span style=3D"margin:0px;pad=
ding:0px;border:0px;font-size:14px;vertical-align:baseline;background-color=
:transparent;color:rgb(0,0,0);background-repeat:initial initial"> </span><s=
pan style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertical-alig=
n:baseline;background-color:transparent;color:rgb(0,0,139);background-repea=
t:initial initial">yield</span><span style=3D"margin:0px;padding:0px;border=
:0px;font-size:14px;vertical-align:baseline;background-color:transparent;co=
lor:rgb(0,0,0);background-repeat:initial initial"> ge
 n</span>&lt;
span class=3D&quot;pun&quot; style=3D&quot;margin: 0px; padding: 0px; borde=
r: 0px; font-size: 14px; vertical-align: baseline; background-color: transp=
arent; color: rgb(0, 0, 0); background-position: initial initial; backgroun=
d-repeat: initial initial;&quot;&gt;.</span><span style=3D"margin:0px;paddi=
ng:0px;border:0px;font-size:14px;vertical-align:baseline;background-color:t=
ransparent;color:rgb(43,145,175);background-repeat:initial initial">Task</s=
pan><span style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertica=
l-align:baseline;background-color:transparent;color:rgb(0,0,0);background-r=
epeat:initial initial">(</span><span style=3D"margin:0px;padding:0px;border=
:0px;font-size:14px;vertical-align:baseline;background-color:transparent;co=
lor:rgb(0,0,0);background-repeat:initial initial">fetch</span><span style=
=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertical-align:baselin=
e;background-color:transparent;color:rgb(0,0,0);background-repeat:initial i=
nitial">)</span><span style=3D"margin:0px;padding:0px;border:0px;font-size:=
14px;vertical-align:baseline;background-color:transparent;color:rgb(0,0,0);=
background-repeat:initial initial">

</span><span style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vert=
ical-align:baseline;background-color:transparent;color:rgb(128,0,0);backgro=
und-repeat:initial initial">@gen</span><span style=3D"margin:0px;padding:0p=
x;border:0px;font-size:14px;vertical-align:baseline;background-color:transp=
arent;color:rgb(0,0,0);background-repeat:initial initial">.</span><span sty=
le=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertical-align:basel=
ine;background-color:transparent;color:rgb(0,0,0);background-repeat:initial=
 initial">engine
</span><span style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vert=
ical-align:baseline;background-color:transparent;color:rgb(0,0,139);backgro=
und-repeat:initial initial">def</span><span style=3D"margin:0px;padding:0px=
;border:0px;font-size:14px;vertical-align:baseline;background-color:transpa=
rent;color:rgb(0,0,0);background-repeat:initial initial"> fetch</span><span=
 style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertical-align:b=
aseline;background-color:transparent;color:rgb(0,0,0);background-repeat:ini=
tial initial">(</span><span style=3D"margin:0px;padding:0px;border:0px;font=
-size:14px;vertical-align:baseline;background-color:transparent;color:rgb(0=
,0,0);background-repeat:initial initial">callback<u></u><u></u>):</span><sp=
an style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertical-align=
:baseline;background-color:transparent;color:rgb(0,0,0);background-repeat:i=
nitial initial">
    response </span><span style=3D"margin:0px;padding:0px;border:0px;font-s=
ize:14px;vertical-align:baseline;background-color:transparent;color:rgb(0,0=
,0);background-repeat:initial initial">=3D</span><span style=3D"margin:0px;=
padding:0px;border:0px;font-size:14px;vertical-align:baseline;background-co=
lor:transparent;color:rgb(0,0,0);background-repeat:initial initial"> </span=
><span style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertical-a=
lign:baseline;background-color:transparent;color:rgb(0,0,139);background-re=
peat:initial initial">yield</span><span style=3D"margin:0px;padding:0px;bor=
der:0px;font-size:14px;vertical-align:baseline;background-color:transparent=
;color:rgb(0,0,0);background-repeat:initial initial">
  gen<u></u><span style=3D"margin:0px;padding:0px;border:0px;font-size:14px=
;vertical-align:baseline;background-color:transparent;color:rgb(0,0,0);back=
ground-repeat:initial initial">.</span><span style=3D"margin:0px;padding:0p=
x;border:0px;font-size:14px;vertical-align:baseline;background-color:transp=
arent;color:rgb(43,145,175);background-repeat:initial initial">Task</span><=
span style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertical-ali=
gn:baseline;background-color:transparent;color:rgb(0,0,0);background-repeat=
:initial initial">(</span><span style=3D"margin:0px;padding:0px;border:0px;=
font-size:14px;vertical-align:baseline;background-color:transparent;color:r=
gb(0,0,0);background-repeat:initial initial">client</span><u></u>.</span><s=
pan style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertical-alig=
n:baseline;background-color:transparent;color:rgb(0,0,0);background-repeat:=
initial initial">fetch</span><span style=3D"margin:0px;padding:0px;border:0=
px;font-size:14px;vertical-align:baseline;background-color:transparent;colo=
r:rgb(0,0,0);background-repeat:initial initial">,</span><span style=3D"marg=
in:0px;padding:0px;border:0px;font-size:14px;vertical-align:baseline;backgr=
ound-color:transparent;color:rgb(0,0,0);background-repeat:initial initial">=
 </span><span>&#39;<a href=3D"http://localhost:8888/" target=3D"_blank">htt=
p://localhost:8888/</a>&#39;</span><span style=3D"margin:0px;padding:0px;bo=
rder:0px;font-size:14px;vertical-align:baseline;background-color:transparen=
t;color:rgb(0,0,0);background-repeat:initial initial">)</span><span style=
=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertical-align:baselin=
e;background-color:transparent;color:rgb(0,0,0);background-repeat:initial i=
nitial">
    callback</span><span style=3D"margin:0px;padding:0px;border:0px;font-si=
ze:14px;vertical-align:baseline;background-color:transparent;color:rgb(0,0,=
0);background-repeat:initial initial">(</span><span style=3D"margin:0px;pad=
ding:0px;border:0px;font-size:14px;vertical-align:baseline;background-color=
:transparent;color:rgb(0,0,0);background-repeat:initial initial">response</=
span><span style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertic=
al-align:baseline;background-color:transparent;color:rgb(0,0,0);background-=
repeat:initial initial">)</span><span style=3D"margin:0px;padding:0px;borde=
r:0px;font-size:14px;vertical-align:baseline;background-color:transparent;c=
olor:rgb(0,0,0);background-repeat:initial initial">

io_loop</span><span style=3D"margin:0px;padding:0px;border:0px;font-size:14=
px;vertical-align:baseline;background-color:transparent;color:rgb(0,0,0);ba=
ckground-repeat:initial initial">.</span><span style=3D"margin:0px;padding:=
0px;border:0px;font-size:14px;vertical-align:baseline;background-color:tran=
sparent;color:rgb(0,0,0);background-repeat:initial initial">add_callback</s=
pan><span style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertica=
l-align:baseline;background-color:transparent;color:rgb(0,0,0);background-r=
epeat:initial initial">(</span><span style=3D"margin:0px;padding:0px;border=
:0px;font-size:14px;vertical-align:baseline;background-color:transparent;co=
lor:rgb(0,0,0);background-repeat:initial initial">g
 o_for_it&lt;
/span&gt;<span style=3D"margin:0px;padding:0px;border:0px;font-size:14px;ve=
rtical-align:baseline;background-color:transparent;color:rgb(0,0,0);backgro=
und-repeat:initial initial">)</span><span style=3D"margin:0px;padding:0px;b=
order:0px;font-size:14px;vertical-align:baseline;background-color:transpare=
nt;color:rgb(0,0,0);background-repeat:initial initial">
io_loop</span><span style=3D"margin:0px;padding:0px;border:0px;font-size:14=
px;vertical-align:baseline;background-color:transparent;color:rgb(0,0,0);ba=
ckground-repeat:initial initial">.</span><span style=3D"margin:0px;padding:=
0px;border:0px;font-size:14px;vertical-align:baseline;background-color:tran=
sparent;color:rgb(0,0,0);background-repeat:initial initial">start</span><sp=
an style=3D"margin:0px;padding:0px;border:0px;font-size:14px;vertical-align=
:baseline;background-color:transparent;color:rgb(0,0,0);background-repeat:i=
nitial initial">()</span></span></span></code></pre>

    <br>
    Leaks memory - looking deeper it appears that ExceptionStackContexts
    are not being cleaned up.=A0 Though I can&#39;t quite figure out why si=
nce
    everything appears to remove all the references.<span class=3D"HOEnZb">=
<font color=3D"#888888"><br>
    <br>
    --koblas<br>
  </font></span></div>

</blockquote></div><br></div></div></div>

--0016e6db2b03444b3b04ce9b9995--
Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
Account Options
