KeyError: 'popitem(): dictionary is empty'

[szuki]

Anyone had such an error or has any clues how could it happen?

Traceback (most recent call last):
File "/home/kronos/kronos3/kronos/server.py", line 80, in
serve_forever
self.__ioloop.start()
File "/usr/lib/pymodules/python2.5/tornado/ioloop.py", line 252, in
start
fd, events = self._events.popitem()


As the nearby lines are:

250 self._events.update(event_pairs)
251 while
self._events:
252 fd, events = self._events.popitem()

only thing that comes to my mind is if two threads would handle this
loop.
But only one is using it others are creating ioloop for there own
purpose.

[Ben Darnell]

Yes, that looks to me like a threading error. I believe when I ran
into this it was because I was creating an AsyncHTTPClient() on
another thread which defaulted to IOLoop.instance() instead of using
the right IOLoop for the thread. You could also be indirectly
operating on the IOLoop from the wrong thread through methods like
RequestHandler.render.

-Ben

[szuki]

Thanks, but the problem is more complicated than that.
It was an error with threads..., but we use haproxy to load balance
between servers and it was cutting down connections when it recived
something strange from server. Thread was finishing connection and
removing handler for master ioloop instance which lead to this error.
So i tried to trace down what was wrong with the answer of server. It
appeared that if server generates segfault when shutting it down the
problems appears with communication. I use strace to find last
function call and it is:

21863 epoll_wait(4, <unfinished ...>
21864 +++ killed by SIGSEGV +++

Every time. I'm stuck with that for the moment.
Any ideas from this moment here? Problem appears on Ubuntu and Debian
machines in python 2.6.6 and python 2.5.2.
We have made the problem reproduced in less than 10 minutes. At least
we can diagnose if it still appears.

[Ben Darnell]

The SIGSEGV is likely a symptom rather than a cause of the underlying threading problem.  How is another thread able to finish the connection?  That sounds like the real problem.  Normally in tornado when a connection is closed by the client or haproxy that event will get queued up in the IOLoop and handled in the usual thread.

-Ben

[szuki]

Ok I will tell a little more about architecture used in the server:
When connection comes to server handler creates new task and adds in
to Dispatcher and one Thread then executes. (post and get handler
methods are marked with decorator @tornado.web.asynchronous)

After that thread class method in handler which encodes the message
end writes it with write method. Then it calls finish method.
Now if i'm not wrong finish (as this is part of tornado code) Firstly
it flushes, then calls HTTPRequest.finish then HTTPConnection.finish.
I'm not sure what happens is stream is still writting, but it not it
closes stream. This is why I told thread finishes connection. Other
IOLoop instances are created when they need to ask other servers for
information. I didn't see any Exception in logs.

I'm not sure when haproxy ends it's connection. I don't see anything
usefull in haproxy logs(we turned them on yestarday). So i'm possibly
wrong about that haproxy was first to end connection.

SIGSEGV maybe is just a symptom, but how this error only appears on
computers when we get SIGSEGV when closing server. That's strange for
me. I can reproduce this error on one of team members machines, but on
my machine it all works all the time. I ask myself where is the
diffrence between this machines, but i didn't find any visible
diffrence both use the same python version with the same libraries in
the same versions.

The error in first post is somehow just a part of the bigger problem.

On 11 Sty, 21:17, Ben Darnell <b...@bendarnell.com> wrote:
> The SIGSEGV is likely a symptom rather than a cause of the underlying
> threading problem.  How is another thread able to finish the connection?
>  That sounds like the real problem.  Normally in tornado when a connection
> is closed by the client or haproxy that event will get queued up in the
> IOLoop and handled in the usual thread.
>
> -Ben
>

[Ben Darnell]

On Wed, Jan 12, 2011 at 3:57 AM, szuki <krzysztof....@gmail.com> wrote:

Ok I will tell a little more about architecture used in the server:
When connection comes to server handler creates new task and adds in
to Dispatcher and one Thread then executes. (post and get handler
methods are marked with decorator @tornado.web.asynchronous)

After that thread class method in handler which encodes the message
end writes it with write method. Then it calls finish method.

This is the problem.  It is not safe to call finish or flush from another thread (actually no RequestHandler method is *guaranteed* to be safe from other threads, but finish and flush are specifically problematic).  You must return to the IOLoop's thread with IOLoop.add_callback finishing the request.

-Ben

[szuki]

On 12 Sty, 20:47, Ben Darnell <b...@bendarnell.com> wrote:

> On Wed, Jan 12, 2011 at 3:57 AM, szuki <krzysztof.szukiel...@gmail.com>wrote:
>
> > Ok I will tell a little more about architecture used in the server:
> > When connection comes to server handler creates new task and adds in
> > to Dispatcher and one Thread then executes. (post and get handler
> > methods are marked with decorator @tornado.web.asynchronous)
>
> > After that thread class method in handler which encodes the message
> > end writes it with write method. Then it calls finish method.
>
> This is the problem.  It is not safe to call finish or flush from another
> thread (actually no RequestHandler method is *guaranteed* to be safe from
> other threads, but finish and flush are specifically problematic).  You must
> return to the IOLoop's thread with IOLoop.add_callback finishing the
> request.
>
> -Ben

Ok now the KeyError problem is fixed => thanks.
I still have the SIGSEGV from epool... any ideas here?