Unexpected timeous using haigha/gevent in uwsgi server

137 views
Skip to first unread message

vitaly

unread,
Aug 18, 2012, 2:53:16 AM8/18/12
to gev...@googlegroups.com
Python 2.6.7
gevent 1.0b2
haigha v0.4.1
uwsgi server v1.2.5

nginx routing to uwsgi server via uwsgi protocol.


During handling of an request, the python handler (invoked from uwsgi worker subprocess) makes use of the haigha AMQP client library with its GeventTransport to send message and wait for responses via RabbitMQ broker.  Each transaction entails connecting to RabbitMQ broker, sending a single request message via Haigha, waiting for a single response message, then tearing down the RabbitMQ connection. 

Usually one or two such transactions succeed, but then all subsequent requests time out while waiting for the response message.  The handler uses "with gevent.Timeout(seconds=timeoutSec):" to implement the timeout.

The exact same python code works perfectly when executed outside of uwsgi server, even when many such transactions are executed concurrently using gevent Greenlets.  I've done plenty of testing without any failures at all to be pretty certain that this haigha-based code works reliably *outside* the uwsgi environment.  The problems occur only when executing the exact same coe in the uwsgi environment.

When Haigha receives a message, a Haigha callback (running in another Greenlet) deposits the message in a gevent Queue.  The handler waits on this gevent Queue.  When gevent.Timeout() expires, the traceback looks like this:
  File "/Users/current/lib/python2.6/site-packages/gevent/queue.py", line 189, in get
    result = waiter.get()
  File "/Users/current/lib/python2.6/site-packages/gevent/hub.py", line 616, in get
    return self.hub.switch()
  File "/Users/current/lib/python2.6/site-packages/gevent/hub.py", line 373, in switch
    return greenlet.switch(self)
Timeout: 30 seconds

uwsgi server is started as follows:

/bin/uwsgi \

  -s 127.0.0.1:19001 \

  -d path-to-log-file.log \

  -M \

  --pidfile pid-file-path.pid \

  --vacuum \

  --idle 300 \

  -p 1 \

  --pythonpath api-server-home-path \

  --pythonpath /home/user/current/lib/python2.6/site-packages \

  --pythonpath /usr/lib/python2.6/site-packages \

  --chdir server-home-dir-path \

  --env MYAPP_CONFIG_FILE=$MYAPP_CONFIG_FILE \

  --module myapp.webapi


Please help.


Thank you,

Vitaly


Roberto De Ioris

unread,
Aug 18, 2012, 3:33:01 AM8/18/12
to gev...@googlegroups.com
I have never used haigha, but the only thing popping in my mind is not
taking in account uWSGI fork() usage. If you create a Queue in
myapp.webappi module (so in the master), it will not be usable by the
workers. Try adding --lazy, if it works i suggest you to upgrade to uWSGI
1.3 and use --lazy-apps (load apps like in lazy mode but maintains the
non-lazy behaviour for all of the other uWSGI parts)


--
Roberto De Ioris
http://unbit.it

vitaly

unread,
Aug 18, 2012, 4:18:34 AM8/18/12
to gev...@googlegroups.com, rob...@unbit.it


On Saturday, August 18, 2012 12:33:01 AM UTC-7, Roberto De Ioris wrote:

I have never used haigha, but the only thing popping in my mind is not
taking in account uWSGI fork() usage. If you create a Queue in
myapp.webappi module (so in the master), it will not be usable by the
workers. Try adding --lazy, if it works i suggest you to upgrade to uWSGI
1.3 and use --lazy-apps (load apps like in lazy mode but maintains the
non-lazy behaviour for all of the other uWSGI parts)


--
Roberto De Ioris
http://unbit.it

Hi Roberto,

thank you for the quick follow-up.  I forgot to mention that we already tried the --lazy option, but still had the same problem.  Based on your description on http://projects.unbit.it/uwsgi/wiki/ThingsToKnow, I expected that --lazy would solve this problem, but unfortunately it did not. It's a complete mystery -- the code runs fine outside of uwsgi, but fails very easily in the uwsgi environment.  Does uwsgi server patch any python built-in APIs? (socket, etc.)

Thank you,
Vitaly

Roberto De Ioris

unread,
Aug 18, 2012, 12:19:16 PM8/18/12
to gev...@googlegroups.com
Can you write a tiny test-script to allow me to reproduce the problem ?

Have you tried with uWSGI 1.3 (there are a bunch of optimizations
gevent-related, maybe it contains some fix i forgot to backport...) ?

vitaly

unread,
Aug 18, 2012, 9:37:23 PM8/18/12
to gev...@googlegroups.com, rob...@unbit.it
On Saturday, August 18, 2012 9:19:16 AM UTC-7, Roberto De Ioris wrote:
Can you write a tiny test-script to allow me to reproduce the problem ?

Have you tried with uWSGI 1.3 (there are a bunch of optimizations
gevent-related, maybe it contains some fix i forgot to backport...) ?

--
Roberto De Ioris
http://unbit.it

Hi Roberto, I haven't tried uWSGI 1.3.  I could a small program to reproduce this, but it would involve calls into the Haigha library.  Is that okay?  I will take a fresh look at this when I return from vacation a week from now.

Best,
Vitaly 

vitaly

unread,
Aug 29, 2012, 5:50:17 PM8/29/12
to gev...@googlegroups.com, rob...@unbit.it
I did more debugging yesterday and finally nailed it down -- it's nothing to do with uWSGI; it was an internal error.  Sorry for the trouble and thank you for all the help.
Reply all
Reply to author
Forward
0 new messages