Managed to replicate the stopping issue

[Dax]

Hi list,

I have managed to make hookbox stop when the number of connections
reach around 800+.
These are just connect and subscribe request to one channel named
'testing'. Two computers were used, one serving, the other running
multiple tabs/browsers.

I get the following:

Traceback (most recent call last):
File "/home/dax/envs/pyramidenv/lib/python2.6/site-packages/
eventlet-0.9.13-py2.6.egg/eventlet/hubs/poll.py", line 92, in wait
readers.get(fileno, noop).cb(fileno)
File "/home/dax/envs/pyramidenv/lib/python2.6/site-packages/
eventlet-0.9.13-py2.6.egg/eventlet/greenthread.py", line 192, in main
result = function(*args, **kwargs)
File "/home/dax/envs/pyramidenv/lib/python2.6/site-packages/
eventlet-0.9.13-py2.6.egg/eventlet/wsgi.py", line 589, in server
client_socket = sock.accept()
File "/home/dax/envs/pyramidenv/lib/python2.6/site-packages/
eventlet-0.9.13-py2.6.egg/eventlet/greenio.py", line 148, in accept
res = socket_accept(fd)
File "/home/dax/envs/pyramidenv/lib/python2.6/site-packages/
eventlet-0.9.13-py2.6.egg/eventlet/greenio.py", line 46, in
socket_accept
return descriptor.accept()
File "/usr/lib/python2.6/socket.py", line 195, in accept
sock, addr = self._sock.accept()
error: [Errno 24] Too many open files
Removing descriptor: 4


Another error I see is:

UnboundLocalError("local variable 'sck' referenced before
assignment",)
2011-02-24 09:45:02,333 - hookbox - WARNING - Exception with webhook
http://192.168.5.40:6543
Traceback (most recent call last):
File "/home/dax/codes/hookbox/hookbox/server.py", line 211, in
http_request
File "/home/dax/codes/restkit/restkit/resource.py", line 189, in
request
File "/home/dax/codes/restkit/restkit/client.py", line 437, in
request
File "/home/dax/codes/restkit/restkit/client.py", line 323, in
perform
File "/home/dax/codes/restkit/restkit/client.py", line 217, in
get_connection
File "/home/dax/codes/restkit/restkit/client.py", line 201, in
connect
UnboundLocalError: local variable 'sck' referenced before assignment


After getting the error I am also not able to connect to the admin
interface

Traceback (most recent call last):
File "/home/dax/envs/pyramidenv/lib/python2.6/site-packages/
eventlet-0.9.13-py2.6.egg/eventlet/hubs/hub.py", line 329, in
fire_timers
timer()
File "/home/dax/envs/pyramidenv/lib/python2.6/site-packages/
eventlet-0.9.13-py2.6.egg/eventlet/hubs/timer.py", line 56, in
__call__
cb(*args, **kw)
File "/home/dax/envs/pyramidenv/lib/python2.6/site-packages/
eventlet-0.9.13-py2.6.egg/eventlet/event.py", line 167, in _do_send
waiter.throw(*exc)
File "/home/dax/envs/pyramidenv/lib/python2.6/site-packages/
eventlet-0.9.13-py2.6.egg/eventlet/greenthread.py", line 192, in main
result = function(*args, **kwargs)
File "/home/dax/codes/hookbox/hookbox/admin/admin.py", line 416, in
_run
self._cleanup()
File "/home/dax/codes/hookbox/hookbox/admin/admin.py", line 421, in
_cleanup
self._admin_app.logout(self)
File "/home/dax/codes/hookbox/hookbox/admin/admin.py", line 80, in
logout
self.conns.remove(conn)
ValueError: list.remove(x): x not in list


I will try simulating with messages being sent and see if other errors
similar to what was indicated in
http://groups.google.com/group/hookbox/browse_thread/thread/aac3377bb1599dd7

Regards,
DaxHi list,

I have managed to make hookbox stop when the number of connections
reach around 800+.
These are just connect and subscribe request to one channel named
'testing'. Two computers were used, one serving, the other running
multiple tabs/browsers.

I get the following:

Traceback (most recent call last):
File "/home/dax/envs/pyramidenv/lib/python2.6/site-packages/
eventlet-0.9.13-py2.6.egg/eventlet/hubs/poll.py", line 92, in wait
readers.get(fileno, noop).cb(fileno)
File "/home/dax/envs/pyramidenv/lib/python2.6/site-packages/
eventlet-0.9.13-py2.6.egg/eventlet/greenthread.py", line 192, in main
result = function(*args, **kwargs)
File "/home/dax/envs/pyramidenv/lib/python2.6/site-packages/
eventlet-0.9.13-py2.6.egg/eventlet/wsgi.py", line 589, in server
client_socket = sock.accept()
File "/home/dax/envs/pyramidenv/lib/python2.6/site-packages/
eventlet-0.9.13-py2.6.egg/eventlet/greenio.py", line 148, in accept
res = socket_accept(fd)
File "/home/dax/envs/pyramidenv/lib/python2.6/site-packages/
eventlet-0.9.13-py2.6.egg/eventlet/greenio.py", line 46, in
socket_accept
return descriptor.accept()
File "/usr/lib/python2.6/socket.py", line 195, in accept
sock, addr = self._sock.accept()
error: [Errno 24] Too many open files
Removing descriptor: 4


Another error I see is:

UnboundLocalError("local variable 'sck' referenced before
assignment",)
2011-02-24 09:45:02,333 - hookbox - WARNING - Exception with webhook
http://192.168.5.40:6543
Traceback (most recent call last):
File "/home/dax/codes/hookbox/hookbox/server.py", line 211, in
http_request
File "/home/dax/codes/restkit/restkit/resource.py", line 189, in
request
File "/home/dax/codes/restkit/restkit/client.py", line 437, in
request
File "/home/dax/codes/restkit/restkit/client.py", line 323, in
perform
File "/home/dax/codes/restkit/restkit/client.py", line 217, in
get_connection
File "/home/dax/codes/restkit/restkit/client.py", line 201, in
connect
UnboundLocalError: local variable 'sck' referenced before assignment


After getting the error I am also not able to connect to the admin
interface

Traceback (most recent call last):
File "/home/dax/envs/pyramidenv/lib/python2.6/site-packages/
eventlet-0.9.13-py2.6.egg/eventlet/hubs/hub.py", line 329, in
fire_timers
timer()
File "/home/dax/envs/pyramidenv/lib/python2.6/site-packages/
eventlet-0.9.13-py2.6.egg/eventlet/hubs/timer.py", line 56, in
__call__
cb(*args, **kw)
File "/home/dax/envs/pyramidenv/lib/python2.6/site-packages/
eventlet-0.9.13-py2.6.egg/eventlet/event.py", line 167, in _do_send
waiter.throw(*exc)
File "/home/dax/envs/pyramidenv/lib/python2.6/site-packages/
eventlet-0.9.13-py2.6.egg/eventlet/greenthread.py", line 192, in main
result = function(*args, **kwargs)
File "/home/dax/codes/hookbox/hookbox/admin/admin.py", line 416, in
_run
self._cleanup()
File "/home/dax/codes/hookbox/hookbox/admin/admin.py", line 421, in
_cleanup
self._admin_app.logout(self)
File "/home/dax/codes/hookbox/hookbox/admin/admin.py", line 80, in
logout
self.conns.remove(conn)
ValueError: list.remove(x): x not in list


I will try simulating with messages being sent and see if other errors
similar to what was indicated in
http://groups.google.com/group/hookbox/browse_thread/thread/aac3377bb1599dd7

Regards,
Dax

[Michael Carter]

Some notes.

1. change the ulimit (that's the error below, look at this site for more info: http://nichol.as/benchmark-of-python-web-servers)

2. eventlet's wsgi.server defaults to 1024 max simultaneous greenlets. You can hack the call to eventlet.spawn(wsgi.server... ) in csp_eventlet/__init__.py to take max_size=10240 (instead of 1024)

3. Python's Base HTTP implementation doesn't give a way to catch errno 110 errors. Each one kills an eventlet greenlet from the pool. This needs to be monkey patched at runtime or a different wsgi server should be used

Maybe others, I'm not sure.

[Dan Gould]

On my local system, I've increased the ulimit and drastically
increased the number of greenlets.

Is anyone running Hookbox within uWSGI/gunicorn/nginx who can report
on their experience?

On Feb 25, 7:41 pm, Michael Carter <cartermich...@gmail.com> wrote:
> Some notes.
>
> 1. change the ulimit (that's the error below, look at this site for more
> info:http://nichol.as/benchmark-of-python-web-servers)
> 2. eventlet's wsgi.server defaults to 1024 max simultaneous greenlets. You
> can hack the call to eventlet.spawn(wsgi.server... ) in
> csp_eventlet/__init__.py to take max_size=10240 (instead of 1024)
> 3. Python's Base HTTP implementation doesn't give a way to catch errno 110
> errors. Each one kills an eventlet greenlet from the pool. This needs to be
> monkey patched at runtime or a different wsgi server should be used
>
> Maybe others, I'm not sure.
>

> >http://groups.google.com/group/hookbox/browse_thread/thread/aac3377bb...

> >http://groups.google.com/group/hookbox/browse_thread/thread/aac3377bb...
>
> > Regards,
> > Dax