Django 1.6.1 not cooperating too well with Gevent

[André Cruz]

Hello.

I have been using Django 1.5.2 + uWSGI + Gevent with no problems for quite some time. I monkey patch my application as soon as I can and the greenlets cooperate with no problems since all the work is network-bound.

I have recently tried to switch to Django 1.6.1, but I found that my greenlets started blocking rather frequently. I produced a stacktrace of all the greenlets of a uWSGI worker and realised most of them were blocked around the same lock:

1513 were blocked on:

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/gevent/greenlet.py", line 327, in run

  result = self._run(*self.args, **self.kwargs)

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 206, in __call__

  response = self.get_response(request)

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 90, in get_response

  response = middleware_method(request)

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/contrib/sessions/middleware.py", line 12, in process_request

  request.session = engine.SessionStore(session_key)

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/contrib/sessions/backends/cache.py", line 14, in __init__

  self._cache = get_cache(settings.SESSION_CACHE_ALIAS)

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/core/cache/__init__.py", line 135, in get_cache

  signals.request_finished.connect(cache.close)

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 116, in connect

  with self.lock:

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/gevent/hub.py", line 331, in switch

  return greenlet.switch(self)

14 were blocked on:

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/gevent/greenlet.py", line 327, in run

  result = self._run(*self.args, **self.kwargs)

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/http/response.py", line 308, in close

  signals.request_finished.send(sender=self._handler_class)

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 184, in send

  for receiver in self._live_receivers(sender):

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 242, in _live_receivers

  with self.lock:

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/gevent/hub.py", line 331, in switch

  return greenlet.switch(self)

One was blocked on:

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/gevent/greenlet.py", line 327, in run#

  result = self._run(*self.args, **self.kwargs)#

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/http/response.py", line 308, in close#

  signals.request_finished.send(sender=self._handler_class)#

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 184, in send#

  for receiver in self._live_receivers(sender):#

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 245, in _live_receivers#

  for (receiverkey, r_senderkey), receiver in self.receivers:#

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/dispatch/saferef.py", line 121, in remove#

  function( self )#

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 270, in _remove_receiver#

  with self.lock:#

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/gevent/hub.py", line 331, in switch#

  return greenlet.switch(self)

The lock is a Gevent lock so other Greenlets are free to run. Did something change from Django 1.5 to Django 1.6 that would cause this?

It seems to be related to sessions, and I use a memcached cache backend for session storage with Pickle serialization, did the memcached driver change?

Best regards and thanks for the help,

André Cruz

[André Cruz]

On Tuesday, February 4, 2014 11:15:26 AM UTC, André Cruz wrote:

One was blocked on:

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/gevent/greenlet.py", line 327, in run#

  result = self._run(*self.args, **self.kwargs)#

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/http/response.py", line 308, in close#

  signals.request_finished.send(sender=self._handler_class)#

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 184, in send#

  for receiver in self._live_receivers(sender):#

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 245, in _live_receivers#

  for (receiverkey, r_senderkey), receiver in self.receivers:#

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/dispatch/saferef.py", line 121, in remove#

  function( self )#

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 270, in _remove_receiver#

  with self.lock:#

File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/gevent/hub.py", line 331, in switch#

  return greenlet.switch(self)

Now that I look at it this last one seems to be the culprit of the deadlock. I'll file a bug report.

André