pymongo: LoopExit: ('This operation would block forever', ...)

[michal....@intel.com]

Hi,

I'm using pymongo with django 1.10 and I started to encounter serious problem after switching from multiple MongoClients (created on demand) to single global object following this gist: https://gist.github.com/josephmosby/4497f8a4f675170180ab

The first approach was bad, because connections handles were not freed properly leading to application crash (too many files open error).

The second approach however works fine for the first HTTP request, but during the next request the first call on mongo collection (find_one) leads to the error below.

This is the same MongoClient object.

As I undestood, pymongo uses gevent as socket backend. Is it possible that somethings corrupts its state?

I tested using django built-in server (manage.py runserver)

Traceback:

Traceback (most recent call last):

  File "/home/user/pycharm_env/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 42, in inner

    response = get_response(request)

  File "/home/user/pycharm_env/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response

    response = self._get_response(request)

  File "/home/user/pycharm_env/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response

    response = self.process_exception_by_middleware(e, request)

  File "/home/user/pycharm_env/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response

    response = wrapped_callback(request, *callback_args, **callback_kwargs)

    [secret code]

  File "/home/user/pycharm_env/local/lib/python2.7/site-packages/pymongo/collection.py", line 1224, in find_one

    for result in cursor.limit(-1):

  File "/home/user/pycharm_env/local/lib/python2.7/site-packages/pymongo/cursor.py", line 1176, in next

    if len(self.__data) or self._refresh():

  File "/home/user/pycharm_env/local/lib/python2.7/site-packages/pymongo/cursor.py", line 1087, in _refresh

    self.__send_message(q)

  File "/home/user/pycharm_env/local/lib/python2.7/site-packages/pymongo/cursor.py", line 924, in __send_message

    **kwargs)

  File "/home/user/pycharm_env/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 1026, in _send_message_with_response

    exhaust)

  File "/home/user/pycharm_env/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 1037, in _reset_on_error

    return func(*args, **kwargs)

  File "/home/user/pycharm_env/local/lib/python2.7/site-packages/pymongo/server.py", line 106, in send_message_with_response

    reply = sock_info.receive_message(request_id)

  File "/home/user/pycharm_env/local/lib/python2.7/site-packages/pymongo/pool.py", line 529, in receive_message

    self._raise_connection_failure(error)

  File "/home/user/pycharm_env/local/lib/python2.7/site-packages/pymongo/pool.py", line 649, in _raise_connection_failure

    raise error

LoopExit: ('This operation would block forever', <Hub at 0x7f42e4434a50 epoll pending=0 ref=0 fileno=39>)

Regards,

Michael

[Bernie Hackett]

This seems to be an issue in Gevent, happening with a number of libraries. The speculation is not monkey patching early enough:

https://github.com/gevent/gevent/issues/585