python thrift client to hbase: AssertionError: This event is already used by another greenlet

[张会]

I have a web server written by python, in the server backend, it will use python thrift client to connect a hbase cluster. in several times(not all), I got the following error stacktrace:

  File "/usr/local/lib/python2.7/dist-packages/thrift/protocol/TBinaryProtocol.py", line 126, in readMessageBegin

    sz = self.readI32()

  File "/usr/local/lib/python2.7/dist-packages/thrift/protocol/TBinaryProtocol.py", line 206, in readI32

    buff = self.trans.readAll(4)

  File "/usr/local/lib/python2.7/dist-packages/thrift/transport/TTransport.py", line 58, in readAll

    chunk = self.read(sz - have)

  File "/usr/local/lib/python2.7/dist-packages/thrift/transport/TTransport.py", line 271, in read

    self.readFrame()

  File "/usr/local/lib/python2.7/dist-packages/thrift/transport/TTransport.py", line 275, in readFrame

    buff = self.__trans.readAll(4)

  File "/usr/local/lib/python2.7/dist-packages/thrift/transport/TTransport.py", line 58, in readAll

    chunk = self.read(sz - have)

  File "/usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.py", line 103, in read

    buff = self.handle.recv(sz)

  File "/usr/local/lib/python2.7/dist-packages/gevent/socket.py", line 432, in recv

    wait_read(sock.fileno(), timeout=self.timeout, event=self._read_event)

  File "/usr/local/lib/python2.7/dist-packages/gevent/socket.py", line 165, in wait_read

    assert event.arg is None, 'This event is already used by another greenlet: %r' % (event.arg, )

AssertionError: This event is already used by another greenlet: (<Greenlet at 0x35052f8: <bound method WorkerThread.__bootstrap of <WorkerThread(Thread-2, started daemon 55595768)>>>, timeout('timed out',))

can anyone please point me how to fix this kind of issue.

my environment:

Flask: 0.9

gunicorn: 0.17.4

thrift: 0.9.0

Thanks

[Jason Madden]

That error message means that the identically same socket object is being used concurrently by two different greenlets. (Note that every native thread has its own set of greenlets, so sharing a socket across native threads causes the same problem.) Sockets are not concurrency safe, so the actual fix is to ensure that sockets are not shared across greenlets.

The specifics of how to fix that depend on each environment; sometimes it's a simple programming error. But one somewhat common cause of this is monkey-patching too late in the process lifecycle. For example, initializing a socket pool that uses locks to prevent concurrent use of the same socket, and *then* monkey patching and spawning multiple greenlets. Because the pool will have been created with non-greenlet aware locks, two different greenlets on the same thread could still use the same socket object. In gunicorn's case when using the gevent workers, this is easy to accomplish if the application is being preloaded: preloading happens in the master (arbiter) process, but monkey patching happens only after each worker is spawned.