py4j-gevent breaks in Python 3

[Rasmus Fogh]

Cross-posting with py4j mailing list - as the problem is between py4j and gevent

Hi,

Code mixing py4j and gevent that worked fine in Python 2 seems to be broken in Python 3.

I have used py4j to handle communication between a scientific instrument control application (Python, using gevent) and an external java process. A cut-down test is given below. The external process is too complex to cut down (sorry); its first action is to call the processMessage function on the Python side with a java object message parameter.

When using the make_gateway function (Python 2.7.13, OpenSUSE, Miniconda environment. Versions: gevent: 1.2.1, greenlet: 0.4.12, py4j: 0.10.4)
the test works and gets to the end of processMessage. The code relies on reloading socket, starting the py4j connection, and then repatching socket.

Without the reload, repatch steps (make_gateway2) in Python 2, or with either version in Python 3 (Python 3.6.13, gevent 1.3.7, greenlet 1.0.0, py4j 0.10.9.2)
the line
txt = str(py4j_message)
raises an error, apparently because gevent is asked to execute a command in a different thread. More detailed error messages attached

Does anyone have any suggestions?

Thanks in advance,

Rasmus Fogh

Global phasing Ltd., Cambridge

[Rasmus Fogh]

The problem is now solved, with the help of colleagues.

The problem is that the program using gevent requires a monkey-patched copy of socket, whereas py4j requires an unpatched copy, i.e. two copies present at the same time.

In python 2 this could be achieved by reloading socket before setting up the py4j gateway, and repatching it afterwards. In Python 3 there is only ever one copy of a loaded module (much better). The same result could then be obtained (see https://stackoverflow.com/questions/11170949/how-to-make-a-copy-of-a-python-module-at-runtime) by something like

import socket as original_socket
del sys.modules["socket"]
del sys.modules["_socket"]
import socket
gevent.monkey.patch_all(thread=False)

followed by

py4j.java_gateway.socket = original_socket
py4j.clientserver.socket = original_socket

Which inserts the unpatched version of socket into py4j.

This is still somewhat hacky, since it depends on the (unguaranteed) internals of py4j, but it works and is an improvement over the Python 2 version.