Supporting recursion between Python and Java

[Emily]

Hello,

I have a recursive method written on the Python side that, for each recursive call, makes a call back to the Java side. The idea is to check for values in a cache that exist on the Java side, and return that value to the Python side if it exists. Essentially a basic recursive formula such as:

def formula(self, n):

   if (n == 0):

      return n

    return formula(n-1)

However, I am finding that py4j is consistently unable to hold more than 140 calls on the stack before throwing the following errors:

On the Java side -- 

py4j.Py4JException: Error while sending a command.

at py4j.CallbackClient.sendCommand(CallbackClient.java:397)

at py4j.CallbackClient.sendCommand(CallbackClient.java:356)

at py4j.reflection.PythonProxyHandler.invoke(PythonProxyHandler.java:106)

at jdk.proxy2/jdk.proxy2.$Proxy17.executeFormula(Unknown Source)

at example.FormulaExecutorService.main(FormulaExecutorService.java:33)

Caused by: py4j.Py4JNetworkException: Error while sending a command: c

t

executeFormula

ro0

ro1

sRecursiveFormula

e

at py4j.ClientServerConnection.sendCommand(ClientServerConnection.java:255)

at py4j.CallbackClient.sendCommand(CallbackClient.java:384)

... 4 more

Caused by: java.io.IOException: Stream closed

at java.base/java.io.BufferedWriter.ensureOpen(BufferedWriter.java:107)

at java.base/java.io.BufferedWriter.write(BufferedWriter.java:224)

at java.base/java.io.Writer.write(Writer.java:249)

at py4j.commands.CallCommand.execute(CallCommand.java:83)

at py4j.ClientServerConnection.sendCommand(ClientServerConnection.java:246)

... 5 more

and on the Python side -- 

Thread-4 (run):There was an exception while executing the Python Proxy on the Python Side.
Traceback (most recent call last):
 ...
  File "C:\...\AppData\Local\anaconda3\Lib\site-packages\py4j\protocol.py", line 334, in get_return_value
    raise Py4JError(
py4j.protocol.Py4JError: An error occurred while calling o7.execute
Thread-4 (run):Exception while sending command.
Traceback (most recent call last):
  File "C:\...\AppData\Local\anaconda3\Lib\site-packages\py4j\clientserver.py", line 525, in send_command
    self.socket.sendall(return_message.encode("utf-8"))
    ^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'sendall'

My guess is that the reference to the web socket on the Python side is garbage collected in the middle of Python code execution. Is there anyway to make this work for more than 140 calls? I am starting up a ClientServer to facilitate communication between the Java and Python sides right now, and I am running py4j version 0.10.9.7

Any suggestions are greatly appreciated. Thanks in advance.