Stopping and restarting the JVM in same interpreter instance

[Ron Budnar]

Hi all,

I'm adapting some code that uses the python-weka-wrapper3 into a linux based docker image. The code will be invoked via a queue system that appears to use the same python interpreter instance for each invokation of our code. We essentially call jvm.start(), run our code, then call jvm.stop() and do some cleanup. I quickly noticed that our code works as expected the first time; however, when the code is invoked for a second time, I receive the following error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/javabridge/jutil.py", line 286, in start_thread
    env = vm.create(args)
  File "_javabridge.pyx", line 654, in _javabridge.JB_VM.create
RuntimeError: Failed to create Java VM. Return code = -1
ERROR:javabridge.jutil:Failed to create Java VM
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/weka/core/jvm.py", line 127, in start
    javabridge.start_vm(args=args, run_headless=True, max_heap_size=max_heap_size)
  File "/usr/local/lib/python3.8/site-packages/javabridge/jutil.py", line 319, in start_vm
    raise RuntimeError("Failed to start Java VM")
RuntimeError: Failed to start Java VM

I tried a minimal repro from the command line (python 3.8) and had the same results as follows:

1. Invoke python from shell:

# python
Python 3.8.3 (default, Jun  9 2020, 17:49:41)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

2. Import the jvm:

>>> import weka.core.jvm as jvm

3. Start the jvm:

>>> jvm.start(max_heap_size="512m")
DEBUG:weka.core.jvm:Adding bundled jars
DEBUG:weka.core.jvm:Classpath=['/usr/local/lib/python3.8/site-packages/javabridge/jars/rhino-1.7R4.jar', '/usr/local/lib/python3.8/site-packages/javabridge/jars/runnablequeue.jar', '/usr/local/lib/python3.8/site-packages/javabridge/jars/cpython.jar', '/usr/local/lib/python3.8/site-packages/weka/lib/weka.jar', '/usr/local/lib/python3.8/site-packages/weka/lib/python-weka-wrapper.jar']
DEBUG:weka.core.jvm:MaxHeapSize=512m
DEBUG:weka.core.jvm:Package support disabled
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by weka.core.WekaPackageClassLoaderManager (file:/usr/local/lib/python3.8/site-packages/weka/lib/weka.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of weka.core.WekaPackageClassLoaderManager
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

4. Stop the jvm:

>>> jvm.stop()

5. Attempt to restart the jvm:

>>> jvm.start(max_heap_size="512m")

DEBUG:weka.core.jvm:Adding bundled jars
DEBUG:weka.core.jvm:Classpath=['/usr/local/lib/python3.8/site-packages/javabridge/jars/rhino-1.7R4.jar', '/usr/local/lib/python3.8/site-packages/javabridge/jars/runnablequeue.jar', '/usr/local/lib/python3.8/site-packages/javabridge/jars/cpython.jar', '/usr/local/lib/python3.8/site-packages/weka/lib/weka.jar', '/usr/local/lib/python3.8/site-packages/weka/lib/python-weka-wrapper.jar', '/usr/local/lib/python3.8/site-packages/weka/lib/weka.jar', '/usr/local/lib/python3.8/site-packages/weka/lib/python-weka-wrapper.jar']
DEBUG:weka.core.jvm:MaxHeapSize=512m
DEBUG:weka.core.jvm:Package support disabled
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/javabridge/jutil.py", line 286, in start_thread
    env = vm.create(args)
  File "_javabridge.pyx", line 654, in _javabridge.JB_VM.create
RuntimeError: Failed to create Java VM. Return code = -1
ERROR:javabridge.jutil:Failed to create Java VM
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/weka/core/jvm.py", line 127, in start
    javabridge.start_vm(args=args, run_headless=True, max_heap_size=max_heap_size)
  File "/usr/local/lib/python3.8/site-packages/javabridge/jutil.py", line 319, in start_vm
    raise RuntimeError("Failed to start Java VM")
RuntimeError: Failed to start Java VM

In our case, explicitly not stopping the JVM seems to resolve the problem; however, I'm not familiar enough with JVM usage to determine if this is a sound approach that won't lead to memory leakages, etc. Any thoughts are appreciated!

Thanks! 

[Peter Reutemann]

The problem of not being able to restart the JVM within a Python session is unfortunately a limitation of the javabridge library that pww uses.

Java uses garbage collection to remove unused objects. Unless you do something stupid like keeping references (eg in lists or hashtables) to all the objects that pass through, you shouldn't have problems with memory leaks.

Cheers, Peter

--
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 858-5174
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/

[Ron Budnar]

Excellent, thank you for the response!

Thanks,

Ron