JEP crashing with IBM JDK 1.8

[GianMaria Romanato]

Hello,

I need some advice for running Jep on the IBM JDK.
I am trying to use Jep from a Java enterprise application deployed on Linux and IBM WebSphere Application Server.

In the past I successfully deployed Jep Linux / IBM WebSphere Liberty with the Oracle JDK.
This time it's the traditional version of WebSphere Application Server which IBM supports only if run using the IBM JDK.

java version "1.8.0"

Java(TM) SE Runtime Environment (build pxa6480sr4fp10-20170727_01(SR4 FP10))

IBM J9 VM (build 2.8, JRE 1.8.0 Linux amd64-64 Compressed References 20170722_357405 (JIT enabled, AOT enabled)

J9VM - R28_20170722_0201_B357405

JIT  - tr.r14.java_20170722_357405

GC   - R28_20170722_0201_B357405_CMPRSS

J9CL - 20170722_357405)

JCL - 20170726_01 based on Oracle jdk8u144-b01

My problem is that when the Java application tries to use JEP the IBM JDK crashes with a GPF due to a segmentation fault.
This was verified both on Ubuntu and on CentOS, with both Jep 3.5.2 (the version I was using) and latest 3.6.4.

It's easy to reproduce because the jep shell reproduces the same crash. Steps to reproduce:

1. Download the IBM JDK 1.8 for Linux here:
   http://www14.software.ibm.com/cgi-bin/weblap/lap.pl?popup=Y&la_formnum=&li_formnum=L-PMAA-A3Z8P2&title=IBM%20SDK,%20Java%20Technology%20Edition,%20Version%208&accepted_url=http://public.dhe.ibm.com/ibmdl/export/pub/systems/cloud/runtimes/java/8.0.4.10/linux/x86_64/ibm-java-sdk-8.0-4.10-x86_64-archive.bin
2. clone jep from github and build it:
1. export JAVA_HOME=<IBM_JDK_INSTALLATION_DIR>
2. export PATH=<IBM_JDK_BIN_FOLDER>:$PATH ;
   
3. python3 setup.py build install
3. With the variables still exported run jep from the shell to reproduce immediately the dump:

[root@wasdev jep]# jep
Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00007FA2B9DBC8F0 Handler2=00007FA2B96F4940 InaccessibleAddress=0000000000000000
RDI=00007FA2B4007550 RSI=00007FA2B4004AF8 RAX=00007FA2BC5AA030 RBX=0000000000000001
RCX=00007FA2BC5AA030 RDX=0000000000000001 R8=0000000000000000 R9=0000000000004D1F
R10=0000000059842230 R11=00007FA2BB71DD90 R12=0000000000000000 R13=00000000FFFFFFFF
R14=00007FA2B4004AF8 R15=0000000000000000
RIP=00007FA2B9DC931B GS=0000 FS=0000 RSP=00007FA2BAC7D050
EFlags=0000000000210246 CS=0033 RBP=00007FA2BAC7D0C8 ERR=0000000000000006
TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000000
xmm0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm1 7265626d656d5f5f (f: 1701666688.000000, d: 1.140736e+243)
xmm2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm5 00007fa2a8f5e648 (f: 2834687488.000000, d: 6.933549e-310)
xmm6 00007fa2a8c7c048 (f: 2831663104.000000, d: 6.933549e-310)
xmm7 00007fa2bc3c3bc0 (f: 3158064128.000000, d: 6.933565e-310)
xmm8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm9 bfca2dbbbbe7ea80 (f: 3152538112.000000, d: -2.045207e-01)
xmm10 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
xmm11 3ede000000000000 (f: 0.000000, d: 7.152557e-06)
xmm12 bcc6000000000000 (f: 0.000000, d: -6.106227e-16)
xmm13 3c5d000000000000 (f: 0.000000, d: 6.288373e-18)
xmm14 3c493711b07a998c (f: 2960824832.000000, d: 2.733828e-18)
xmm15 402bb9d3beb8c600 (f: 3199780352.000000, d: 1.386294e+01)
Module=/opt/ibm/WebSphere85/AppServer/java_1.8_64/jre/lib/amd64/compressedrefs/libj9vm28.so
Module_base_address=00007FA2B9D41000 Symbol=J9_GetCreatedJavaVMs
Symbol_address=00007FA2B9DC9260
Target=2_80_20161013_322271 (Linux 3.10.0-514.2.2.el7.x86_64)
CPU=amd64 (2 logical CPUs) (0xe6eb5000 RAM)
----------- Stack Backtrace -----------
J9_GetCreatedJavaVMs+0xbb (0x00007FA2B9DC931B [libj9vm28.so+0x8831b])
JNI_GetCreatedJavaVMs+0x58 (0x00007FA2BA2544A8 [libjvm.so+0xe4a8])
pyembed_get_env+0x25 (0x00007FA2A8CB72E5 [libjep.so+0x192e5])
(0x00007FA2A8CA9E9B [libjep.so+0xbe9b])
PyObject_Call+0x8c (0x00007FA2BBFF6DCC [libpython3.4m.so+0x62dcc])
PyEval_EvalFrameEx+0x3ce2 (0x00007FA2BC0A9F12 [libpython3.4m.so+0x115f12])
PyEval_EvalCodeEx+0x88e (0x00007FA2BC0AF54E [libpython3.4m.so+0x11b54e])
PyEval_EvalFrameEx+0x6df2 (0x00007FA2BC0AD022 [libpython3.4m.so+0x119022])
PyEval_EvalFrameEx+0x8220 (0x00007FA2BC0AE450 [libpython3.4m.so+0x11a450])
PyEval_EvalCodeEx+0x88e (0x00007FA2BC0AF54E [libpython3.4m.so+0x11b54e])
PyEval_EvalFrameEx+0x6df2 (0x00007FA2BC0AD022 [libpython3.4m.so+0x119022])
PyEval_EvalFrameEx+0x8220 (0x00007FA2BC0AE450 [libpython3.4m.so+0x11a450])
PyEval_EvalCodeEx+0x88e (0x00007FA2BC0AF54E [libpython3.4m.so+0x11b54e])
(0x00007FA2BC01F5AE [libpython3.4m.so+0x8b5ae])
PyObject_Call+0x8c (0x00007FA2BBFF6DCC [libpython3.4m.so+0x62dcc])
_PyObject_CallMethodIdObjArgs+0xd0 (0x00007FA2BBFF78B0 [libpython3.4m.so+0x638b0])
PyImport_ImportModuleLevelObject+0x7e3 (0x00007FA2BC0C2373 [libpython3.4m.so+0x12e373])
(0x00007FA2BC0A34DF [libpython3.4m.so+0x10f4df])
PyEval_EvalFrameEx+0x8992 (0x00007FA2BC0AEBC2 [libpython3.4m.so+0x11abc2])
PyEval_EvalCodeEx+0x88e (0x00007FA2BC0AF54E [libpython3.4m.so+0x11b54e])
PyEval_EvalFrameEx+0x6df2 (0x00007FA2BC0AD022 [libpython3.4m.so+0x119022])
PyEval_EvalCodeEx+0x88e (0x00007FA2BC0AF54E [libpython3.4m.so+0x11b54e])
(0x00007FA2BC01F5AE [libpython3.4m.so+0x8b5ae])
PyObject_Call+0x8c (0x00007FA2BBFF6DCC [libpython3.4m.so+0x62dcc])
_PyObject_CallMethodIdObjArgs+0xd0 (0x00007FA2BBFF78B0 [libpython3.4m.so+0x638b0])
PyImport_ImportModuleLevelObject+0x5fd (0x00007FA2BC0C218D [libpython3.4m.so+0x12e18d])
(0x00007FA2BC0A34DF [libpython3.4m.so+0x10f4df])
PyObject_Call+0x8c (0x00007FA2BBFF6DCC [libpython3.4m.so+0x62dcc])
PyEval_CallObjectWithKeywords+0x47 (0x00007FA2BC0A5F07 [libpython3.4m.so+0x111f07])
PyEval_EvalFrameEx+0x1420 (0x00007FA2BC0A7650 [libpython3.4m.so+0x113650])
PyEval_EvalCodeEx+0x88e (0x00007FA2BC0AF54E [libpython3.4m.so+0x11b54e])
PyEval_EvalCode+0x3b (0x00007FA2BC0AF61B [libpython3.4m.so+0x11b61b])
(0x00007FA2BC0CB6C4 [libpython3.4m.so+0x1376c4])
PyRun_StringFlags+0x75 (0x00007FA2BC0CD695 [libpython3.4m.so+0x139695])
pyembed_eval+0x5f (0x00007FA2A8CB7F4F [libjep.so+0x19f4f])
Java_jep_Jep_eval+0x2c (0x00007FA2A8CB658C [libjep.so+0x1858c])
(0x00007FA28AA05C5C [<unknown>+0x0])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2017/08/04 09:28:48 - please wait.
JVMDUMP032I JVM requested System dump using '/root/jep/core.20170804.092848.19742.0001.dmp' in response to an event
JVMDUMP010I System dump written to /root/jep/core.20170804.092848.19742.0001.dmp
JVMDUMP032I JVM requested Java dump using '/root/jep/javacore.20170804.092848.19742.0002.txt' in response to an event

Thanks.
GianMaria.

[Ben Steffensmeier]

GianMaria

It looks like the IBM JDK will not accept NULL for the third argument to JNI_GetCreatedVMs which we are using to access the java environment from a python call. It works for me if I just pass in a valid pointer for the third arg, like this:

JNIEnv* pyembed_get_env(void)
{
    JavaVM *jvm;
    JNIEnv *env;
    jsize nVMs;

    JNI_GetCreatedJavaVMs(&jvm, 1, &nVMs);
    /*
     * If the thread is already part of the JVM, the daemon status is not
     * changed. If this is a new thread, started by Python then this tells
     * Java to allow the process to exit even if the thread is still attached.
     * Since there are no hooks to detach the thread later daemon is the only
     * way to let the process exit normally.
     */
    (*jvm)->AttachCurrentThreadAsDaemon(jvm, (void**) &env, NULL);

    return env;
}

I'm a little surprised no one has tried an IBM JDK before, or perhaps it worked on earlier versions. With that change it is passing all our unit tests on my ubuntu machine. It would still be good if you could try that out and see if it makes it through your real world usage also. I will put that change in, I'll reply when we figure out which release it can make it in.

We are in the process of setting up travis-ci to automate testing for a variety of environments. After looking at this I would like to add tests using the IBM JDK and I was dissappointed that they aren't supporting them. The good news is that it looks like they may be getting support soon which would help ensure that we stay compatible with the IBM JDK for all future changes.

Ben

[GianMaria Romanato]

Hi Ben,

thanks for the prompt reply. This is not the first time myself or a colleague of mine ask for help here, and we have always been supported very quickly.
You and Nathan really deserve a big praise: your technical support for Jep is top notch, by a large margin better than most commercial products I used in my life!

On Friday, August 4, 2017 at 11:54:24 AM UTC+2, Ben Steffensmeier wrote:

GianMaria

It looks like the IBM JDK will not accept NULL for the third argument to JNI_GetCreatedVMs which we are using to access the java environment from a python call. It works for me if I just pass in a valid pointer for the third arg, like this:

I confirm that the blocking crash is gone with that trivial change. Before the end of the day I'll rebuild our reference application with a patched 3.5.2, run the test suite and post results here.
 

We are in the process of setting up travis-ci to automate testing for a variety of environments. After looking at this I would like to add tests using the IBM JDK and I was dissappointed that they aren't supporting them. The good news is that it looks like they may be getting support soon which would help ensure that we stay compatible with the IBM JDK for all future changes.

That sounds great.

Thanks again.
GianMaria.

[Ben Steffensmeier]

That change is now in the 3.7 branch which will be released will be next week.

Ben

[GianMaria Romanato]

Hi Ben,

With a bit of delay, I confirm that the proposed change fixes our crashes on the IBM JDK.

Thanks a lot.

GianMaria.