Problem getting Python float32 as Java Float instead of Double

[HSommer]

Hi, I'm evaluating Jep 3.8.2 and feel stuck with a floating type conversion issue.

I'm new to Jep and Python though.

My Java application hands over an object of a protobuf-generated class to a Python script, for some user-supplied computation. The class contains data of type List<Float> that should be modified in Python and then be written back to the Java protobuf object.

It's fine to convert the list of Java Floats to a float32 numpy array on the Py side, but the problem is getting back the float32 to Java.

There is always auto-conversion by Jep. When I acess the data afterwards in Java, my List<Float> contains Double objects, and casting is a mess. Especially since my application is generic, i.e. it does not know the protobuf classes.

In isolated testing, the conversion would work using jep.getValue("myFloat", Float.class), but in the real project my floats are not top-level, instead they are embedded inside the protobuf class.

With jep.getValue("np.float32(myFloat)") I still get Double, and struct.pack("=f", myFloat) gives String.

It's a scientifc project with high data rates and volumes, so that defining the protobuf types as double is probably not an option.

The Py scripts are meant to be written by non-software people and should ideally not contain Jep- oder Java-specific code.

Can anyone help me how to set 32 bit floats from Python on a Java object?

With type erasure, I'm especially sceptical what Jep could do for the case of List<Float>. But let's see...

[Ben Steffensmeier]

In the current version of Jep, the only solution I can see would be to import java.lang.Float in python and construct a new Float for every element in the list before sending it to python, similar to your examples, it would look something like this:

jep.eval("from java.lang import Float as JavaFloat")
Object thisIsAFloat = jep.getValue("JavaFloat(myFloat)")

I know you were hoping to avoid pulling Java classes into your Python but I see no other way to accomplish exactly what you are requesting.

If you are concerned about performance, then I would recommend float[], NdArray, or DirectNdArray instead of List<Float>, since those types support bulk conversion while maintaining 32 bits without creating the boxed objects.

The problem is that Jep only handles conversion for a limited number of types, in general new conversions are only added when someone needs them. For numpy types only ndarrays are currently supported, Jep is completely clueless when it comes to the regular numpy numeric types like float32. The only reason float32 is converted to Double is because it extends the Python builtin float type, so the Jep conversion for that type is handling it, which defaults to Double since a python float is 64 bits on most platforms. That behavior also seems to be platform dependent within numpy, on the system I am using right now a numpy float32 does not extend a Python float so Jep will only convert it to a String, however numpy float64 does extend a Python float so it inherits the smarter Jep conversion.

It would be relatively simple to modify Jep to add conversions for the obvious numpy types, and I agree that would be more correct and resolve your issue. I have opened an issue on github and I currently plan to try to add that for the 3.9 release next year. If you would like to test it out before that let me know and I can fix it sooner, it won't be in an official release but if you would like to run it off of a development branch then I would appreciate the extra testing.

Ben

[HSommer]

Hi Ben,

many thanks for your very fast and helpful feedback! Creating Java Floats from Python works in my test. It may even be OK for my customers in terms of "tainted Python".

It would be great to get more numpy types supported in Jep, as you requested.

I guess though that this would only remove the explicit Float creation for scalars, not for List<Float>, right?  I'd nonetheless be happy to run some tests for you.

For performance I wish I could use DirectNdArray etc, but the protobuf-Java spec demands List<Float>, so that's what I'm stuck with, unless I code-generate some wrapper class around the protobuf classes...

(It looks like I can only reply to my own post, not to Ben's reply...)

[Ben Steffensmeier]

I have checked in a fix in the dev_3.9 branch in commit 59523fd. I think this will fix your issues even in List<Float>. Type erasure prevents us from knowing that you need a List<Float>, so Jep will essentially try to create a List<Object>, however the change affects the default conversions for numpy types so when Jep needs an Object to fill in the List and it has a float32 it will convert to a Float since that is now the most accurate Object conversion, the end result should be a List<Float>. You just have to be very careful that the Python list only contains numpy.float32 and not the builtin float type.

Ben

[HSommer]

I've built and installed dev_3.9, and now get the following JVM crash on the line

jep.eval("import numpy as np");

From the fatal error log I get the frame

C  [multiarray.cpython-35m-x86_64-linux-gnu.so+0x11064b]  PyArray_Item_INCREF+0x92b

My other tests are still passing without crashes, so I suspect that the problem is particular to the new changes.

Should I send you the error log or some gdb output?

[Ben Steffensmeier]

If you have just imported numpy, then you should still be too early for any of the new code to have run. That particular crash usually means that you have used numpy from two different embedded interpreters, something that numpy does not support. We have seen issues on this in this before. You need to add numpy to the list of shared modules in your JepConfig or use SharedInterpreters instead of plain Jep objects.

Ben

[HSommer]

In the end you were right that I did not use SharedInterpreters correctly for numpy.

But it was not so obvious, since there were no issues with my unit test running with Jep 3.8.2., and I have to share numpy even for a Jep instance that seems like it would not use numpy.

In case you'd like to reproduce the crash, I've slimmed down my TestNG code to the following:

    @Test
    public void testCrash() throws Exception {
        // Here it's not obvious (to me) that we load numpy:
        try (Jep jep = new JepConfig().addIncludePaths("../test_py")
//                .addSharedModules("numpy")  // no crash if we share numpy
                .createJep()) {
           
            List<Double> myList = new ArrayList<>();
            myList.add(1.234);
            jep.eval("from JepCrash import *");
            jep.invoke("listArgChangeAndAddValue", myList);
        }
       
        // For the second instance of Jep, the numpy import is enough to crash
        try (Jep jep = new JepConfig().addSharedModules("numpy").createJep()) {

            jep.eval("import numpy as np");
        }
    }

This Java code calls the following function in JepCrash.py (note that both list manipulation lines are needed to get the crash!):

def listArgChangeAndAddValue(myList):
    myList[0] = 1.1
    myList.add(12345.678)

I'm using CentOS Linux release 7.4.1708 (Core), Python 3.5.3 Anaconda custom (64-bit), openjdk version "1.8.0_161", TestNG.

[HSommer]

Still with a recent new build of the 3.9 branch, I must work around JVM crashes by including the seemingly unnecessary call "addSharedModules("numpy")" in cases like in the test code I posted above.

These crashes did not occur with jep 3.8.2, so I still think that they were introduced in 3.9, either as part of the 59523fd fix or elsewhere.

Aside from the shared module issue, the improved handling of numpy scalar types in jep 3.9 works for me in cases like

  jep.getValue("np.float32(myFloat)")

where now I get a Java Float.

There are other cases where it's not working, e.g.

  jep.eval("ndarray = np.array(myFloat, dtype=np.float32)");
  jep.getValue("ndarray.item()");

because I still get a Java Double.

[Ben Steffensmeier]

It looks like array.item() returns a python float, not a numpy float32., see the python snippet below. We always return Double for python floats unless you specify Float.class in getValue().

>>> myFloat = 3.5
>>> type(np.float32(3.5))
<type 'numpy.float32'>
>>> ndarray = np.array(myFloat, dtype=np.float32)
>>> type(ndarray.item())
<type 'float'>

Ben

[HSommer]

Thanks for pointing out the issue with item(). Also my other remaining float32 issues were due to some unintentional conversions to float64.

Now I have a recipe for manipulating a protobuf-generated Java object, which contains List<Float>, in a Python script:

1. Set the object as Python veriable: jep.set("myStruct", myStruct);
2. Copy the List<Float> to a numpy array: jep.eval("ndarray = np.array(myStruct.myList, dtype=np.float32)");
3. Change (or append etc) values: jep.eval("ndarray[0] = np.float32(2.345)");
4. Convert the numpy array to a Python list and set it on the Java object: jep.eval("myStruct.myList = list(ndarray)");

In this way the float32 type is preserved, and in Java I see again the expected Float objects.

It is nice that I don't need to work with Java types in Python. The conversion to numpy array and later back to Python list I find not bad.

Just a small improvement proposal: Would it make sense to support a direct assignment of the numpy array in step 4, as in "myStruct.myList = ndarray"?

At the moment I get "jep.JepException: <class 'RuntimeError'>: Error matching ndarray.dtype to Java primitive type"

The "myList" field in Java is defined as "public final List<Float> myList = new ArrayList<>();"

Thanks!

Heiko

[Nathan Jensen]

We support ndarray -> primitive array, e.g. numpy.float32 array to Java float[].  We can't safely support List<Float> due to Java generics since at runtime it is just List<Object> and we can't determine that they are Floats (or supposed to be Floats) inside the List.

--
You received this message because you are subscribed to the Google Groups "Jep Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jep-project...@googlegroups.com.
To post to this group, send email to jep-p...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jep-project/bd12f26b-8fe2-4142-b70d-64bc7d15c177%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[HSommer]

On the issue with Java generics, Ben wrote (above, see 2018-10-30): "when Jep needs an Object to fill in the List and it has a float32 it will convert to a Float since that is now the most accurate Object conversion".

Wouldn't the same argument also apply for the conversion of ndarray of float32 to a Java List?

[Ben Steffensmeier]

Heiko,

I agree it would be nice to support automatic ndarray->java.util.List conversion. Unfortunately there are limitless possibilities for different type conversions and I do not have much time for Jep development so I cannot promise it in the near future. If you would like to open a github issue to remind us then it is more likely it will get included in the future. If you would like to try writing the code yourself, I could point you at the right code in Jep and you could submit a pull request on github to get it in Jep 3.9. Currently all the conversions are written in C, although another feature I would like to add someday is the ability to write conversions in Python or Java to make it easier for contributors to add them.

Ben