How to distribute an app bundle with jre, python and Jep

[ma...@robertosanchez.me]

Hi, I've been reading some posts in github about how to select the PYTHONPATH programmatically, but if I understood correctly, that's no possible, Jep always use the Python environment configured externally by the system (env. variable PYTHON_HOME or similar).

My application is a JavaFX application that is distributed with the JRE, currently I'm using jython as python environment and It works more or less correctly, but I would like to use a "real" cpython env. with the latests enhancements and bugfixes, so It seems that Jep could be a good choice, HOWEVER, It seems that with Jep I cannot distribute my app with a python environment because I cannot select it when I run Jep.

Honestly I don't understand why the Python home cannot be selected programmatically (for instance in PyConfig.setPythonHome(...)), I suppose that there are technical constraints that I no see, but It would be a great feature in order to allow an app packaging with all the dependencies.

If I'm wrong and I can create an app bundle with all environments and use it from my app, please, let me know.

--

Roberto

[ma...@robertosanchez.me]

By the way, It exists a Python method to set the Python home in the Python C API

https://docs.python.org/2/c-api/init.html#c.Py_SetPythonHome

So maybe we can call it just like other initial params from PyConfig, in pyembed_preinit() or in pyembed_startup() just before the call to Py_Initialize(), I don't know if this approach has some drawback, but It seems quite simple.

--

Roberto

[Nathan Jensen]

Thank you for looking up the methods that exist in the C-API to do it.  I'm not quite understanding the use case where you can't set the environment variable before launching the application.  Can you elaborate?  What operating system(s) are you targeting?

The biggest concern would be if the PYTHON_HOME that is set programmatically would conflict with the other environment variables such as LD_LIBRARY_PATH.  The python library, for example libpython2.7.so, is already loaded by the time you get into Jep, so I'm not sure what setting PYTHON_HOME would do after that.  But other than that concern, I don't see an issue with adding an ability to set PYTHON_HOME to Jep's API since CPython provides the method.

--
You received this message because you are subscribed to the Google Groups "Jep Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jep-project+unsubscribe@googlegroups.com.
To post to this group, send email to jep-p...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jep-project/89375b7c-0eb4-4013-abcb-c469a4011619%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[ma...@robertosanchez.me]

Hi Nathan, mi problem with the env variables is that, AFAIK, I cannot set it using the standard packaging system provided by JavaFX (based in Inno Setup for Windows).

In other words, the final program is a .exe file (in Windows) that makes the use of Java transparent to final user, the Inno Setup script can modify user or global variables, but It cannot not set a env. variable only for the program execution.

I can set the LD_LIBRARY_PATH, using -Djava.library.path, so It would be nice a -Dpython.home solution or similar, I mean, I don't need an specific API but something I can manage using the java parameters, the packaging system allows to add parameters to your program command line, but not set env. variables previous to the program call, at least I couldn't find a way to do it automatically in the app installation.

On Friday, September 8, 2017 at 6:25:57 PM UTC+2, Nathan Jensen wrote:

Thank you for looking up the methods that exist in the C-API to do it.  I'm not quite understanding the use case where you can't set the environment variable before launching the application.  Can you elaborate?  What operating system(s) are you targeting?

The biggest concern would be if the PYTHON_HOME that is set programmatically would conflict with the other environment variables such as LD_LIBRARY_PATH.  The python library, for example libpython2.7.so, is already loaded by the time you get into Jep, so I'm not sure what setting PYTHON_HOME would do after that.  But other than that concern, I don't see an issue with adding an ability to set PYTHON_HOME to Jep's API since CPython provides the method.

On Fri, Sep 8, 2017 at 4:34 AM, <ma...@robertosanchez.me> wrote:

By the way, It exists a Python method to set the Python home in the Python C API

https://docs.python.org/2/c-api/init.html#c.Py_SetPythonHome

So maybe we can call it just like other initial params from PyConfig, in pyembed_preinit() or in pyembed_startup() just before the call to Py_Initialize(), I don't know if this approach has some drawback, but It seems quite simple.

--

Roberto

On Friday, September 8, 2017 at 10:45:56 AM UTC+2, ma...@robertosanchez.me wrote:

Hi, I've been reading some posts in github about how to select the PYTHONPATH programmatically, but if I understood correctly, that's no possible, Jep always use the Python environment configured externally by the system (env. variable PYTHON_HOME or similar).

My application is a JavaFX application that is distributed with the JRE, currently I'm using jython as python environment and It works more or less correctly, but I would like to use a "real" cpython env. with the latests enhancements and bugfixes, so It seems that Jep could be a good choice, HOWEVER, It seems that with Jep I cannot distribute my app with a python environment because I cannot select it when I run Jep.

Honestly I don't understand why the Python home cannot be selected programmatically (for instance in PyConfig.setPythonHome(...)), I suppose that there are technical constraints that I no see, but It would be a great feature in order to allow an app packaging with all the dependencies.

If I'm wrong and I can create an app bundle with all environments and use it from my app, please, let me know.

--

Roberto

--
You received this message because you are subscribed to the Google Groups "Jep Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jep-project...@googlegroups.com.

To post to this group, send email to jep-p...@googlegroups.com.

[ma...@robertosanchez.me]

In this example (from mercurial) they set the python home if there is no defined by PYTHONHOME variable and load the python dll manually:

https://github.com/spraints/for-example/blob/master/mercurial/exewrapper.c

Actually, It's quite similar to the approach that I tried to explain, in this example It seems that the LD_LIBRARY_PATH previous value is not a problem or It's ignored. Besides in the python C API doc, doesn't say anything about the problem with LD_LIBRARY_PATH if the python home is set programatically.

Anyway, this weekend, if I can I'll try to test it by myself, I'll post here the result.

[ma...@robertosanchez.me]

Hi, I've been doing a proof of concept about using the method Py_SetPythonHome and I think that It works, I couldn't test it on windows and It doesn't work on python 3.x because the signature method has changed from (char*) to (wchar_t *) as input parameter.

I did a fork on my account that contains the modified files: https://github.com/rsc1975/jep

With this change we can use Jep in an environment that hasn't a python env preconfigured in the standard locations, so we can use whatever python env in the machine, including an environment distributed with the own application, that is, we can use it without any previous action over the system but the installation of out own application bundle.

To test it I used the following code:

public static void main(String[] args) throws JepException {

File pythonHome = new File("src/main/deploy/python-env/");

PyConfig conf = new PyConfig();

conf.setPythonHome(pythonHome);

Jep.setInitParams(conf);

try(Jep jep = new Jep()) {

    jep.eval("import platform; print('Platform:', platform.platform())");

    jep.eval("print('python_version:', platform.python_version())");

}

}

The program load the python library located in the provided pythonHome variable, I added the main code in pyembed.c file, you can see it in: https://github.com/rsc1975/jep/blob/master/src/jep/pyembed.c#L238

I haven't programmed in C in the last 15 years, so, It's pretty sure that my code contains some issue :-) so please, consider it as a proof of concept:

void loadPythonEnv(const char* pythonHome)

{

    char* pyLibFilename = get_python_lib_filename(pythonHome);

    if (pyLibFilename == NULL) {

        fprintf(stderr, "ERROR: Python lib doesn't found on python home: %s\n", pythonHome);

        return;

    }

    char* pyLibPath = malloc((strlen(pythonHome) + 25)*sizeof(char));

    char* pyHomeNormalized = malloc((strlen(pythonHome)+1)*sizeof(char));

    strcpy(pyHomeNormalized, pythonHome);

    if (pyHomeNormalized[strlen(pyHomeNormalized) - 1] == FILE_SEP) {

        pyHomeNormalized[strlen(pyHomeNormalized) - 1] = '\0';

    }

    sprintf(pyLibPath, "%s%clib%c%s", pyHomeNormalized, FILE_SEP, FILE_SEP, pyLibFilename);

    

    void* pyLibrary = dlopen(pyLibPath, RTLD_NOW | RTLD_LOCAL);

    free(pyLibPath);

    if(pyLibrary == NULL) 

    {

        free(pyHomeNormalized);

        fprintf(stderr, "%s\n", dlerror());

        return;

    }

    

    Py_SetPythonHome(pyHomeNormalized);

    printf("HOME final: %s\n\n", Py_GetPythonHome());

 }

I only tested it on OSX, I need to add the C code for Windows to make it usable for my use case. 

If you think that is interesting to add to the project, I can complete it and adapt it to the project style with your help and create a pull request, if not, well, I can use my own fork, but honestly, I think that is a good feature, make the lib usable in more situations.

[Ben Steffensmeier]

Hi Roberto,

I looked over your fork and I don't understand the purpose of the dlopen. I'm not an expert in how the library loading works but from what I have seen by the time you dlopen the libpython that is used with jep will have already been loaded. In the static initializer for Jep.java we do System.loadLibrary("jep"), since jep depends on libpython the dynamic linker will load a libpython and link it with libjep. In my experieince if you don't have a libpython on the library path then this call will fail so at this point you have already selected a libpython. Since your code runs after this, it is to late to change which libpython is linked to libjep.

I think if we truly wanted to be able to select which python to load programatically we would have to delay loading of libjep until after the python home is set and then find libjep and libpython relative to PYTHON_HOME. Based off what you have said, I don't think you need all of that, it sounds like you have a mechanism to load libjep and libpython successfully but your PYTHON_HOME is out of sync. I think you have a good case for just adding a simple API for calling through to Py_GetPythonHome and ignoring the loading of the library. Given the variety of ways to manipulate the library path I think it would be impossible to always guarantee that the libpython and the PYTHON_HOME match but the addition you have suggested would at least give developers a way to bring them back together if they don't align.

Ben

[ma...@robertosanchez.me]

Hi Ben, the actual dlopen() replace the previously loaded library by the new one the user has selected, but I understand what you say, maybe the S.O. has already loaded a pyhonlib and that makes that my test works, anyway, If the jeplib is loaded in Java, I should remove the dlopen(pythonlib) from C code and move it to Java just before the load of jeplib, in that approach the loads (python and jep libs) should be done in the moment we invoke the interpreter because at that moment we know if we want to use the system python env or a custom one.

For me, the goal is to get the capability of distribute and run an application in an machine that has not any python environment previously, especially for Desktop apps in Windows.

Thanks for your comments.

On Monday, September 11, 2017 at 8:27:21 PM UTC+2, Ben Steffensmeier wrote:

Hi Roberto,

I looked over your fork and I don't understand the purpose of the dlopen. I'm not an expert in how the library loading works but from what I have seen by the time you dlopen the libpython that is used with jep will have already been loaded. In the static initializer for Jep.java we do System.loadLibrary("jep"), since jep depends on libpython the dynamic linker will load a libpython and link it with libjep. In my experieince if you don't have a libpython on the library path then this call will fail so at this point you have already selected a libpython. Since your code runs after this, it is to late to change which libpython is linked to libjep.

I think if we truly wanted to be able to select which python to load programatically we would have to delay loading of libjep until after the python home is set and then find libjep and libpython relative to PYTHON_HOME. Based off what you have said, I don't think you need all of that, it sounds like you have a mechanism to load libjep and libpython successfully but your PYTHON_HOME is out of sync. I think you have a good case for just adding a simple API for calling through to Py_GetPythonHome and ignoring the loading of the library. Given the variety of ways to manipulate the library path I think it would be impossible to always guarantee that the libpython and the PYTHON_HOME match but the addition you have suggested would at least give developers a way to bring them back together if they don't align.

Ben

[Nathan Jensen]

Hi Roberto, I opened an issue for this so we don't forget about it.  https://github.com/ninia/jep/issues/92  You can add more commentary there or subscribe to notifications on the issue.  Thank you for providing your sample code.  You could also provide the code in a pull request, though at this point I'm not sure if we want to set it in Java or in C.

To unsubscribe from this group and stop receiving emails from it, send an email to jep-project+unsubscribe@googlegroups.com.

To post to this group, send email to jep-p...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jep-project/3fff2955-84e2-43b6-9eb8-898f2ba5c833%40googlegroups.com.

[ma...@robertosanchez.me]

That's great :-) I'll add my comments shortly, I think that we need both actions, load the lib. en Java and set the pythonHome in C (even the pythonPath), but I would like to test it previously on my own, I'll post my results/comments on github issue.