Attempt to embed julia in ruby

[Remy Drouilhet]

Hello,

I am mostly an R and ruby user. I recently discovered  the very promissing julia (I love her too!) .  I like the fact that R is very easily embeddable in C code and then in ruby code. I would like to embed julia in ruby.

To do that, I am using  or adapting the  jlapi.c file.

Here are a few comments from my very early trials:

.)  jl_bytestring_ptr is missing in src/julia.expmap (linux fails to compile even with DLLEXPORT)

.) htableh.inc is not directly embedded in the include/julia directory but only in the src/support. As a direct consequence, I can't compile my ruby extension. My solution was to copy it by hand in  include/julia. Is it possible to do it in the "make install" command? 

.) in client.jl: is it possible to split the function init_load_path into 2 files like this: 

function init_load_path()

    vers = "v$(VERSION.major).$(VERSION.minor)"

    global const LOAD_PATH = ByteString[

        abspath(JULIA_HOME,"..","local","share","julia","site",vers),

        abspath(JULIA_HOME,"..","share","julia","site",vers)

    ]

    # To delete: global const DL_LOAD_PATH = ByteString[]

end

function init_dl_load_path()

    global const DL_LOAD_PATH = ByteString[]

end 

Of course,  for the julia repl to be still useable, one needs to update the run_repl function as follows:

init_load_path()      

init_dl_load_path() # to be added

The reason why I am asking this is that, I need to load  init_dl_load_path() first when embedding julia in ruby in order to  load  libpcre properly. Indeed, joinpath (called by init_load_path) uses libpcre.In particular abspath("/Users") returns false instead of true. LOAD_PATH is then abnormally prefixed with my current working directory. No require or using is then possible. One solution is to set LD_LIBRARY_PATH to the lib/julia directory but I think it is much better to do the same task by setting DL_LOAD_PATH used in dlopen by julia.

[Remy Drouilhet]

As a complement maybe more helpful, I put this preliminary devel  to github (see  https://github.com/rcqls/jl4rb). 

[Remy Drouilhet]

Sorry, but I think there is a better solution for the 3rd point: client.jl could be updated as follows (with no use of abspath which requires DL_LOAD_PATH when finding libpcre):

function init_load_path()
        vers = "v$(VERSION.major).$(VERSION.minor)"

        global const DL_LOAD_PATH = ByteString[
            join([JULIA_HOME,"julia"],Base.path_separator)
        ]

        global const LOAD_PATH = ByteString[
            abspath(JULIA_HOME,"..","local","share","julia","site",vers),
            abspath(JULIA_HOME,"..","share","julia","site",vers)
        ] 

end 

Definition of LD_LOAD_PATH depends on the definition of JULIA_HOME which seems to be related to the location of sys.ji (actually the same in the default installation).

[Viral Shah]

It is really exciting to see you try this out. Since you are a trailblazer on this, please feel free to submit PRs that make your job easier to embed. I do see that Steve Johnson made some updates to jl_init.

There is an open issue on this, but this again reminds me that we should start documenting jlapi, and also have a test that embeds it in a C program.

-viral

[Jameson Nash]

I think this is a good idea. I encountered the same difficulty as you (with libpcre) when I set LD_LIBRARY_PATH to MacPorts.

[bobnn...@gmail.com]

I've always had better luck setting DYLD_FALLBACK_LIBRARY_PATH instead of DYLD_LIBRARY_PATH or LD_LIBRARY_PATH. man dyld has some useful info.

Bob

[Remy Drouilhet]

After ruby, I gave a try to embed julia in R (see the R package at https://github.com/rcqls/jl4R). It is very experimental but it is fun to see julia working inside R session (even if I think that is not the main purpose to use it inside a console). Of course, a lot of stuff remain to be done.

As a comment, I have noticed that in the initialization process similar to _start() called by repl.c and jl_init() in jlapi.c, Base.reinit_stdio() failed to reload properly STDOUT and STDERR. As a direct consequence, no way to print something and even worth since it leads to a segmentation fault. After many many attempts, I managed to redirect STDOUT and STDERR to newly created IOBuffer (allowing us to capture julia output inside ruby and R environment) . Actually, it is a better solution to redirect STDOUT and STDERR when embedding julia inside another C language.

[Steven G. Johnson]

On Thursday, July 25, 2013 11:50:34 AM UTC-4, Remy Drouilhet wrote:

After many many attempts, I managed to redirect STDOUT and STDERR to newly created IOBuffer (allowing us to capture julia output inside ruby and R environment) . Actually, it is a better solution to redirect STDOUT and STDERR when embedding julia inside another C language.

Ability to redirect stdio to an arbitrary stream is a priority for us since we need it for the IPython front-end.  See https://github.com/JuliaLang/julia/issues/3823

[Remy Drouilhet]

Hi Steve, I don't know if the solution I found is good or not since I am really new in julia development. I am not an expert in debugging but by accident I realized that Base.reinit_stdio() (thanks for this code, it was really helpful) failed to reload STDOUT and SDTERR. Actually, before my last try, I called julia code to initialize  STDOUT as an IOBuffer and I realized that the objet was still a TTY. My last try was then to  do the same by mixing julia code and C code just after the call of reinit_stdio which seems to load properly the uv_ objects. I guess better solution directly in julia may be investigated even from now it is sufficient for what I am doing.

[Remy Drouilhet]

Sorry Steven,  I misspelt your first name. In addition, I only tested my ruby gem and R package on MacOSX 10.8. I made a very quick check on linux (via virtualbox) and I am not sure my solution is still working. I need to check longer. 

[Remy Drouilhet]

longer check on linux. The gem jl4rb seems to work correctly after addingin julia.expmap: jl_bytestring_ptr, jl_unbox_long, jl_unbox_bool (and the modification of client.jl to avoid LD_LIBRARY_PATH setting).

The R package jl4R does not work on linux but I guess that it is because R package was developed for R for x86_64 and my linux R is not. As I said, it is experimental.

 

[Jameson Nash]

In 1a17b6425e4727c074ea0dd7e18f122341444ad3, I have implemented a variant of your suggestion.

-jameson

[Remy Drouilhet]

Great!