Embedding Julia with C++

[Kostas Tavlaridis-Gyparakis]

Hello,
I am trying to embed Julia in C++ but I currently face some sort of issues.
I am trying to follow the instructions shown here.
First things first, I run Ubuntu 15.04 and my Julia version is v. 0.3.2 (it's the
version that is automatic installed when installing julia from ubuntu center).
I use eclipse for my C++ projects, yet again I have the following issues, ac-
cording to the instructions before trying to write any julia code in C or C++ you
need first to:
1) link the julia library (assuming I undersand correctly this refers to  libjulia.so),
which should be located in "Julia_DIR/usr/lib", yet again in my julia directory
there is no folder under the name usr. I did though find a libjulia.so file in an
other directory of my pc ("/usr/lib/x86_64-linux-gnu/julia") and added this one
instead.
2) include the path of julia.h which should be located in "Julia_DIR/inclue/julia"
now again in my julia directory there are no such folders and in general there
is nowhere in my pc any file such as julia.h. I did sth that is probably wrong
and stupid but couldn't come up with anything else I downloaded this and I
included the location of where julia.h is located to eclipse as well with the direc-
tions of all the other header files that were inculuded inside julia.h.

Now when in Eclipse I am trying to compile and run a few simple julia commands
having included julia.h i receive an error saying that there is no uv.h file in my
system which is needed in one of the julia header files.
I know that my whole approach is wrong, but yet again I couldn't find anywhere
in my pc the proper folders or files in order to follow the steps that were sugges-
ted in the julia website for running julia code inside C++.
Any help would be much appreciated.

Also, one more thing I wanted to ask is the following, in general writing Julia
code inside a C++ code is limited?
What I want to do in general is write a JuMP model inside C++, so in general
is this possible, in the sense that by embedding Julia inside C++, will I be able
to use all of the tools and code of Julia language or is this only limited to a cer-
tain amount of commands and packages?

[Isaiah Norton]

The distro install file locations are going to be different, as you have found (which we should probably note in the docs -- feel free to open an issue or ideally a pull-request).

 i receive an error saying that there is no uv.h file in my
system which is needed in one of the julia header files.

Julia uses a fork of libuv, so I'm not sure how to get a proper uv.h for a linux distro build (not at a linux computer right now). Maybe Milan or Tony will have an answer. (see also: https://github.com/JuliaLang/julia/pull/8880)

Also, one more thing I wanted to ask is the following, in general writing Julia
code inside a C++ code is limited?
What I want to do in general is write a JuMP model inside C++, so in general
is this possible, in the sense that by embedding Julia inside C++, will I be able
to use all of the tools and code of Julia language or is this only limited to a cer-
tain amount of commands and packages?

Technically, you should be able to do anything through the embedding API that you can do in the interactive shell (i.e.: you can call `jl_eval` with any legal Julia string). Practically, writing more than a few lines of Julia code via the embedding API sounds kind of painful (to me). The alternative is to write as much code as possible in pure Julia, and then expose a very small API to set up the problem and call the solver from C (see `cfunction`: http://docs.julialang.org/en/release-0.3/manual/calling-c-and-fortran-code/#passing-julia-callback-functions-to-c).

[Jeff Waller]

Embedded Julia is of particular interest to me. To answer your question, everything in Julia is available via embedded Julia.

I would very much discourage use of version 0.3.2; avoid it if you can.  I think that particular version has the uv.h problem which is fixed in later versions. Can you gain root on this host?  If so you can get 0.3.9 via PPA.  Or even better if you can get ahold of one of the nightly builds, then 0.4.x comes with julia_config.jl, which figures out all of the right compile flags automatically.  You just have to cut and paste in a Makefile.  But if no makefile, you can run it and know the necessary compile time flags.

[Kostas Tavlaridis-Gyparakis]

I did download the 0.4 nightbuilt which includes the above mentioned files in the proper location, but now Eclipse is throwing me a different
error I can not sort out how to overcome. When I try to run a small cpp file with a few julia comands Eclipse is compiling the file but when I
try to run it it throws me the following message:

"System image file "/home/kostas/workspace/juli/Debug/../lib/x86_64-linux-gnu/julia/sys.ji" not found "

Futhermore since I am really new to Julia I am not sure and I don't know a lot of the existing tools, is it possible to write a function in julia
that takes as an argument some data creates a model and solves it and call this function from inside my c++ project?
I am asking this as in the example in the link attached by Isaiah with the qsort, the whole process is done inside julia framework.
Whereas in my case I would be interested to write a julia program, like the one described above that I would be able to call as a function
(I want it to solve a subproblem actually) inside my c++ project in eclipse.
Is this relatively easy to be done?
Because I think this would be the best approach for my case.

[Isaiah Norton]

You probably need to call `jl_init(NULL)` at the beginning of the program. If you have not done so yet, I would suggest to read the embedding documentation:

    http://docs.julialang.org/en/release-0.3/manual/embedding/

and start with the embedding example in the source:

    https://github.com/JuliaLang/julia/blob/master/examples/embedding.c

[Kostas Tavlaridis-Gyparakis]

I am simply trying to run the first example attached in the embedding documantation which is the following block of code:

#include <iostream>
#include <julia.h>
using namespace std;

int main() {
      /* required: setup the julia context */
        jl_init(NULL);

        /* run julia commands */
        jl_eval_string("print(sqrt(2.0))");

        /* strongly recommended: notify julia that the
             program is about to terminate. this allows
             julia time to cleanup pending write requests
             and run all finalizers
        */
        jl_atexit_hook();

    return 0;
}

And when I try to run the program in eclipse (after having linked the library and defined the path of the header file) the above mentioned
error message appeas which says:

- System image file "/home/kostas/workspace/juli/Debug/../lib/x86_64-linux-gnu/julia/sys.ji" not found

[Isaiah Norton]

I guess this is still a distro path issue. The following suggestion is not very general, but to at least get going, you could try:

jl_init("/home/kostas/workspace/juli/Debug/../lib/x86_64-linux-gnu/julia/")

[Kostas Tavlaridis-Gyparakis]

Didn't manage to make the code run. I am really wondering what I am missing here...

[Kevin Squire]

It's not just that julia is misspelled as juli in the path, is it?

[Kostas Tavlaridis-Gyparakis]

Νο, unfortunately it's not that.
I just gave to the project the name "juli" I forgot to type the a and didn't bother correct it aftewrards, so
the path name is correct.

[Tony Kelman]

Are you running on latest master? This is probably another casualty of https://github.com/JuliaLang/julia/pull/11640

[Kostas Tavlaridis-Gyparakis]

Sorry didn't get your question.
In general the problem is that Eclipse doesn't seem to be able to recognize/include this sys.ji file, and I
don't know how to fix it.

[Scott Jones]

 There is no .ji file anymore, although there is a command line option to produce it, 

--output-ji <name>

[Kostas Tavlaridis-Gyparakis]

But, sys.ji does exist in my folder of the path "/home/kostas/workspace/juli/Debug/../lib/x86_64-linux-gnu/julia/"

[Kostas Tavlaridis-Gyparakis]

So, finally issue was solved by this answer here.

[Kostas Tavlaridis-Gyparakis]

Hello again,
I am writing to this post, as I face some more problems with trying to link Julia functions with C++ code.
The thing is that I changed my current Julia version, I installed Julia running the source code as I wanted
to install the Cxx package as well.
So, I just went to run the first smallest example with some c code and c++ code that uses a julia function
as presented here, so I only try to run the following file:

#include <julia.h>

int main(int argc, char *argv[])

{
    /* required: setup the julia context */
    jl_init(NULL);

    /* run julia commands */
    jl_eval_string("print(sqrt(2.0))");

    /* strongly recommended: notify julia that the
         program is about to terminate. this allows
         julia time to cleanup pending write requests
         and run all finalizers
    */
    jl_atexit_hook();
    return 0;
}

The first thing that was different was that when I was including just the path of julia.h and libjulia.so and I
run the command like this:

gcc -o test main.c -I /home/kostav/julia/src -L/home/kostav/julia/usr/lib -ljulia

I receive an error saying:

In file included from main.c:1:0: /home/kostav/julia/src/julia.h:12:24: fatal error: libsupport.h: No such file or directory  #include "libsupport.h" compilation terminated.

Then when I add the path of libsupport.h I receive a similar error msg saying that uv.h is missing so I have
to add its path as well, and as a result I finally run the following command:

gcc -o test main.c -I /home/kostav/julia/src -I /home/kostav/julia/usr/include -I /home/kostav/julia/src/support -L/home/kostav/julia/usr/lib -ljulia

In that case I receive various errors about undefined references regarding llvm functions and as a result I include
as well the equivelant library and finally run the following command:

gcc -o test main.c -I /home/kostav/julia/src -I /home/kostav/julia/usr/include -I /home/kostav/julia/src/support -L/home/kostav/julia/usr/lib -ljulia -lLLVM-3.7svn

On this case the program does compile but when I try to run it, I receive the following error:

error while loading shared libraries: libjulia.so: cannot open shared object file: No such file or directory


The exact same implies for the equivalent program in c++. Note that when I try to call a julia function inside a c++ more complicated
structure (namely in a class) I receive a different error. More specifically just for the very simple class that exists
in this Cxx example for just using the same simple julia function inside the class that leads to the following slighltly different cpp file

#include "ArrayMaker.h"
#include <iostream>

using namespace std;

ArrayMaker::ArrayMaker(int iNum, float fNum) {

    jl_init(NULL);
    jl_eval_string("print(sqrt(2.0))");

    cout << "Got arguments: " << iNum << ", and " << fNum << endl;
    iNumber = iNum;
    fNumber = fNum;
    fArr = new float[iNumber];
    jl_atexit_hook();
}

float* ArrayMaker::fillArr() {
    cout << "Filling the array" << endl;
    for (int i=0; i < iNumber; i++) {
        fArr[i] = fNumber;
        fNumber *= 2;
    }
    return fArr;
}

With the header file being almost the same:

#ifndef ARRAYMAKER_H
#define ARRAYMAKER_H

#include <julia.h>

class ArrayMaker
{
    private:
        int iNumber;
        float fNumber;
        float* fArr;
    public:
        ArrayMaker(int, float);
        float* fillArr();
};

#endif

When I try to compile in terminal using the following command:

g++ -o test ArrayMaker.cpp -I /home/kostav/julia/src -I /home/kostav/julia/usr/include -I /home/kostav/julia/src/support -L/home/kostav/julia/usr/lib -ljulia -lLLVM-3.7svn

I do receive the following error:

/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crt1.o: In function `_start': /build/buildd/glibc-2.21/csu/../sysdeps/x86_64/start.S:114: undefined reference to `main'
collect2: error: ld returned 1 exit status



I am really not sure how to resolve any of the two errors and in general the second one is also pretty important to me, as in general my end goal
is starting from a julia file to use my self-written c++ code, where inside the c++ code I call some other self-written julia functions, I know it does
sound kind twisted, but that's what I am interested to do, as I am trying to tackle a big Mathematical Problem by builing the original problem in Jump
then decompose it and use a solver that is written in c++ (that's why I need to call c++ inside julia) which then needs again to send back information
to the original model and probably solve some subproblems with CPLEX (that's why I will need inside the c++ code to call julia functions again). That
with as much as few words as possible in order not to bother you with a lot of unnecessary information.

[Jeff Waller]

It's not clear to me which version you are using?  Depending on the version, It is referred

to in the URL you linked...  

I'll just cut to the chase use 0.4 and julia_config.jl as described in the doc, create a

Makefile just cut-and-paste the example, and augment with your source.  All but

one of your errors is a result of the wrong compile/link flags.

The last error is that main() is either not being compiled or linked, that's just straight-up

C programming, and has nothing to do with Julia.

As far as eclipse goes, I'm confident it's possible, I can't imagine eclipse not supporting

compilation using Makefiles, but even if it doesn't you can still automate things, but just

get something working first and you can embellish later.

TL;DR

0.4,  julia_config, cut-paste Makefile, add your source, done

[Kostas Tavlaridis-Gyparakis]

Hello thanks a lot for your answer.
You assumed correctly I do use version 0.4.
If I got right your proposal about the linking error is to use
the command jl_inti(JULIA_INIT_DIR) as shown in this
example and use a makefile where I just copy paste the
commands shown in the example, so the makefile looks like that:

JL_SHARE = $(shell julia -e 'print(joinpath(JULIA_HOME,Base.DATAROOTDIR,"julia"))')
CFLAGS   += $(shell $(JL_SHARE)/julia-config.jl --cflags)
CXXFLAGS += $(shell $(JL_SHARE)/julia-config.jl --cflags)
LDFLAGS  += $(shell $(JL_SHARE)/julia-config.jl --ldflags)
LDLIBS   += $(shell $(JL_SHARE)/julia-config.jl --ldlibs)

all: main


and my source code looks like that:

#include <julia.h>

int main(int argc, char *argv[])
{
    /* required: setup the julia context */

    jl_init(JULIA_INIT_DIR);

    /* run julia commands */
    jl_eval_string("print(sqrt(2.0))");

    /* strongly recommended: notify julia that the
         program is about to terminate. this allows
         julia time to cleanup pending write requests
         and run all finalizers
    */
    jl_atexit_hook();
    return 0;
}

Yet again when I try to compile I receive the following errors:

make: /home/kostav/julia/usr/bin/../share/julia/julia-config.jl: Command not found
make: /home/kostav/julia/usr/bin/../share/julia/julia-config.jl: Command not found
make: /home/kostav/julia/usr/bin/../share/julia/julia-config.jl: Command not found
cc     main.c   -o main
main.c:1:19: fatal error: julia.h: No such file or directory
 #include <julia.h>
                   ^
compilation terminated.
<builtin>: recipe for target 'main' failed
make: *** [main] Error 1


Note that I have really no experience with makefiles so I can not really handle them
properly, but I assumed that I should be working if I did all it was saying in the example.
Yet again I receive the above errors.

[Isaiah Norton]

make: /home/kostav/julia/usr/bin/../share/julia/julia-config.jl: Command not found

This is the problem, or part of it anyway. That file lives under `julia/contrib` in a source build.

The shell is your friend:

http://www.cyberciti.biz/faq/search-for-files-in-bash/

[Kostas Tavlaridis-Gyparakis]

Thanks a lot for the answer.
I have located the path of julia-config (/home/kostav/julia/contrib) and I did try to run it with setting this path.
Both by just writing the command in the terminal:

/home/kostav/julia/contrib/julia-config.jl --cflags --ldflags --ldlibs | xargs gcc embed_example.c

But I do receive error saying that: fatal error: julia.h: No such file or directory  #include <julia.h>


Or via the makefile, which I change accordingly into this:

CFLAGS   += $(shell $/home/kostav/julia/contrib/julia-config.jl --cflags)
CXXFLAGS += $(shell $/home/kostav/julia/contrib/julia-config.jl --cflags)
LDFLAGS  += $(shell $/home/kostav/julia/contrib/julia-config.jl --ldflags)
LDLIBS   += $(shell $/home/kostav/julia/contrib/julia-config.jl --ldlibs)

all: embed_example

And I receive the following errors:

make: home/kostav/julia/contrib/julia-config.jl: Command not found
make: home/kostav/julia/contrib/julia-config.jl: Command not found
make: home/kostav/julia/contrib/julia-config.jl: Command not found
cc     embed_example.c   -o embed_example
embed_example.c:1:19: fatal error: julia.h: No such file or directory

 #include <julia.h>
                   ^
compilation terminated.

<builtin>: recipe for target 'embed_example' failed
make: *** [embed_example] Error 1

Let me just remind you that in the case where I use jl_init(NULL);
and I run in terminal this command:

gcc -o test main.c -I /home/kostav/julia/src -I /home/kostav/julia/usr/include -I /home/kostav/julia/src/support -L/home/kostav/julia/usr/lib -ljulia -lLLVM-3.7svn

Program does compile and the error comes when I try to run the program where I receive msg saying that:

error while loading shared libraries: libjulia.so: cannot open shared object file: No such file or directory

So, maybe it would be simpler to try to solve this one?
I don't know, again any suggestions would be really welcome!

[Jeff Waller]

CFLAGS   += $(shell $/home/kostav/julia/contrib/julia-config.jl --cflags)

take out the 2nd $

CFLAGS   += $(shell /home/kostav/julia/contrib/julia-config.jl --cflags) 

what results?

[Kostas Tavlaridis-Gyparakis]

Thanks a lot, that fixed this error, but now it returns me the same error that I get when I just run the compile command
in terminal:

fatal error: julia.h: No such file or directory  #include <julia.h> compilation terminated.

[Jeff Waller]

This is the problem, or part of it anyway. That file lives under `julia/contrib` in a source build.

Hmm  this should be bundled into all 0.4, oh oh.

[Jeff Waller]

Ok this is turning into kind of a debugging session, but 

/home/kostav/julia/contrib/julia-config.jl --cflags

just run it on the command line.  What's the output?

What version of 0.4?  Is it current as of a few days ago?  In the version I'm using

all is well. I will obtain/compile the latest

[Kostas Tavlaridis-Gyparakis]

Any ideas on how to fix the compile error: julia.h: No such file or directory  #include <julia.h> compilation terminated.?

[Jeff Waller]

On Monday, July 13, 2015 at 10:54:57 AM UTC-4, Kostas Tavlaridis-Gyparakis wrote:

Any ideas on how to fix the compile error: julia.h: No such file or directory  #include <julia.h> compilation terminated.?

Yea of course.  This is a result of the -I flag having the wrong value.  

There was a space of time that this command (julia_config.jl) was not working

as a result of the new use of  sys.so instead of sys.ji, but that has sense been fixed, so

that's why I'm asking what version it was.  It would be working in the newest version, but

I am verifying that now.

The cause of libjulia.so not found is the link step is missing -Wl,-rpath, which julia_config gives you

that's why I keep coming back to it.

[Kostas Tavlaridis-Gyparakis]

Ok, my current Julia version (that I installed via running the source code) is: Version 0.4.0-dev+5841 (2015-07-07 14:58 UTC)
So, what should I do different so that -I flag gets the proper value?

[Jeff Waller]

On Monday, July 13, 2015 at 11:36:34 AM UTC-4, Kostas Tavlaridis-Gyparakis wrote:

Ok, my current Julia version (that I installed via running the source code) is: Version 0.4.0-dev+5841 (2015-07-07 14:58 UTC)
So, what should I do different so that -I flag gets the proper value?

Could you run /home/kostav/julia/contrib/julia-config.jl  --ldlibs and tell me what it returns?

Did you do a make install of Julia itself and are running out of that

installed directory or are you running out of where it compiled or possibly

simply copied the directories? That script assumes the former and I think

maybe you're doing the latter.

If so you can still use it, but you have to specify which julia so something like this

/home/kostav/julia/bin/julia  /home/kostav/julia/contrib/julia-config.jl  --ldlibs

is there any difference?  To get something working, cut-paste the output to the

semi-working step above

to use in a Makefile modify to 

$(shell /home/kostav/julia/bin/julia /home/kostav/julia/contrib/julia-config.jl --cflags) 

etc

[Kostas Tavlaridis-Gyparakis]

Hello,
Sorry for the delay in my answer.

>Could you run /home/kostav/julia/contrib/julia-config.jl  --ldlibs and tell me what it returns?

It returns: -Wl,-rpath,/home/kostav/julia/usr/lib -ljulia

>Did you do a make install of Julia itself and are running out of that

>installed directory or are you running out of where it compiled or possibly

>simply copied the directories? That script assumes the former and I think

>maybe you're doing the latter.

I downloaded the source code in folder (home/julia) and run inside this file make
and make install of Julia inside.

>/home/kostav/julia/bin/julia  /home/kostav/julia/contrib/julia-config.jl  --ldlibs

I actually don't have any bin folder inside my julia folder, so I can't do any of
the suggested modifications.
What could I do different in this case?

[Jeff Waller]

It returns: -Wl,-rpath,/home/kostav/julia/usr/lib -ljulia

Ok next test, what happens when you paste that into the almost working

step above (the one the works except for when you run it, it can't find libjulia?

make sure that -ljulia and -lLLVM3.7svn come after the -Wl,rpath directives

Related, what's in the directory /home/kostav/julia/usr/lib

 

>Did you do a make install of Julia itself and are running out of that

>installed directory or are you running out of where it compiled or possibly

>simply copied the directories? That script assumes the former and I think

>maybe you're doing the latter.

I downloaded the source code in folder (home/julia) and run inside this file make
and make install of Julia inside.

This should result in a directory with a hex-number associated, didn't it?

 

>/home/kostav/julia/bin/julia  /home/kostav/julia/contrib/julia-config.jl  --ldlibs

I actually don't have any bin folder inside my julia folder, so I can't do any of
the suggested modifications.
What could I do different in this case?

Where is it?  Use that instead.

[Kostas Tavlaridis-Gyparakis]

>Related, what's in the directory /home/kostav/julia/usr/lib

When I run the following command:

g++ -o test main.cpp -Wl,-rpath,/home/kostav/julia/usr/lib -I /home/kostav/julia/src -I /home/kostav/julia/usr/include -I /home/kostav/julia/src/support -L/home/kostav/julia/usr/lib -ljulia -lLLVM-3.7svn

And I also initialize julia inside c++ with the following command:

jl_init_with_image("/home/kostav/julia/usr/lib/julia", "sys.so");

It does work properly finally!

>Where is it?  Use that instead.

It's located at /home/kostav/julia/usr/bin/julia

Yet again when I upgrade the makefile accordingly and it does look like this:

CFLAGS   += $(shell /home/kostav/julia/usr/bin/julia  /home/kostav/julia/contrib/julia-config.jl --cflags)
CXXFLAGS += $(shell /home/kostav/julia/usr/bin/julia  /home/kostav/julia/contrib/julia-config.jl --cflags)
LDFLAGS  += $(shell /home/kostav/julia/usr/bin/julia  /home/kostav/julia/contrib/julia-config.jl --ldflags)
LDLIBS   += $(shell /home/kostav/julia/usr/bin/julia  /home/kostav/julia/contrib/julia-config.jl --ldlibs)
all: embed_example

I do receive the following error:

cc -DJULIA_INIT_DIR=\"/home/kostav/julia/usr/lib\" -I/home/kostav/julia/usr/include/julia  -L/home/kostav/julia/usr/lib  embed_example.c  -Wl,-rpath,/home/kostav/julia/usr/lib -ljulia -o embed_example
embed_example.c:1:19: fatal error: julia.h: No such file or directory
 #include <julia.h>
                   ^
compilation terminated.

[Scott Jones]

Your path for includes doesn't have julia/src, which is where julia.h lives.

[Kostas Tavlaridis-Gyparakis]

Ok, thanks a lot for all your help for the moment I stay with just using a makefile
and include the paths following the above example case that does work.

[Jeff Waller]

Your path for includes doesn't have julia/src, which is where julia.h lives.

This is in part true, but is not the whole story.  Yes it is true that julia.h is found

in src in the source directory tree, but the file julia.h as well as julia the executable,

libjulia, etc get copied into the "installed" directory structure when "make install"

is executed.  It's that structure that should be used (and optionally lifted/copied

out of the build and put somewhere else) and in that tree, the src directory does

not exist. Nor does it exist in any packaged version of Julia, so it would be

better to assume that structure as this whole approach can eventually be applied

to releases, nightly builds, etc.

I think what is going on is even though Kostas is performing a "make install" he's 

continuing to use the source directory structure instead of the installed

directory structure which is causing confusion -- for both julia-config and me.

> jl_init_with_image("/home/kostav/julia/usr/lib/julia", "sys.so");

This is unfortunate if required, it should not be required.

It's good that something is working now, however, can we try one more thing

Perform make install again, and note the directory (it will be something like julia-XXXXXXXX)

where XXXXXXXX is some hexadecimal number that all of this gets installed

to as described above, and copy that entire tree elsewhere (using cp -a),  you can change

the name if you want, it's not necessary to preserve the name of the directory.  Once that is done

change the PATH and/or use symbolic links if that's more convenient so that julia is

the julia in THAT directory not the one in the source tree.

Then revert back to the cut-and-paste Makefile version rather than this modified

version but keep the old one around somewhere just in case it does't work out

so you don't have to re-create it from scratch.  That way if this attempt ultimately

fails, you can as least have something going with what you have now by switching

PATH back to what it is now.