Julia C++ FFI code release

[Keno Fischer]

I promised at JuliaCon that I'd release the code for the Julia C++
interface, as soon
as I got it running on top of staged functions, which is today so here
it is. I put the
code up at https://github.com/Keno/CXX.jl. For those of you who
weren't at JuliaCon,
it allows you to write julia code like:

function code_graph(f,args)
v = @cxx std::string()
os = @cxx llvm::raw_string_ostream(v)
graphf = code_llvmf(f,args)
@cxx llvm::WriteGraph(os,graphf)
@cxx os->flush()
bytestring((@cxx v->data()), (@cxx v->length()))
end

and have it seamlessly call the appropriate C++ function without the
need to write wrappers of any kind. This is achieved by using clang to
parse headers and do the codegen where necessary.

Right now it's still a little hard to build, which I will fix over the
next few days, but I know people were waiting for this, so I want to
just put it out there and see what happens.
If you're expecting it to just work, I'd recommend holding off for a
couple of weeks, while the package forms more fully, but if you want
to hack on it, here's what you need to do
to your copy of julia:

- In your Make.user:
BUILD_LLVM_CLANG=1
BUILD_LLDB=1

Ideally also set LLVM_ASSERTIONS=1. It will make your life better

- Merge the following branches:
- kf/llvmcall
- kf/staged

- Apply the following patch in codegen.cpp:

-static LLVMContext &jl_LLVMContext = getGlobalContext();
+DLLEXPORT LLVMContext &jl_LLVMContext = getGlobalContext();
static IRBuilder<> builder(getGlobalContext());
static bool nested_compile=false;
-static ExecutionEngine *jl_ExecutionEngine;
+DLLEXPORT ExecutionEngine *jl_ExecutionEngine;

PLEASE NOTE: I have only tested this version on OS X so far, so it
probably won't work anywhere else. e.g., I'm sure that the expmap will
need to be adjusted on Linux and there's probably other things that
I'm forgetting.

In the mean time, if that hasn't scared you off yet, feel free to have
a look at it and file issues. I'll be continuously improving this over
the next few days and weeks.

Happy hacking,
Keno

[Tim Holy]

This is really unbelievable. From everything I've seen and heard, this is an
unprecedented capability. It will open many doors.

My own laptop naturally follows master, but for the lab machines I have
periodically tried to use a stable release. I'm pleased to see one more reason
why 0.4-pre is going to offer so many goodies that I won't be able to resist
using it :-).

--Tim

[Stefan Karpinski]

This is really unprecedented – the only other systems that can do this kind of thing are C++ debuggers (and they often don't do it well). This could make Julia one of the best interactive environments for exploring C and C++ libraries. The ability to dynamically instantiate templates is absolutely amazing.

One thought I have is that while it's ok to have a C++ parser in  memory when using this interactively, do you have any thoughts on how to avoid that once someone has developed an interface to a C++ library? At that point, it would be nice to pre-generate the necessary bindings or something like that.

[Keno Fischer]

Yeah, I think it's quite possible that this will just fall out of
precompiling packages if done correctly. The other nice thing about
C++ is that as long as you don't arbitrarily instantiate templates
(i.e. instantiate a template based on a julia value), then the set of
function you can call is fixed, so you can go through your program and
precompile all the functions that call C++.

[Tobi]

I have argued on the mailing list that Qt will be hard to wrap and now comes this...

I have not yet tried but it sounds awesome. Especially if header parsing can be hidden in a precompiled package this is huge.

Who is the first to wrap a simple Qt program and making screenshot ;-)

Tobi

[Isaiah Norton]

As you know I'm kind of giddy about this. While my computer whirs away with llvm+clang, I'll just join the chorus and say:

[Jake Bolewski]

Glad you decided to release this earlier than later Keno, :-). the refactored version turned out really nice and with comments (thank you) it is easy to follow. I still can't get over that it is under 2000 lines of code to implement something crazy like this.

In my opinion, this is one of those features that if we really nail has the potential to massively drive adoption. As Stephan said no one else has this, and looking at how this is implemented, this is not something that you simply tack on. staged functions are going to be really huge and I think they represent the benefit of exposing parts of the compiler with a well defined interface. Looking forward to more instances of those, and to following to progress of this over the next couple of months.

Jake

[Keno Fischer]

Me!

using CXX

const qtlibdir =
"/Users/kfischer/Projects/qt-everywhere-opensource-src-5.3.1/qtbase/~/usr/lib/"
const QtCore = joinpath(qtlibdir,"QtCore.framework/")
const QtWidgets = joinpath(qtlibdir,"QtWidgets.framework/")

addHeaderDir(qtlibdir; isFramework = true, kind = C_System)

dlopen(joinpath(QtCore,"QtCore_debug"))
dlopen(joinpath(QtWidgets,"QtWidgets_debug"))

cxxinclude("QApplication")
cxxinclude("QMessageBox")


x = Ptr{Uint8}[pointer("julia"),C_NULL]
app = @cxx QApplication(int32(0),pointer(x))

@cxx QMessageBox::about(cast(C_NULL,pcpp"QWidget"), pointer("Hello
World"), pointer("This is a QMessageBox!"))

@cxx app->exec()

[Tobi]

Woohoo, impressive. Despite my enthusiasm on Gtk.jl this opens a new door for GUI development in Julia. And the cross-platform capabilities of Qt are really appealing if I am honest to myself.

[Tim Holy]

Nice!

While I lobbied for Qt from the start (before it was practical), having
invested a lot in a great Gtk package and yet having so much trouble with the
binary dependency thing makes me cautious. I'd say first and foremost it would
be worth seeing whether distribution of Gtk.jl can be fixed without going to a
month's worth of heroics. But if the Gtk binaries problem looks too hard to be
worth the time investment, even then I wouldn't dream of working on a Qt
wrapper until some basic tests have been done: post an absolutely minimal Qt
wrapper (e.g., Keno's demo code plus a `deps/build.jl` that hopes to install
Qt for people) and find out how many people have trouble installing it. Make
sure it works for basically everyone, on every platform, before going to any
effort to actually write the wrapper.

--Tim

[jtravs]

Hey, this looks amazing!

It is not entirely true that no one else has this, Cling from CERN has an interactive C++ interpreter based on CLANG: http://root.cern.ch/drupal/content/cling

Maybe some useful ideas can be taken from them? But I agree that no other language has this. Cling is *only* interactive C++, wheres this allows us to mainly use Julia and dip into C++ where necessary.

As I have a huge and complex C++ code base, I was struggling to reconcile how to incrementally move to Julia. SWIG wasn't an option and writing c-shims wasn't going to cut it, so I had nearly given up. This gives me a whole new strategy for C++ to Julia migration.

Also, Qt GUI would really settle everything! I'm not at all a fan of GTK, so this (and I think for many others) could be a huge win just in itself.

Cheers!

John

[Tobi]

Good thoughts Tim. I feel very similar. I really like the Gtk API and its Gtk.jl wrapping and while I have it running on all three platforms there are some quirks here and there on OSX and Windows. Its nice to have a new option on the table, even if its just to demo the potential of Kenos C++ FFI.

[Chris Foster]

Oh wow, this is really extremely impressive. Just a few years ago
this would have fallen into the category of "basically impossible".

[Keno Fischer]

Yes, I know about Cling. It's a very impressive project, albeit with
different goals (turning C++ into a high level dynamic language vs
using C++ from such a language).

[Stefan Karpinski]

On Mon, Jul 21, 2014 at 2:54 AM, jtravs <jtr...@gmail.com> wrote:

As I have a huge and complex C++ code base, I was struggling to reconcile how to incrementally move to Julia. SWIG wasn't an option and writing c-shims wasn't going to cut it, so I had nearly given up. This gives me a whole new strategy for C++ to Julia migration.

Excellent – you can be one of the first guinea pigs for the Cxx package ;-)

[Sebastien Binet]

It's not completely correct.
Cling is also used by the python bindings (pyroot) to access the c++ api.

-s

sent from my droid

[Stefan Karpinski]

On Mon, Jul 21, 2014 at 2:54 AM, jtravs <jtr...@gmail.com> wrote:

It is not entirely true that no one else has this, Cling from CERN has an interactive C++ interpreter based on CLANG: http://root.cern.ch/drupal/content/cling

Maybe some useful ideas can be taken from them? But I agree that no other language has this. Cling is *only* interactive C++, wheres this allows us to mainly use Julia and dip into C++ where necessary.

Good point. Does cling do template instantiation on the fly for you?

[Sebastien Binet]

Yes.

-s

[Keno Fischer]

Oh, I didn't know that. I thought pyroot still used wrappers. Do you
happen to have some examples I could look at where this is done?

[Sebastien Binet]

(Resend from the correct address)

sent from my droid

On Jul 21, 2014 8:02 PM, "Sebastien Binet" <bi...@cern.ch> wrote:

Pyroot always used what is called 'dictionaries' in the HEP community - these dictionaries provide reflection capabilities to C++ code.

Before ROOT-6, we relied on gccxml+reflex to generate them.
Now, with ROOT-6 (and CLing) the introspection code is talking to cling directly.

I'll post some pointers when in front of a real keyboard.

-s

sent from my droid

[Jameson Nash]

I think Qt support is an excellent development. I'm excited to see where that will lead. I'll just point out that there remains a lot of work between showing the ability to call external code (e.g. ccall) and having a user-friendly wrapper (e.g. Gtk.jl) that simultaneously feels familiar to a Julia and Qt user.

In the meantime, Gtk.jl master now has been updated to use < 0.1% CPU time when idle (as opposed to the latest tagged Gtk.jl release and Tk.jl which use about 3% of one core of my i7 processor when doing nothing).

I'm hoping to make various advances in the packaging of Gtk binaries for 0.4. This should hopefully also help other binary sources such as Python. The main sticking point is that one of the more attractive features of GLib is that it is a framework, not just a standalone library. This means there are many extensions for it (including Gtk) that users may want to add (http://lazka.github.io/pgi-docs/).

[Tobi]

Yes, I absolutely agree with you Jameson. I am internally using Gtk.jl for some work in my company and the Julia interface of Gtk is just fun to use. But also the internal Gtk.jl code is very clean and having contributed some code I can say that Gtk and Julia fit quite well together. That Cairo is already in intensive use in the Julia ecosystem comes ontop of this. So I am really hoping that packaging of Gtk binaries will get easier. Getting nice theming on Windows and OSX plus recent versions (3.12) on Windows would be a dream come true.

For me, the end goal is to have a high quality cross-platform GUI library one Pkg.add away.

[Tobi]

And thanks so much for all the work that you put into Gtk.jl Jameson. It is really appreciated. Your recent Winston fix (imagesc slow) turned my Gtk app from "not usable" to "usable"...

[Sebastien Binet]

On Mon, Jul 21, 2014 at 8:04 PM, Sebastien Binet <seb....@gmail.com> wrote:
> (Resend from the correct address)
>
> sent from my droid
>
> On Jul 21, 2014 8:02 PM, "Sebastien Binet" <bi...@cern.ch> wrote:
>>
>> Pyroot always used what is called 'dictionaries' in the HEP community -
>> these dictionaries provide reflection capabilities to C++ code.
>>
>> Before ROOT-6, we relied on gccxml+reflex to generate them.
>> Now, with ROOT-6 (and CLing) the introspection code is talking to cling
>> directly.
>>
>> I'll post some pointers when in front of a real keyboard.

here is the entry point to generate python classes from C++ ones:
https://github.com/root-mirror/root/blob/master/bindings/pyroot/src/TPyClassGenerator.cxx

the ROOT API for (C++) classes:
https://github.com/root-mirror/root/blob/master/core/meta/inc/TClass.h

which relies on the ROOT interpreter API:
https://github.com/root-mirror/root/blob/master/core/meta/inc/TInterpreter.h

with the concrete implementation based on CLing:
https://github.com/root-mirror/root/blob/master/core/meta/src/TCling.h
https://github.com/root-mirror/root/blob/master/core/meta/src/TCling.cxx

"enjoy."
-s

[Tim Holy]

Sounds great, Jameson. I'd really like to see 0.4 ship with a co-installed
graphical toolkit by default---until that happens, I am a little skeptical
that graphics can really flourish the way it should.

From my perspective, getting the binaries working on everyone's machine is
_the_ make-or-break issue. Since I use Linux exclusively and it's already
working on Linux, it's not something I can practically help with,
unfortunately. (Sometime I might go back to trying to use Julia on one our
data-acquisition systems running Windows, in which case that will change.)

--Tim

[Martin DeMello]

The lablqt (ocaml bindings for qt) developer decided to give up on the bindings route and use qml instead. http://kakadu.github.io/lablqt/

That seems like it would be an easier option to get up and running, though with the tradeoff that you can't define new widgets in julia.

martin

[Keno Fischer]

First of all let me say that all of this so far is a proof of concept
for the C++ FFI to make sure I cover common use cases that people
might encounter. If at some point this should evolve into something
more than that, I have to say I'm not interested at all in writing
bindings. What I'm interested in is to make the C++ FFI sufficiently
general to be able to express anything you want just using the C++
API. At most I can see a small couple-hundred line library that
integrates the Qt event loop and perhaps other things of that sort.

[Tobi]

It might be different for Qt but in my opinion what makes Gtk.jl so convenient is that it
is not just a low level wrapper which_would_lead_to_very_long_function_names but that
it tries to fit into the Julia as much as possible.
Just few examples:

- Wherever possible, the usual Julia methods are used (length, push!, ...)
- For dispatching this kind of requires to map the C types to Julia types that hold a handle to the C types.
- Constructors with keyword arguments make the construction of various widgets very convenient
- The signalling mechanism also fits very well into the Julia syntax

But there are of course also good reasons to not integrate so tightly. C++ has function overloading and
polymorphism so that there is not such a pressing need so make the syntax more terse.

[Jameson Nash]

> Getting nice theming on Windows and OSX plus recent versions (3.12) on Windows would be a dream come true

That part is easy.

WinRPM uses the Opensuse Build Service which works much like a fully integrated github/travis setup:

https://build.opensuse.org/request/show/242556

I submitted a pull request last night to update Gtk to 3.12 and it was accepted this morning.

I will continue to encourage all Windows developers to use this service rather than packaging their own binaries. This centralizes and distributes the work of maintaining and updating binaries. You also easily make your own home project area in OBS and add the url to your WinRPM sources.list file, fully integrating your work with the rest of their package ecosystem. (ps. big plus: you don't even need windows, since it can all be done online :)

[Miles Lubin]

How difficult is it to add new packages? Has anyone here contributed a package specifically to use it within julia?

[Jameson Nash]

As I said it's rather like github (e.g. with revision control, forks, an API for local access, and free): you add a new package by creating an account and writing a .spec file that lists the dependencies and build steps. After some delay, their build engine packages up your code. Then you take the repository url it gives you and stick it in your WinRPM/sources.list file so that WinRPM can see your shiny new binaries. (I imagine it is also a lot like Ubuntu's https://launchpad.net/, but I have no personal experience in that area)

[Tony Kelman]

We're way off-topic, but anyway...

Going through OBS seems much slower and more roundabout than just doing it myself, for anything that doesn't already exist there as a cross-compiled package. I tried using FFTW from WinRPM many months back, but it didn't have Julia patches and probably wasn't configured the way we need it to be. Similar with PCRE, GMP, MPFR, we could try migrating to the OpenSUSE cross-compiled binaries for those as well. Or putting together new packages for things that aren't there yet, like LLVM, OpenBLAS, suitesparse, arpack, LibGit2, various JuliaOpt packages, etc.

Having to edit sources.list, and the recent connectivity/mirroring issues making WinRPM unusable for a lot of people, give me pause here. I do like the prospect for automation though, if this can be made to work reliably and easily then it is a better idea to use existing infrastructure than try to come up with our own.

[Jameson Nash]

The hard part isn’t getting the binary to someone’s computer. The problem is getting exactly one copy of a binary, and making sure it is the correct version, built against the correct libc, and plays nice with the other binaries that want to depend upon it.

I realize that for the typical package developer, running ./configure && make && make install DISTDIR=`pwd`/install && tar -czf `pwd`/install is both very easy and very familiar.

Unfortunately, it is bad for the Windows package ecosystem.

Currently, Mac has a very challenging packaging system because there are so many incompatible ways of getting a binary onto your system (self-compile, Homebrew, Homebrew.jl, MacPorts, Fink, Apple) that it is very difficult to put together a cohesive set of binaries that have any sort of interesting dependency relationship.

Windows currently doesn’t have very many sources of binaries, so we have a chance (to our annoyance or benefit, depending on your view on this) to make it anything that we want. Accordingly, I’ve tried to make WinRPM the defacto package manager for managing binary dependencies in a sensible manner. It was intended to be reasonably extensible too, so if you wanted to switch out the backend for something different, that isn’t much of a problem. I started with using the RPMmd format and OpenSUSE OBS, because they had already done a lot of the hard work of writing the package descriptions for over 300 libraries (and it included Gtk), and they provide an open build service, which meant anyone could become involved. (Plus, it also meant that I could distribute the work of keeping the binaries up-to-date, independent of the state of the Julia package)

It’s true that Julia duplicates a few binaries that could be downloaded from OpenSUSE, but then you start to run into a chicken-and-egg problem where you you need Julia built and running to download the packages to build Julia. make win-extras is essentially there to help with that bootstrapping problem. On OS X, we had seen (and fixed) some issues with the packaged libpcre conflicting with the built-in libpcre, but in general, most libraries don’t maintain global shared state, so it isn’t too much of a problem to have multiple copies loaded.

Having to edit sources.list, and the recent connectivity/mirroring issues

We’ve seen similar issues with archive.org binary hosting, so these sorts of problems aren’t entirely unique to WinRPM. However, if everyone is using a unified platform, then the user only has to learn and maintain one system. (and there are also hopefully more developers & interest in maintaining and improving that core system, rather than lots of “just sufficient” solutions in each individual package).

Regards,
Jameson

​

[Bob Nnamtrop]

I wonder if wrapping the Ceres Solver (http://ceres-solver.org/index.html) would be a good project to show off this package :-)

Bob

[Iain Dunning]

Indeed, lots of solvers with C++ only interfaces that I'm looking forward to having access too!

[Michel Lesoinne]

This development is quite impressive. Could someone give a little more details on how to try it? I am on Mac OS X

I have the julia git and it seems kf/llvmcall is already merge. Merging kf/staged created a conflict but it seems to be only a test script, so I ignored it.

The patch has already been done as well.

Is this correct so far?

Now where I am stuck is with Cxx.jl. I have no idea where it should be located relative to the julia directory. I defined JULIA_HOME so that it would find deps/Versions.make and Make.inc

After that, make failed with it included clang/AST/ASTContext.h

It found old versions of that file in my /usr/local/include. I doubt this is the version it expects though.

Is there a more detailed set of instruction to try this great feature?

As an additional question, does this system take care of calling destructors correctly? How do the created objects interact with the Julia GC system?

[Alex]

Hi Michel,

I am not sure if it is helpful for you, but there is a (longish) issue here for installing on OSX: https://github.com/Keno/Cxx.jl/issues/4

Some of it is obsolete though (as you have observed).

Regarding Cxx.jl: you can just use

Pkg.clone("https://github.com/Keno/Cxx.jl")

which will put it in your .julia/ directory. Then you can build it as usual

Pkg.build("CXX")

Best,

Alex.

[Keno Fischer]

After merging kf/staged,

Here's my current Make.user file:
LLVM_VER=svn
LLVM_ASSERTIONS=1
#LLVM_DEBUG=1
BUILD_LLVM_CLANG=1
BUILD_LLDB=1
LLDB_VER = master
USE_LLVM_SHLIB=1
LLDB_DISABLE_PYTHON=1

With that and
Pkg.clone("https://github.com/Keno/Cxx.jl")
Pkg.build("CXX")

I don't think there's anything else you need to do.

[Michel Lesoinne]

With LLVM_VER=svn in Make.user, make complains about a Circular llvm-svn <- llvm-svn dependency.

If I comment that, everything goes fine in compiling julia but then when doing Pkg.build("CXX") I get:

../src/bootstrap.cpp:16:10: fatal error: 'clang/Sema/ScopeInfo.h' file not found

So should I come back to LLVM_VER=svn and change something else?

[Keno Fischer]

That file definitely exists:
https://github.com/llvm-mirror/clang/tree/master/include/clang/Sema

Can you go into deps/llvm-svn/tools/clang and do `git status` and `git
log` to make sure your clang is up to date? If that directory doesn't
exist clang may not have been built for some reason. The circular
dependency is odd, I'll have a look.

[Keno Fischer]

Can you show exactly what the circular dependency was. I didn't think
there was an llvm-svn target.

[Michel Lesoinne]

I had done a parallel make, but I restarted from scratch without parallel make and here is what the error looks like:

clang: warning: argument unused during compilation: '--param max-inline-insns-single=1800'

make[2]: Circular llvm-svn <- llvm-svn dependency dropped.

make[2]: Circular /build_Release/Release/lib/libLLVMJIT.a <- llvm-svn dependency dropped.

make[2]: Circular /build_Release/config.status <- llvm-svn dependency dropped.

make[2]: Circular /configure <- llvm-svn dependency dropped.

make[2]: *** [.src.tar.gz] Error 1

make[1]: *** [julia-release] Error 2

make: *** [release] Error 2

There is no directory deps/llvm-svn/

This is all from starting with a fresh git clone merging staged, and just creating the Make.user with the same content you gave earlier.

[Keno Fischer]

Did you accidentally introduce some sort of newline in you LLVM_VER?

On Sun, Aug 24, 2014 at 3:20 AM, Michel Lesoinne

[Michel Lesoinne]

No accidental newline. However, after you asked, I realized the cut and paste of your Make.user had put a space at the end of each line.

Removing the spaces seems to fix the issue now the llvm-svn is created and compiled.

So a warning to others who may want to do this, make sure your cut and paste does not mess up the end of line with a space.

I do get another error however, but I presume I can fudge it. There is a similar issue in the MacOS version of the file

julia/deps/llvm-svn/tools/lldb/source/Host/posix/HostInfoPosix.cpp:184:24: error: use of undeclared identifier 'PY_MAJOR_VERSION'

After all this, I got to Pkg.build("CXX")  and now got a new error:

julia> Pkg.build("CXX")
INFO: Building CXX
Tuning for julia installation at: /Users/michel/Tools/3/julia/usr/bin
    CC /Users/michel/.julia/v0.4/Cxx/deps/build/bootstrap.o
    LINK /Users/michel/.julia/v0.4/Cxx/deps/usr/lib/libcxxffi.dylib
ld: library not found for -llldbInterpreter
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [usr/lib/libcxxffi.dylib] Error 1

[Keno Fischer]

I have a patch pending to lldb to fix the python issue. Let me see
about the other one.

On Mon, Aug 25, 2014 at 5:11 AM, Michel Lesoinne

[Kevin Squire]

One thing to be careful of: it looks like CXX.jl was renamed to Cxx.jl. On windows or OSX without case sensitive filesystems, this may not matter, but linux is case sensitive. 

Cheers, Kevin 

[Scott Jones]

If you are on Mac OS X, like me, and go back and forth with Unix/Linux, I just make a sparseimage filesystem (encrypted) to hold my work, and set it to case sensitive... have it mounted at boot... sometimes it's a pain having to type the correct case, but it makes like easier with the Linux VMs sharing the filesystem.

From this discussion, I just need to set up my Make.user file like yours, Keno, to start using LLVM-svn?

Thanks!