Julia2C initial release

[Hongbo Rong]

This is to announce the release of Julia2C, a source-to-source translator from Julia to C. This initial version converts basic Julia types and expressions into the corresponding C types and statements. It is open sourced at   https://github.com/IntelLabs/julia/tree/j2c. You can download and start playing with the code. 

Enjoy!

Hongbo

[Jey Kottalam]

Interesting. What are the sorts of usecases envisioned for this?

[Tim Holy]

Interesting! I'd be even happier about a translator that works in the opposite
direction. Have you given that any thought?

Best,
--Tim

[Hongbo Rong]

By translating Julia to C, we leverage the high-level abstractions (matrix, vector, ..), which are easier to analyze, and can potentially add the rich extensions of C (like openmp, tbb, ...). 

The tool may also extend Julia to new architectures where the only available tool chain is for C

[Hongbo Rong]

Translation from C to Julia might be harder. Some issues have to be addressed: (1) how to recognize and lift up the abstraction level of data structures and their operations? For example, how to raise a uint* to Vector object, and a loop of a[i]+b[i] to vectorA + vectorB? (2) Memory management: All alloc and free have to be replaced and hooked to garbage collector. If the C code uses customized memory allocators, memory pools, etc. it can cause problems. (3) Extensions like pragmas, embedded assembly, intrinsics, etc. could bring some implementation restrictions.

[Keith]

Maybe a way to get to a standalone executable?

[Tom Short]

Wow. That's a lot of work. It's very entwined in the core Julia code. Is there hope that it could be separated into a package?

Or, are you hoping to get it merged into Julia?

[Tony Kelman]

If I were the Mathworks, this would scare the crap out of me. Simulink Real-time Workshop and C code generation for embedded real-time control applications is a substantial industrial use case for Matlab/Simulink that hasn't really been replicated by any high-level open source language as of yet.

I'll have to look at this in more detail and try it out. Does this expand macros? Include code used from Julia packages? Does the generated C require any pieces of the libjulia runtime library?

[Hongbo Rong]

Yes, this can be a possible usage.

So far, the concept is: the user specifies 1 Julia function to be translated into native C; then in the Julia code generation phase, we recursively compile this function and all its direct and indirect callees into C; that is, the whole call graph is translated. Then a C compiler is invoked to translate the C code into a shared library, and the original call to the user Julia function is replaced by a call to this shared library.

If the user specifies the main function to be translated into C, he/she should get a standalone executable as you imagined.

[Hongbo Rong]

There are two main parts of the code:

(1) traverse the AST tree of a function. This is embedded into codegen.cpp and a few other files to minimize our coding efforts. But a (much) better way would be to make it a self-contained module.

(2) map Julia types and AST nodes to C types and statements. This is in j2c.cpp.

Since the code works at Julia code generation phase, it seems more appropriate to merge it into Julia, instead of as an external package.  

We hope that this initial version will give the community a starting point for Julia to C translation, and people can re-shape it to be better. 

On Wednesday, November 12, 2014 6:46:58 PM UTC-8, Tom Short wrote:

[Hongbo Rong]

All macros and included Julia code have been processed before J2C happens. The generated C is  standalone: a call graph rooted at a user-specified function is translated, including any Julia runtime function called. So it is a kind of whole-program translation. The C does not call back to libjulia during execution. It may call C runtime instead (for printf, for example).

[Jameson Nash]

Wow. That is quite a large change set in that commit.

It's perhaps worth pointing out that Julia has been emitting native executables for a little over a year now. This translator is different in that it adds an intermediate representation in C, rather than emitting the binaries directly as we do now.

Julia can be compiled for any computer which can be targeted by the LLVM/Clang compiler. It is theoretically already possible to compile Julia directly for an embedded application, although it is poorly exposed by the current frontend (libjulia would be a very small library in such a situation, due to the removal of the libLLVM libraries, so as to be nearly insignificant. Regardless, every programming language includes a similar such set of standard routines: libc, libgfortran, libpython, etc., so it isn't particularly meaningful or beneficial to remove it).

Since this is a source transform (codegen.ccp is also just a source transform in this respect), it could easily be implemented in a pure julia package (inference.jl is another quite similar source transform) just by recursively calling expand() on every function call and emitting an equivalent C file. I think you could even leverage libclang to pretty-print your C AST in that case. Note that a significant portion of Julia code cannot be emitted as a direct transform to C. You will need the jl_apply_generic function from the libjulia library to handle the general case of calling an unknown function (such as when using first-class functions or when calling map or when type inference is incomplete).

[Viral Shah]

Hi Hongbo

First of all - a big thanks to Intel for this contribution. Large organisations are not easy to convince on open source, and this is huge.

A question for everyone is if we should open have j2c as a PR? How should we integrate it? There is a fair amount of work to do before it is complete.

There is a case for embedded deployments where a certified C compiler is required to compile and this gives a way to deploy Julia in such cases if we can make the support first class.

[Tom Short]

I tried compiling j2c on Arch Linux. In the j2c branch, I get the
following error:

codegen.cpp:46:32: fatal error: llvm/ADT/OwningPtr.h: No such file or directory
#include "llvm/ADT/OwningPtr.h"
^

The master branch compiles fine. In both cases, I was using this
invocation (taken from the AUR script that compiles Julia):

make USE_SYSTEM_LLVM=1 USE_SYSTEM_LIBUNWIND=1
USE_SYSTEM_PCRE=1 USE_SYSTEM_LIBM=1 USE_SYSTEM_OPENLIBM=0
USE_SYSTEM_OPENSPECFUN=0 USE_SYSTEM_DSFMT=0 USE_SYSTEM_BLAS=1
USE_SYSTEM_LAPACK=1 USE_SYSTEM_FFTW=1 USE_SYSTEM_GMP=1
USE_SYSTEM_MPFR=1 USE_SYSTEM_ARPACK=1 USE_SYSTEM_SUITESPARSE=0
USE_SYSTEM_RMATH=0 USE_SYSTEM_LIBUV=0
USE_SYSTEM_UTF8PROC=0 USE_SYSTEM_MOJIBAKE=0 USE_INTEL_MKL=0
USE_BLAS64=0 USE_LLVM_SHLIB=0

[Hongbo Rong]

There is a condition before that include: 

#if defined(LLVM_VERSION_MAJOR) && LLVM_VERSION_MAJOR == 3 && LLVM_VERSION_MINOR >= 5

So it looks like the issue is related with the LLVM version used.

Note: the J2C branch is behind the main Julia branch in commits after April 10th, 2014. We have not updated Julia since then. So the compilation of the main Julia and J2C may have differences.

[Tom Short]

I got julia to compile using USE_SYSTEM_LLVM=0. Now, my problem is
that when I run `../julia j2c.jl`, it creates an empty out.cpp file in
../j2c/, and there's an error message saying it can't find function
j2c_sumOfThree.

[Hongbo Rong]

I cannot repro this error. Here is what I tried, with a fresh checkout, build, and test:

git clone -b j2c https://github.com/IntelLabs/julia.git

cd julia/

make debug

export JULIA_ROOT=/home/hrong1/j2c-release/julia

cd test

../julia j2c.jl

[Tom Short]

I tried it again, this time in a Mint Linux virtual machine. I got the
same issue: a zero-length out.cpp file.

[Hongbo Rong]

Can you show the complete error messages, if any?

[Tom Short]

****Matrix size: 1000*1000
ERROR: ccall: could not find function sumOfThree_ in library
/home/tshort/julia2c/j2c/libout.so.1.0
in j2c_sumOfThree at /home/tshort/julia2c/test/j2c.jl:65
in anonymous at no file:73
in include_from_node1 at loading.jl:128
while loading /home/tshort/julia2c/test/j2c.jl, in expression starting
on line 70

I tried pasting the code from j2c.jl. Here's the only thing that I was
that might indicate something. I'm not sure what the return value
means.

julia> offload(sumOfThree, (Int,))
2

[Hongbo Rong]

From the output, it seems that out.cpp is not successfully generated. If so,  libout.so.1.0 will contain a function called sumOfThree_.

To figure out why, try this: 

1. Make sure sumOfThree function is indeed processed by j2c. 

In codegen.cpp, line 3209, 

    bool prev_in_j2c = in_j2c;

    bool should_j2c_cur_func = J2C_FLAG_J2C(lam->j2cflag) || should_j2c(ctx.funcName.c_str());

    

Check if should_j2c_cur_func is true. If not, force it to be true, and see if out.cpp is generated. 

You can also dump out the function name to see the current function in code gen: 

        JL_PRINTF(JL_STDOUT, "******* emitfunc %s****\n", ctx.funcName.c_str());

2. Make sure an out.cpp is dumped.

In j2c.cpp, line 2133

for (size_t i = 0; i < functions.size(); i++) {

if (functions[i]->is_root_func) {

dump_entry_function(fs, functions[i]);

}

dump_func(fs, functions[i]);

}

Print or trace if functions.size > 0. In our case, it should be 1. 

[Sebastian Luther]

Hi,

I get the same error when running j2c.jl.

On Monday, November 17, 2014 4:27:16 AM UTC+1, Hongbo Rong wrote:

From the output, it seems that out.cpp is not successfully generated. If so,  libout.so.1.0 will contain a function called sumOfThree_.

To figure out why, try this: 

1. Make sure sumOfThree function is indeed processed by j2c. 

In codegen.cpp, line 3209, 

    bool prev_in_j2c = in_j2c;

    bool should_j2c_cur_func = J2C_FLAG_J2C(lam->j2cflag) || should_j2c(ctx.funcName.c_str());

    

Check if should_j2c_cur_func is true. If not, force it to be true, and see if out.cpp is generated. 

 The variable should_j2c_cur_func is always false for me. When setting it to true in gdb I get a message and a segfault later on. How do I move on from here?

(gdb) set should_j2c_cur_func=true
(gdb) c
Continuing.
warning: false
Create a data type without considering parameters
#<null>.Any
Unimplemented: new_sym
Unimplemented: emit jlcall: others

...

(gdb) set should_j2c_cur_func=true
(gdb) c
Continuing.
Unimplemented: new_sym
Unimplemented: emit jlcall: others
[New Thread 0x7ffff19ee700 (LWP 12407)]
[New Thread 0x7ffff11ed700 (LWP 12408)]
[New Thread 0x7ffff09ec700 (LWP 12409)]

Breakpoint 1, emit_function (lam=0x1395240, cstyle=false) at codegen.cpp:3211
3211        in_j2c = should_j2c_cur_func;
(gdb) set should_j2c_cur_func=true
(gdb) c
Continuing.
warning: false
Create a data type without considering parameters
Any...

Program received signal SIGSEGV, Segmentation fault.
__parse_one_specmb (format=0x7ffff773ca03 "%s", posn=posn@entry=0, spec=spec@entry=0x7fffff7ff010, max_ref_arg=0x7fffff7ffa40) at printf-parsemb.c:64

(gdb) bt
#0  __parse_one_specmb (format=0x7ffff773ca03 "%s", posn=posn@entry=0, spec=spec@entry=0x7fffff7ff010, max_ref_arg=0x7fffff7ffa40) at printf-parsemb.c:64
#1  0x00007ffff6761b7c in _IO_vfprintf_internal (s=s@entry=0x7fffff7fff10, format=<optimized out>, format@entry=0x7ffff773c9fc "struct %s", ap=ap@entry=0x7fffff800038) at vfprintf.c:1777
#2  0x00007ffff678745b in __IO_vsprintf (string=0x7ffff7cf8f5e <c_code+500510> "struct ", format=0x7ffff773c9fc "struct %s", args=args@entry=0x7fffff800038) at iovsprintf.c:42
#3  0x00007ffff676a4b7 in __sprintf (s=<optimized out>, format=<optimized out>) at sprintf.c:32
#4  0x00007ffff6de7175 in create_c_struct_type (v=0x64bc50) at j2c.cpp:431
#5  0x00007ffff6de78e4 in create_c_type (v=0x64bc50) at j2c.cpp:539
#6  0x00007ffff6de71f1 in create_c_struct_type (v=0x64f340) at j2c.cpp:440
#7  0x00007ffff6de78e4 in create_c_type (v=0x64f340) at j2c.cpp:539
#8  0x00007ffff6de71f1 in create_c_struct_type (v=0x64bc50) at j2c.cpp:440
#9  0x00007ffff6de78e4 in create_c_type (v=0x64bc50) at j2c.cpp:539
#10 0x00007ffff6de71f1 in create_c_struct_type (v=0x64f340) at j2c.cpp:440
#11 0x00007ffff6de78e4 in create_c_type (v=0x64f340) at j2c.cpp:539
#12 0x00007ffff6de71f1 in create_c_struct_type (v=0x64bc50) at j2c.cpp:440
#13 0x00007ffff6de78e4 in create_c_type (v=0x64bc50) at j2c.cpp:539
... (looks like infinite recursion)

[Hongbo Rong]

This is a good step toward figuring out the issue. In setting that var in gdb, you are turning on J2C translation for all the functions that get into emit_function(). That will expose some unimplemented features of J2C. In j2c.jl, J2C is on explicitly only for function sumofThree() and all its callees.

 At this place

    bool should_j2c_cur_func = J2C_FLAG_J2C(lam->j2cflag) || should_j2c(ctx.funcName.c_str());

 Can you add 

      JL_PRINTF(JL_STDOUT, “j2c flag is %d for function %s\n", J2C_FLAG_J2C(lam->j2cflag), ctx.funcName.c_str())

make again, and run again? 

Let’s see if J2C_FLAG_J2C(lam->j2cflag) is always false, or should_j2c(ctx.funcName.c_str()) is always false. 

Thanks,

Hongbo

[Sebastian Luther]

Am Mittwoch, 3. Dezember 2014 00:58:37 UTC+1 schrieb Hongbo Rong:

This is a good step toward figuring out the issue. In setting that var in gdb, you are turning on J2C translation for all the functions that get into emit_function(). That will expose some unimplemented features of J2C. In j2c.jl, J2C is on explicitly only for function sumofThree() and all its callees.

 At this place

    bool should_j2c_cur_func = J2C_FLAG_J2C(lam->j2cflag) || should_j2c(ctx.funcName.c_str());

 Can you add 

      JL_PRINTF(JL_STDOUT, “j2c flag is %d for function %s\n", J2C_FLAG_J2C(lam->j2cflag), ctx.funcName.c_str())

make again, and run again? 

Let’s see if J2C_FLAG_J2C(lam->j2cflag) is always false, or should_j2c(ctx.funcName.c_str()) is always false. 

 Both are always false.

[Hongbo Rong]

Can you confirm that with that JL_PRINTF, the output contains something like  "j2c flag is X for function sumOfThree"?

It is wrong that J2C_FLAG_J2C(lam->j2cflag) is always false. You should expect the X above is 1 for function sumOfThree

[Sebastian Luther]

Am Donnerstag, 4. Dezember 2014 20:44:29 UTC+1 schrieb Hongbo Rong:

Can you confirm that with that JL_PRINTF, the output contains something like  "j2c flag is X for function sumOfThree"?

It is wrong that J2C_FLAG_J2C(lam->j2cflag) is always false. You should expect the X above is 1 for function sumOfThree

 You are right, I didn't look back far enough in the list. There are two functions which don't give 0 0:

j2c flag is 1 0 for function sumOfThree
j2c flag is 0 1 for function +

I used this line:
JL_PRINTF(JL_STDOUT, "j2c flag is %d %d for function %s\n", J2C_FLAG_J2C(lam->j2cflag), should_j2c(ctx.funcName.c_str()), ctx.funcName.c_str());

[Hongbo Rong]

great. this is the right output. So j2c is indeed enabled for sumofThree. 

Now set a breakpoint at src/j2c.cpp line 2130 in function j2c_dump_all_funcs() and step through. This is the final phase to dump out the cpp file. It first dumps to a file called temporary. After line 2145, you should be able to see if that temporary (a text file) is there or empty.

[Sebastian Luther]

Am Donnerstag, 4. Dezember 2014 22:00:20 UTC+1 schrieb Hongbo Rong:

great. this is the right output. So j2c is indeed enabled for sumofThree. 

Now set a breakpoint at src/j2c.cpp line 2130 in function j2c_dump_all_funcs() and step through. This is the final phase to dump out the cpp file. It first dumps to a file called temporary. After line 2145, you should be able to see if that temporary (a text file) is there or empty.

I was able to find the problem. Your code relies on the tool bcpp. After installing it the error is gone.

Now  to the next problem. The converted code produces wrong results for N>=2.
For N=2 it prints 528.0, but it should be 1188.0.

[Hongbo Rong]

Thanks for finding out the issue. I will fix the README, and look at the N=2 results.

[Hongbo Rong]

I debugged out.cpp with N=2. It is correct. The issue seems to be in j2c.jl, function j2c_sumOfThree(), which somehow, makes the size of the array returned from out.cpp to be 2, instead of 2*2. So only the first 2 elements are summed up. That is why the result is wrong. I asked Paul to take a look at it.

[Viral Shah]

Perhaps a check in the code for existence of the tool, that produces the appropriate error message, may help.

-viral

[Hongbo Rong]

good idea. I fixed the error code returning from system(). If the tool does not exist, the error code should be non-zero, and we will simply go without beautifying the c output.

[Hongbo Rong]

The bug is fixed and checked in.  Here is the info from Paul:

There is a typo at line 64 of test/j2c.jl, and instead of zeros(Cint, 1), it should be zeros(Cint, 2).

The 2 here means it expects the return result to be a 2D array, and the length of each dimension will be stored in this ret_out_dims array.

[Sebastian Luther]

Am Samstag, 6. Dezember 2014 08:15:13 UTC+1 schrieb Hongbo Rong:

The bug is fixed and checked in.  Here is the info from Paul:

 Works now. Thanks. I have some questions though.

What is the current state of this implentation? Is there an ongoing development? I tried some toy examples and found the user interface rather restrictive (hard coded library path, no automatic wrapper generation for the created function) and it's rather easy to break. Have you considered expanding the test scenarios? Here are two functions where the conversion results in compile errors (from gcc).

g() = 1

function f(x)
    if x > 0
        return 1
    else
        return -1
    end
end

[Hongbo Rong]

The current release should handle common cases. I added "current status" to j2c/README. Please pull to take a look. We provide the framework as a good starting point, and expect the open source community to develop this tool to its full maturity.

The two examples you show are not fully typed and thus cannot be converted to C, which is a statically typed language. I suggest you add those two as new test cases, and add j2c handling for them as an exercise, if needed. I would be happy to help with any question.

The automatic generation of a wrapper: I ask Anderson Todd to answer this question. 

[Hongbo Rong]

Paul Liu will check in an automatic generator of a wrapper next week.

[webus...@gmail.com]

Can this be used to call Julia functions from golang?

On Wednesday, November 12, 2014 2:22:51 PM UTC-5, Hongbo Rong wrote:

This is to announce the release of Julia2C, a source-to-source translator from Julia to C. This initial version converts basic Julia types and expressions into the corresponding C types and statements. It is open sourced at   https://github.com/IntelLabs/julia/tree/j2c. You can download and start playing with the code. 

Enjoy!

Hongbo

[Hongbo Rong]

I noticed this sentence from https://golang.org/doc/faq:

  "The cgo program provides the mechanism for a “foreign function interface” to allow safe calling of C libraries from Go code."

 So you may call a Julia function's corresponding J2C version (C function) for that purpose.

[webus...@gmail.com]

Yes. C functions can be called from golang quite easily with cgo. However, Can I take a slice in golang (or an array) and pass it to a Julia function? Is the representation similar? What about multi dimensional arrays? How would they be passed in from golang...

[John Myles White]

Julia represents numerical arrays the same way that C or Fortran would -- with the added constraint that arrays are column-major. If Go lets you pass around arbitrary pointers to arbitrary code, I think things will work -- assuming that Go lays out arrays the way that C and Fortran would.

 -- John

[webus...@gmail.com]

It's quite easy to pass the underlying data pointer in a slice to a c function: (see example below)

http://michael.stapelberg.de/Artikel/cgo_uint32/

I guess I was asking the following:

1. How would you handle multi dimensional julia areas

2. Would all the underlying c math libraries that Julia uses be also available in the code that Julia2c uses. In other words, what things in Julia are not going to be supported in Julia2C?

3. Is this (in this form or some other form) going to be supported by the core devs... 

[Isaiah Norton]

Just in case it is not clear: julia2C is not strictly required to call Julia from Go. Several people are using the Julia C embedding API directly already, and Julia can also generate C-compatible function pointers using "cfunction" (see e.g.: http://julialang.org/blog/2013/05/callback/)

1. How would you handle multi dimensional julia areas

I'm not familiar with Go's internal representation, but from a glance at your link it sounds like you might be able to convert array dimensions to Go "SliceHeader" objects to take care of striding for you (as JMW said: if the layouts are compatible).

2. Would all the underlying c math libraries that Julia uses be also available in the code

If the linker paths are set correctly, then that should be possible.

that Julia2c uses. In other words, what things in Julia are not going to be supported in Julia2C?

3. Is this (in this form or some other form) going to be supported by the core devs...

The Julia C API and "cfunction" are fully supported. Others can speak more authoritatively on julia2C.

[Isaiah Norton]

not strictly required to call Julia from Go.

(to be clear: I'm not sure if anyone has done this from Go yet, but if you search the lists there are examples available for c, python luajit, erlang, and possibly others. the best example right now is probably the embedding example in base, and pyjulia)

[Hongbo Rong]

Paul Liu just checked in an automatic generator of a wrapper. Please take a look at the example test/j2c.jl. It should be pretty clear.  Sorry for the delay.

[Hongbo Rong]

A short tech memo (Julia2C.pdf) is checked in.