Returning julia objects with ccall

[Carlos Becker]

Hello everyone, 

I just got started with Julia, and I wanted to try to wrap a C/C++ library to Julia to check whether it would work out for my purposes.

I tried out many ways of passing arrays and other objects from C back to Julia.

So far it seems that it takes a lot of extra code if I want to return, for example, a simple double-array or an array of types (eg structs).

Then I thought that I could call the Julia API from the ccalled binary, to allocate an array and return it to julia,

then use unsafe_pointer_to_objref() and get a neat Julia object directly.

You can see a very simple example here https://gist.github.com/anonymous/8888647

This would simplify _significantly_ a lot of code from the C side, at least with what I am working right now.

Now, my question is: is it safe to call functions such as jl_alloc_array_1d() from the C binary?

would this be a problem in some situations?

I understand that it may mess memory up if those functions are called outside the main thread, but I would certainly not do that.

Thanks in advance,

Carlos

[Jeff Bezanson]

This is a totally reasonable thing to do. You just have to be careful
that objects are rooted across other object allocations; passing julia
objects around as Ptr{Void} hides them from the GC.

If "Any" is used as the return type of the ccall, the result will be
treated as a julia reference and you can skip
unsafe_pointer_to_objref.

A variant of this is to allocate the array in julia, and pass it to
the C function to be filled in (ccall will effectively call
jl_array_ptr for you to pass the array to C).

[Carlos Becker]

Hi Jeff, thanks for the quick reply.

If "Any" is used as the return type of the ccall, the result will be
treated as a julia reference and you can skip
unsafe_pointer_to_objref.

If returning "Any", would the GC take care of it?

 

A variant of this is to allocate the array in julia, and pass it to
the C function to be filled in (ccall will effectively call
jl_array_ptr for you to pass the array to C).

Right, but if the final size is not known before ccall(), then it may not be as easy as creating it from C.

Now, I am also looking for something a bit more involved.

For example, I need to return what in C is an array of structs (std::vector<struct type>).

Do you see some problems when creating and filling a specific Julia type, to add several of them

into a Julia array? Would you point me to some of the functions of the API that could be used for this?

Thanks,

Carlos

[Carlos Becker]

To formulate my question a bit more specifically (about the last part of my email)

Suppose I want to call jl_new_struct() to instantiate a 'type', I need to provide the type, so I guess

I have to pass the right jl_datatype_t *, which might be found with jl_get_global().

Now, jl_get_global() needs the module pointer, which is not available as a linking symbol sin this would come from an

external module (let's say called MyModule). Is there a function to lookup the pointer of a module given its string name?

Thanks.

[Steven G. Johnson]

On Saturday, February 8, 2014 2:21:36 PM UTC-5, Carlos Becker wrote:

I tried out many ways of passing arrays and other objects from C back to Julia.

So far it seems that it takes a lot of extra code if I want to return, for example, a simple double-array or an array of types (eg structs)

 
This should be quite easy.   For example, return a double array, just allocate a double* array in C with malloc, and return it along with the size (if needed), then convert it to a Julia array with pointer_to_array on the Julia side.   If you want Julia to take charge of freeing the array when you are done with it, pass own=true to pointer_to_array.   For example:

in C:
double *myarray(int n)
{
    double *a = malloc(sizeof(double) * n);
    for (int i = 0; i < n; ++i) a[i] = i;
    return a;
}

in Julia:
myarray(n) = pointer_to_array(ccall((:myarray,"mylib"), Ptr{Float64}, (Cint,), n), n, true)

To return an array of structs, do the same thing, except instead of Float64 in Julia declare a custom immutable type that mirrors the C struct.

[Carlos Becker]

Hi Steven,

I tried that before, I know it is possible, but if the size is unknown to julia, it must be returned as another variable, which makes

coding more difficult if many of such return arrays are needed.

That is why I think it would be interesting to see the julia-api side of it, to see if this can be avoided.

(I prefer to write a bit more of C code but make the ccall clearer in julia)

About my previous question, I am still struggling to create a custom type with julia's C API and fill it in, to pass it back then.

Has anyone done this in such scenario before? I wonder if the necessary symbols are already exported for 3rd party use.

Cheers.

[Carlos Becker]

I think I finally made it work, close to what I wanted.

This could be good for future reference for anyone trying to do something similar.

The code is at https://gist.github.com/anonymous/8897943.

I have two questions about it:

  1) calling jl_gc_disable() and jl_gc_enable() is ok in that context? (see link above)

  2) I still don't know how to avoid passing a pointer to the module I want to get the type from to the C call.

Thanks.

[Jameson Nash]

You need to check jl_gc_is_enabled() before calling
jl_gc_disable/jl_gc_enable. However, this is not recommended. Instead
you should use JL_GC_PUSHn to root each argument.

You also want to avoid using pointer_from_objref since it hides the
object reference from the GC. You can use a ccall argument type of Any
to get the same effect.

You can get modules the same way you get other globals, with jl_get_global.

Why do you want to avoid passing parameters to the ccall? It would
seem much easier to pass a reference to the preallocated MyType
instance that you want filled.

[Tobias Knopp]

Carlos, the code that you showed can be completely written in Julia. It would be helpful if you could give us more insight what you want to achieve. Is there a specific API that you want to wrap? You said that the API returns a double pointer but the length of the memory is not know (if I get that right) How can one use this pointer if the length is unknown?

[Carlos Becker]

Hi Tobias,

I want to be able to return different types from ccall(), according to what happens inside my C/C++ code,

without the need for telling julia what I want to return, its size, etc. Argument passing gets complicated in such cases,

and I believe returning julia objects directly is neater.

I know this means that I have to write more C code to interface with julia, but at the same time I don't have to simultaneously maintain C and Julia code

if something changes in the C code (eg, I add a member in a struct, or I change the output type)

Another example is with a boosting library I wrote, for which I wrote matlab wrappers (here).

In this case its train function accepts a structure with options, plus some arrays. It returns an array of structs

as a 'model' that can be later used. It is nice to have a interface that needs almost no tuning from the matlab or julia end.

I think many would consider this 'direct julia type return' as an advantage, particularly people coming from python/matlab.

I also like very much how ccall can handle void pointers, which is necessary in some situations, but in some other cases

it may be cleaner to work with julia object directly.

I hope I was clear. If I have nice code running I will make it available for other developers.

Cheers.

------------------------------------------

Carlos

[Tobias Knopp]

Carlos,

So if I understand you correctly you have some C functions. Ontop of that you put a C wrapper that dispatches different Julia objects based on some internal state.
If this is the case it would be much cleaner if you would make this state accessible via ccalls and do the dispatching on the Julia side. I also don't see the maintainance argument. It is IMHO much nicer to maintain Julia code instead of C code. But if you add new types to the C (core) code you will have to touch the wrapper anyway.

There is a bunch of code out there in base/ and many different packages and I have never seen that extra C wrappers have been written just to expose functionality of the core libraries.
Wrapping is always done in Julia. Often in a way that there is a thin wrapper that directly exposes the C API and ontop of that a nicer Julia API is written.

By the way: If you are dealing with arrays of structs, making the according type immutable will guarantee that the memory layout is compatible with C.

Cheers

Tobi

[Carlos Becker]

Hi Tobias,

it may be better to look at an example. For instance, to train my SQB classifier:

model = SQBMatrixTrain( featureMatrix, labelVector, maxIters, options )

where featureMatrix and labelVector are matrices (2-dimensional) and vectors (1-dimensional), maxIters is a uint32 and options a structure of the form:

options.<something> = value, where value may be a uint32, string, array, etc.

Lastly, "model" is an array of structs, where each struct also contains arrays in some of its members.

In this case most types/structs are known at compile-time, except for the length of featureMatrix, labelVector and model.

(and in the future may be the size of some arrays inside the structures of the array model may as well vary).

Wrapping such call in Julia with ccall seems overly complicated to me.

About structs, I can declare an immutable struct in Julia and do the exact equivalent in C, but I fear that at some time

I can mess up the order or types, and I have no way (AFAIK) of knowing if I made a mistake from the Julia or C side.

I would get garbage, or hopefully a segfault.

On the other hand, if I write some general wrappers for Julia, I could check whether the passed Julia type is an array or type

of struct I am looking for, check if a struct has field 'name' of type 'type', etc. I think this is a great advantage.

Once something like this is written, modifying structs is straightforward, and if an error happens, I can know what is happening.

Does this make sense? Or am I missing something fundamental from ccall's design?

Thanks.

------------------------------------------

Carlos

[Tobias Knopp]

Am Montag, 10. Februar 2014 14:17:02 UTC+1 schrieb Carlos Becker:

model = SQBMatrixTrain( featureMatrix, labelVector, maxIters, options )

Is this the function declaration of the C function or is this how you want your Julia API to look like?
If its the C signature and model is a pointer to an array of structs, this function cannot be used at all as the user cannot know the length of model.
Note, by the way that the length/size of an array is not a compile time parameter. It can be changed during runtime.
 

[Carlos Becker]

Hi Tobias,

model = SQBMatrixTrain( featureMatrix, labelVector, maxIters, options )

That is how the matlab call looks like, so it is very transparent.

That is how I want the Julia call look like, and I think that it is better to pass Julia objects directly to ccall() in those cases.

It also offers more freedom for C/C++ programmers. I have some code already working, and to me it looks neater than

creating new function calls to pass/return array sizes/etc, and added to that my fear to mis-matched structs between C and Julia.

For example, with what I have now working in C++, to create a struct of type MyStruct from a module,

with members str2, str1 and num, one does:

extern "C" {

void * createStructTest( void *module )

{

JL::Struct::Type s = JL::Struct::create( module, "MyStruct" );

JL::Struct::setField( s, "str2", "This is str2" );

JL::Struct::setField( s, "num", (int64_t)1234 );

        JL::Struct::setField( s, "str2", "another string" );

return s;

}

}

and it automatically checks that MyStruct is a valid type, that str2 is a member and it is a string,

that num is a member and is a int64, etc. If  not, it throws a Julia exception and ends cleanly.

As I said before, I think this could help Julia newcomers when wrapping their C/C++ libraries.

Thanks.

------------------------------------------

Carlos

[Tobias Knopp]

Fair enough, if you wrap the C-API in C++ this can get quite neat. And having myself pushed the documentation of the Julia C-API I definately think that Julia's CAPI is quite good. My intention at that time was, however, embedding Julia in C, in which case the C-API is definately required.

So, it seems that you know what you are doing and for a C++ library the answer might not be as simple as "do it using ccall in Julia".

I would welcome it if you could push your c++ wrapper into the main Julia source code!

Cheers,

Tobi

[Carlos Becker]

Hi Tobias, thanks for you support!

I am still working on it, but when I have something ready for pushing I will let you know.

I have two specific questions now:

1) I think that there would be a few issues with the lack of julia's exported symbols (non-DLLEXPORTed symbols in julia.h)

   The ones related to arrays and basic types are exported now, but others are not, and therefore runtime linking errors occur.

  Shall I create a diff file with a proposal of the additional ones to DLLEXPORT, to check if you agree and we can add that to the main repo?

  Having those symbols exported would also be great for anyone calling Julia from C (aka Embedding Julia).

2) I am still not sure how to get a pointer to a module, given its name. Is this possible with the current julia api?

    I mean to be able to do something like jl_module_t *moduleptr = jl_get_module_by_name("MyModule") or similar.

Thanks.

------------------------------------------

Carlos

[Tobias Knopp]

Am Montag, 10. Februar 2014 18:27:03 UTC+1 schrieb Carlos Becker:

1) I think that there would be a few issues with the lack of julia's exported symbols (non-DLLEXPORTed symbols in julia.h)

   The ones related to arrays and basic types are exported now, but others are not, and therefore runtime linking errors occur.

When I worked on the C-API example that is now in master I had the same issue. The way to go:

- Create Github account

- Fork Julia

- Checkout Julia and compile it. Simply add DLLEXPORT to make the symbols available.

- Create a pull request on github so that your changes can be merged

Note that I have no commit rights to Julia. Such core C things are usually merged by Jeff who is exceptional good in finding flaws in code one has not even thought about :-)

2) I am still not sure how to get a pointer to a module, given its name. Is this possible with the current julia api?

    I mean to be able to do something like jl_module_t *moduleptr = jl_get_module_by_name("MyModule") or similar.

I don't know that

 

[Patrick Foley]

Hi Steven,

I just ran into the same problem.  Your response to Carlos was very helpful and worked for my code.  Thanks!

- Patrick