Stuck on passing struct from Julia to C

[Sharmila Gopirajan Sivakumar]

I'm working on the mysql client lib for Julia. I'm using the C api that support statement mode.  To retrieve data after executing the query, it needs C structs (MYSQL_BIND) to be created and passed to C.  Each field of the row retrieved from db will be stored in the corresponding struct and I should read the value from the struct.

 It is unfortunate that I did not notice the line in Julia documentation stating "Currently, it is not possible to pass structs and other non-primitive types from Julia to C libraries." and spent an inordinate amount of time trying to debug what I did wrong.

This particular struct is defined in mysql/mysql.h.  What are the options I have?

1.  Is there a julia call that can read the header files and create the struct or an equivalent feature?
2.  The non-statement mode doesn't seem to have complex structures.  Is that the better approach? 

For reference, to test passing structs, I created the following files
teststruct.jl
somenum= uint64(1)

type MyStruct
    somefield :: Ptr {Uint64}
end

function check(my :: MyStruct)
    ccall((:check, "structtest.so"), Cint, (Ptr{Void},) , pointer_from_objref(my))end

newstruct = MyStruct(pointer_from_objref(somenum))
print("The pointer address stored in somefield as seen in julia: ")
println(newstruct.somefield)
print("The pointer address of newstruct as seen in julia: ")
println(pointer_from_objref(newstruct))
check(newstruct)

structtest.c
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

typedef struct st_mystruct
{
    unsigned long *somefield;
    } MyStruct;

int check(MyStruct *my)
{

  printf("The pointer address of struct as seen in c: %p\n", my);
  printf("The pointer address stored in somefield as seen in c: %p\n", my->somefield);
 }


The output was
The pointer address stored in somefield as seen in julia: Ptr{Uint64} @0x00000000020c1560
The pointer address of newstruct as seen in julia: Ptr{Void} @0x00000000022e5540
The pointer address of struct as seen in c: 0x22e5540
The pointer address stored in somefield as seen in c: 0x206ee90


As we can see, the value of 'somefield' ends up garbled.

[Kevin Squire]

On Tue, Feb 18, 2014 at 10:54 AM, Sharmila Gopirajan Sivakumar <sharmila....@gmail.com> wrote:

I'm working on the mysql client lib for Julia. I'm using the C api that support statement mode.  To retrieve data after executing the query, it needs C structs (MYSQL_BIND) to be created and passed to C.  Each field of the row retrieved from db will be stored in the corresponding struct and I should read the value from the struct.

 It is unfortunate that I did not notice the line in Julia documentation stating "Currently, it is not possible to pass structs and other non-primitive types from Julia to C libraries." and spent an inordinate amount of time trying to debug what I did wrong.

This particular struct is defined in mysql/mysql.h.  What are the options I have?

One option is the StrPack.jl package, which serializes/deserializes Julia types into/from C structs.

There is ongoing work to make Julia types bitwise identical to C structs (see https://github.com/JuliaLang/julia/pull/2818), but this won't be generally available until at least v0.4.

1.  Is there a julia call that can read the header files and create the struct or an equivalent feature?

There's no Julia call for this, but the Clang.jl package also might be useful.  I believe it has some struct support.

Hope this helps.

Cheers,

   Kevin

[Jameson Nash]

Where is that statement in the manual? It should be updated/removed. The unsafe* functions hide an object from the gc and should generally not be used. There is a section in the manual about how to pass struct a to/from C. There's also many packages you can reference for various strategies for doing this (including a MySQL wrapper, iirc)

[Sharmila Gopirajan]

Hi Kevin,

    I will check out StrPack.jl.  Thank you!

[Sharmila Gopirajan]

This is from the latest documentation

http://docs.julialang.org/en/latest/manual/calling-c-and-fortran-code/

Note that no C header files are used anywhere in the process. Currently, it is not possible to pass structs and other non-primitive types from Julia to C libraries. However, C functions that generate and use opaque structs types by passing around pointers to them can return such values to Julia as Ptr{Void}, which can then be passed to other C functions as Ptr{Void}. Memory allocation and deallocation of such objects must be handled by calls to the appropriate cleanup routines in the libraries being used, just like in any C program.

i'm extending the MySQL package that you mentioned. In the documentation, structs are mentioned only under Type Correspondence as

struct T* (where T represents an appropriately defined bits type)		Ptr{T} (call using &variable_name in the parameter list)
struct T (where T represents an appropriately defined bits type)		T (call using &variable_name in the parameter list)
jl_value_t* (any Julia Type)		Ptr{Any}

1. Its not clear what an appropriately defined bits type is.  Should I determine the size of the c struct and create a bits type with that many bytes?  If so how do I access the fields? 
2. What is jl_value_t ? does that also get translated to struct?

3.  Will you please look at the small piece of code in the first message and let me know why the value of 'somefield' is garbled?  i would be grateful if you can show me the right way to do it.

Thank you,

regards,

Sharmi

[Isaiah Norton]

1. Its not clear what an appropriately defined bits type is.  Should I determine the size of the c struct and create a bits type with that many bytes?  If so how do I access the fields?  

Use `immutable`. it should look like this definition linked below, except with `type` -> `immutable`:

https://gist.github.com/ihnorton/9081076#file-libmysql_h-jl-L1359-L1379

 

Search for `isbits` in the manual, which will let you check whether a given type is C-compatible.

(this documentation is a bit scattered and can certainly be improved).

2. What is jl_value_t ? does that also get translated to struct?

No. It is possible to pass a pointer to a jl_value_t, but that is not necessary for mysql.

 

jl_value_t is a struct containing a type tag element. Much of the Julia C code is built around checking

`((jl_value_t*)v)->type`, casting to a more specific type, and then branching appropriately.

3.  Will you please look at the small piece of code in the first message and let me know why the value of 'somefield' is garbled?  i would be grateful if you can show me the right way to do it.

There are a number of examples of passing pointer-to-struct in Julia packages. A relatively simple one is

Nettle.jl (https://github.com/staticfloat/Nettle.jl) and a complex one is PyCall.

Regarding the code you sent, see a simplified version:

https://gist.github.com/ihnorton/9081625

I haven't followed the objref calls through in the original, but they are not necessary. probably not corruption so much as looking at the wrong thing. Note that `print(val)` or just `val` at the REPL will display the address for any Ptr value automatically.

Best,

Isaiah

[Sharmila Gopirajan]

Thank you Isaiah,

           that worked perfectly.