Passing composite types to C libraries (libgit2)

[Anthony Urena]

I am attempting to implement a Julia package that utilizes libgit2. I have been searching for a way to pass nested structs in to a C library but I'm not sure how to do it.

Here is one example of a fairly common struct:

git_repository

As you can see, this has many nested structs that have nested structs themselves:

git_odb, git_refdb, git_config, git_index, git_cache, etc.

Here is an often used function that requires a git_repository struct as input:

git_revparse

So as you can see I will need to pass a lot of structs back and forth between Julia and libgit2. I looked through the manual and the group pages but some of the information seems to be outdated. I also found StrPack.jl, but I believe the intent of this project is to eventually merge it with base, so I'm hesitant to have it be dependent upon other packages. I'm also not sure if StrPack is something that would help, or if I'm looking for the wrong thing. I think arpack.jl is doing some kind of struct passing but I'm not sure.

I'm not sure how to do this if I don't define the same structures on the Julia side.

Does anyone know how I could define Julia types that map to C structs, or is that not possible? Would it be a good idea to redefine the C structs on the Julia side, or is that bad form?

[Jacob Quinn]

It will probably depend on the use case in the source code.

For example, in the ODBC package, there are pre-defined ODBC types for Date, Time, and TimeStamp. These are simple C structs that are used to allocate the appropriate space and into which the ODBC manager stores retrieved values from an RDBMS. The API here allows that the SQLDate, SQLTime, and SQLTimestamp structs are defined as immutable types in Julia and passed to the ODBC manager. Once the values are retrieved, there's a simple conversion to access the immutable type fields to convert to a Date/DateTime object in Julia (from the Datetime.jl package).

That may or may not have made any sense, but what I'm trying to say is I think it's ok to define the structs using Julia immutable types which can be passed into the API. My guess is they've made this fairly easy to work with since it's made to be language agnostic. I'll try to take a look to see if it is indeed a simple case like the one in ODBC.

-Jacob

[Alessandro "Jake" Andrioni]

Right now you have to rewrite them "normalized" in Julia. Say, for
example, you have:

struct A {
struct B var1;
struct C var2;
int var3;
}
struct B {
void *b1;
double b2;
}
struct C {
long c1;
}

You would then have to create the following Julia types and use them
in ccall as if they were the structs:

type A
var1_b1::Ptr{Void}
var1_b2::Float64
var2_c1::Clong
var3::Cint
end
type B
b1::Ptr{Void}
b2::Float64
end
type C
c1::Clong
end

Here's another example from my MPFI wrapper:
https://gist.github.com/andrioni/179d068793ae8cf90b33

[Jacob Quinn]

Looks like it's more complicated than I've had to deal with, the `gitref_spec` definition for example contains single bitfields. Maybe Keno or Jameson can comment on how this would be done or if it's possible as I know they've done some work on struct-passing.

struct git_refspec {
    char *string;
   char *src;
      char *dst;
      unsigned int force :1,
          push : 1,
          pattern :1,
          dwim :1,
          matching :1;
};

On Thursday, October 10, 2013 8:39:04 AM UTC-4, Anthony Urena wrote:

[Alessandro "Jake" Andrioni]

I'm pretty sure that the bit field will just be equivalent to an
unsigned int in this case, so

type git_refspec
string::Ptr{Uint8}
src::Ptr{Uint8}
dst::Ptr{Uint8}
bitfield::Cuint
end

would work.

[Alessandro "Jake" Andrioni]

(to be a bit more specific, the bit field starts with least
significant bit, so 0x00000001 means `force` is set, 0x00000002 means
`push` is set, etc)

[Patrick O'Leary]

On Thursday, October 10, 2013 7:39:04 AM UTC-5, Anthony Urena wrote:

I also found StrPack.jl, but I believe the intent of this project is to eventually merge it with base, so I'm hesitant to have it be dependent upon other packages. I'm also not sure if StrPack is something that would help, or if I'm looking for the wrong thing.

No, there is no intent to take that functionality as-is and transition it to Base. Certain applications of StrPack (not originally intended by its author) will hopefully be subsumed by improvements to ccall(), and some already have, but the overall purpose of the package is to parse/create structured binary streams.

[Anthony Urena]

Thank you for all of your feedback, I understand this process better now. There's one thing that I'm still unsure of.
In the examples I've seen so far, the types that have other types inside them have to have all of the attributes of the nested types.

For example, type A in Alessandro's example has var1_b1::Ptr{Void}. Will that pointer contain a reference to an Julia object (object might be the wrong term) of type B when returned from the C API? If that's the case, why do all of the attributes of the nested types have to be re-listed inside the parent type? Shouldn't you have a reference to the nested structs from that pointer?'

Here's an example of the type of nested structs that I need to passinto the API:
struct git_refdb {
    git_refcount rc;
    git_repository *repo;
    git_refdb_backend *backend;
};

typedef struct {
    git_atomic refcount;
    void *owner;
} git_refcount;

typedef struct {
#if defined(GIT_WIN32)
    volatile long val;
#else
    volatile int val;
#endif
} git_atomic;

Is there an easier way to create the nested types on the Julia side,
or does it have to be this way for reasons I don't understand?

In the manual under "Mapping C Types to Julia" it says that calls to
`convert` will automatically be inserted to convert each argument to
the specified type. This is the example that it has:
ccall( (:foo, "libfoo"), Void, (Int32, Float64),
 convert(Int32, x), convert(Float64, y))
Is there some way for me to define a new conversion that will handle
the nested structs when they are passed to and from the API?

Quick recap:
Will a Ptr{Void} in a parent type get filled with a reference to an object of the nested type when passed to an API?
Why are all of the attributes of the nested types re-listed inside the parent type, and is there a way to inline that process?
Is there an easier way to create the nested types to pass to C in Julia?
Will defining a new conversion for the nested structs make the API passing easier?Thank you, I realize I have a lot of questions. The support here has been phenomenal so far; I really do appreciate it.