C compatibility: nested structs?

[Isaiah Norton]

I wrote some code for struct reflection in the clang wrapper generator - for example, this almost-correct definition is fully generated:

https://github.com/ihnorton/libAV.jl/blob/806b223d/src/libav_h.jl#L716-L940

(at 226 fields, as Tim said: "one or two more fields than I'm willing to manually type out")

However, there are two  common struct field types that I'm not sure how to wrap:

- fixed-size arrays: Jeff's proposal for efficient tuples in #2496 sounds like it will solve this.

- nested structs, for example this one from libcurl:

struct CURLMsg {
  CURLMSG msg;       /* what this message means */
  CURL *easy_handle; /* the handle it concerns */
  union {
    void *whatever;    /* message-specific data */
    CURLcode result;   /* return code for transfer */
  } data;
};

I tried defining this as:

julia> immutable CURLMsg_data

         whatever::Ptr{Void}

         result::Int32

       end

julia> isbits(CURLMsg_data)

true

julia> immutable CURLMsg     

         msg::Int32

         easy_handle::Ptr{Void}

         data::CURLMsg_data

       end

However, isbits(CURLMsg) == false. As I understand it, this means that CURLMsg does not have C-compatible memory layout (per the aggregates blog post). So two questions:

1) is there any way to define such a nested struct in Julia, with C memory alignment?

2) if not, is this on the radar?

Thanks,

Isaiah

[Jameson Nash]

I'll preface this by saying there are some changes I would like to make that would make recursive structs possible to declare from Julia as easy as using types.

First though, the struct you point to below is not an example of a nested struct, it is a union (of size/alignment `Int`). However, that's a difference issue. For that, Jeff mentioned to me that it might be good to have something completely general for declaring offsets. I opened an issue to track discussion of syntax possibilities for this https://github.com/JuliaLang/julia/issues/2555.

For embedded structs, it may be changed in the future such that the immutable declaration embeds the struct. That is not true currently; I'm not even sure if it is completely decided now for best representation.

For recursive structs (structs with pointers to other structs), I think this will take a larger change to the julia jl_value_t representation, which is possible, but could be tricky.

Finally, fixed arrays also need be added for good C compatibility, as you mentioned this also needs to be addressed in some fashion.

[Patrick O'Leary]

On Monday, March 25, 2013 12:43:44 AM UTC-5, Jameson wrote:

I'll preface this by saying there are some changes I would like to make that would make recursive structs possible to declare from Julia as easy as using types.

First though, the struct you point to below is not an example of a nested struct, it is a union (of size/alignment `Int`). However, that's a difference issue. For that, Jeff mentioned to me that it might be good to have something completely general for declaring offsets. I opened an issue to track discussion of syntax possibilities for this https://github.com/JuliaLang/julia/issues/2555.

In the meantime, you an manually pad out the structure with either a Uint8 array or a StrPack.PadByte array.

 

For embedded structs, it may be changed in the future such that the immutable declaration embeds the struct. That is not true currently; I'm not even sure if it is completely decided now for best representation.

For recursive structs (structs with pointers to other structs), I think this will take a larger change to the julia jl_value_t representation, which is possible, but could be tricky.

Finally, fixed arrays also need be added for good C compatibility, as you mentioned this also needs to be addressed in some fashion.

These were added in the package version of StrPack.jl if you need something to work with while native Julia support for these more complex structures is in progress.

[Isaiah Norton]

Thanks, Jameson.

In the meantime, you an manually pad out the structure with either a Uint8 array or a StrPack.PadByte array.

Does @struct automatically handle this padding if a field is declared as a Julia Union?

 

For embedded structs, it may be changed in the future such that the immutable declaration embeds the struct. That is not true currently; I'm not even sure if it is completely decided now for best representation.

For recursive structs (structs with pointers to other structs), I think this will take a larger change to the julia jl_value_t representation, which is possible, but could be tricky.

Finally, fixed arrays also need be added for good C compatibility, as you mentioned this also needs to be addressed in some fashion.

These were added in the package version of StrPack.jl if you need something to work with while native Julia support for these more complex structures is in progress.

Cool, that's really good to hear. Does StrPack support inline aggregate declarations, or is it necessary to pre-declare as in my example? (s/union/struct/)

Thanks,

Isaiah

 

[Patrick O'Leary]

On Monday, March 25, 2013 8:36:11 AM UTC-5, Isaiah wrote:

Thanks, Jameson.

In the meantime, you an manually pad out the structure with either a Uint8 array or a StrPack.PadByte array.

Does @struct automatically handle this padding if a field is declared as a Julia Union?

Since union types in Julia are not bitstypes, you can't use them in a @struct. You'll need to use an integer in place of the union and reinterpret it. However if you do that and use the appropriate padding strategy (see the StrPack.jl manual, https://strpackjl.rtfd.org/) you can avoid padding manually. The The StrPack.show_struct_layout() method can help to visualize the resulting memory layout.
 

 

For embedded structs, it may be changed in the future such that the immutable declaration embeds the struct. That is not true currently; I'm not even sure if it is completely decided now for best representation.

For recursive structs (structs with pointers to other structs), I think this will take a larger change to the julia jl_value_t representation, which is possible, but could be tricky.

Finally, fixed arrays also need be added for good C compatibility, as you mentioned this also needs to be addressed in some fashion.

These were added in the package version of StrPack.jl if you need something to work with while native Julia support for these more complex structures is in progress.

Cool, that's really good to hear. Does StrPack support inline aggregate declarations, or is it necessary to pre-declare as in my example? (s/union/struct/)

I'm not sure what you mean by an "inline declaration" here. Can you give an example? (I suspect the answer is "no".)

[Isaiah Norton]

Does @struct automatically handle this padding if a field is declared as a Julia Union?

Since union types in Julia are not bitstypes, you can't use them in a @struct. You'll need to use an integer in place of the union and reinterpret it. However if you do that and use the appropriate padding strategy (see the StrPack.jl manual, https://strpackjl.rtfd.org/) you can avoid padding manually. The The StrPack.show_struct_layout() method can help to visualize the resulting memory layout.

Ok, so this will require some user intervention for now. I will make the generator use the largest union item as a place-holder, and print a warning to manually set up StrPack for each applicable struct.

  

I'm not sure what you mean by an "inline declaration" here. Can you give an example? (I suspect the answer is "no".)

I was thinking of something like:

julia> immutable CURLMsg     

           msg::Int32

           easy_handle::Ptr{Void}

           data::immutable

                     whatever::Ptr{Void}

                     result::Int32

           end

         end

But on second thought I guess this is probably a non-starter. I can do the pre-declaration automatically.