Wrapping an array from C

[Kiran Pamnany]

I have a C function:

int foo(foo_t *handle, int idx, void **ptr);

foo() fills in ptr which points at (part of) an array of elements that was previously allocated. I want to call this function from Julia.

For a bits type (i.e. Int64), the following works:

p = [C_NULL]
ccall((:foo, libfoo), Cint, (Ptr{Void}, Cint, Ptr{Ptr{Void}}), h, i, pointer(p, 1))
return unsafe_wrap(Array, convert(Ptr{atyp}, p[1]), hi[idx]-lo[idx]+1)

atyp stores the type of the elements in the array that foo() returns.

If I use a composite type, Bar, instead of a bits type, then I get back an array of Bar, but each element is #undef. The memory for the elements of the array is present (sizeof(Bar) was used to allocate the space), but the array isn't set up correctly. How can I do this?

Any help is much appreciated!

Thanks in advance,

K

[Steven G. Johnson]

On Tuesday, August 16, 2016 at 4:29:01 PM UTC-4, Kiran Pamnany wrote:

If I use a composite type, Bar, instead of a bits type, then I get back an array of Bar, but each element is #undef.

Declare your type Bar to be immutable (i.e. define it as "immutable Bar" and not "type Bar") if you want to mirror an array of structs in C. 

[Kiran Pamnany]

But I don't want the array elements to be immutable -- I want the Julia code to be able to change their contents.

Why does this work, anyway?

[Steven G. Johnson]

The Julia code can change the array contents, by replacing an array element with another immutable of the same type:

array[i] = Bar(new values...)

 

Why does this work, anyway?

Think about what it means for an object to be mutable: it means if you change a field of the object, then all references to the object must see the change. This means that an array of mutable objects is not like an array of C structs, it is like an array of pointers to C structs.

For example, consider:

      type Foo

           x::Int

      end

      a = [Foo(3), Foo(15)]

      b = a[2]

At this point, a[2] and b are essentially "pointers" to the same Foo(15) object in memory.   If we do 

      b.x = 17

then a[2].x will yield 17 as well.    To achieve this functionality, a Julia Array{Foo} must be (and is) internally equivalent to an array of some kind of pointers to Foo objects.

In contrast, consider an immutable type:

     type Bar 

        x::Int

     end

     a = [Bar(3), Bar(15)]

     b = a[2]

Because the Bar(15) object cannot be mutated, the compiler is free to make a copy of the underlying bits when assigning b = a[2], rather than making them a pointer to the same object.   Furthermore, since it no longer needs to keep a unique blob of bits in memory for each Bar object, Julia can store an Array{Bar} as an array of the underlying data, equivalent to an array of C structs (and, in this case, equivalent in storage to an Array{Int}).

This does not mean that you can't change the contents of the array a, however.  You cannot do a[2].x = 17 (the compiler will give an error since Bar is immutable), but you are perfectly free to do

     a[2] = Bar(17)

That is, while you cannot change an existing Bar object to have different contents, you can replace one Bar object with another in an array.

This distinction, and the existence of immutable types, is really crucial to have efficient code for arrays (and other data structures) of user-defined objects.  It is easy for a C programmer to understand this, because working with an array of structs is almost always going to be more efficient (in both space and speed) than working with an array of pointers to structs.  At the same time, mutable data structures are useful too, so Julia needs to have both.

[Steven G. Johnson]

On Tuesday, August 16, 2016 at 10:21:11 PM UTC-4, Steven G. Johnson wrote:

In contrast, consider an immutable type:

     type Bar 

 

Whoops, I meant "immutable Bar", of course.

[Kiran Pamnany]

Thanks for the comprehensive response Steve!

The memory underlying this array needs to be managed by the C library (it must be pinned and its location published to allow remote memory access). I'd have to provide a specialized allocator, but even that wouldn't work because I cannot fragment the array. So, making the type immutable isn't possible for this purpose.

Is there a way I can create this array of pointers to structs in Julia and make each pointer point at offsets into the block of memory returned by the library?

[Kiran Pamnany]

If this is possible, it'd be even better if I could make those pointers immutable, so that the array can't be messed with, but the Julia code can write into the structs that the array elements point to.

[Bart Janssens]

On Tue, Aug 16, 2016 at 10:29 PM Kiran Pamnany

If I use a composite type, Bar, instead of a bits type, then I get back an array of Bar, but each element is #undef. The memory for the elements of the array is present (sizeof(Bar) was used to allocate the space), but the array isn't set up correctly. How can I do this?

It looks like ptr is actually an array of pointers? If so, you could access the first element using:
unsafe_load(convert(Ptr{atyp}, p[1]))

Cheers,

Bart

[Steven G. Johnson]

On Wednesday, August 17, 2016 at 12:39:51 AM UTC-4, Kiran Pamnany wrote:

The memory underlying this array needs to be managed by the C library (it must be pinned and its location published to allow remote memory access). I'd have to provide a specialized allocator, but even that wouldn't work because I cannot fragment the array. So, making the type immutable isn't possible for this purpose.

 This is fine.  You can have an array of immutables whose memory is allocated in C.

An array of immutables is just an array of structs.  Replacing an element just overwrites that memory in an array, it doesn't involve any allocation.

[Kiran Pamnany]

On Wednesday, August 17, 2016 at 5:46:34 AM UTC-7, Steven G. Johnson wrote:

An array of immutables is just an array of structs.  Replacing an element just overwrites that memory in an array, it doesn't involve any allocation.

So the line of code you posted:

array[i] = Bar(x, y, ...)

Will simply overwrite the Bar that's at array[i]?

Is there a way to assert that a type is immutable?

[Kiran Pamnany]

On Wednesday, August 17, 2016 at 12:49:58 AM UTC-7, Bart Janssens wrote:

It looks like ptr is actually an array of pointers? If so, you could access the first element using:
unsafe_load(convert(Ptr{atyp}, p[1]))

ptr is a pointer to the beginning of an array of structs. Steve has explained that there is really no such thing in Julia, except by using immutable types.

[Steven G. Johnson]

On Wednesday, August 17, 2016 at 12:08:12 PM UTC-4, Kiran Pamnany wrote:

On Wednesday, August 17, 2016 at 5:46:34 AM UTC-7, Steven G. Johnson wrote:

An array of immutables is just an array of structs.  Replacing an element just overwrites that memory in an array, it doesn't involve any allocation.

So the line of code you posted:

array[i] = Bar(x, y, ...)

Will simply overwrite the Bar that's at array[i]?

Yes. 

Is there a way to assert that a type is immutable?

You declare it as

     immutable MyType

         ....

     end

[Kiran Pamnany]

On Wednesday, August 17, 2016 at 11:49:14 AM UTC-7, Steven G. Johnson wrote:

Will simply overwrite the Bar that's at array[i]?

Yes. 

I verified this. Fantastic, thanks!

 

Is there a way to assert that a type is immutable?

You declare it as

     immutable MyType

         ....

     end

No, I mean when I am passed a type as a function parameter, is there a way to check if it has been declared as an immutable type?

 

[Steven G. Johnson]

On Wednesday, August 17, 2016 at 4:51:43 PM UTC-4, Kiran Pamnany wrote:

No, I mean when I am passed a type as a function parameter, is there a way to check if it has been declared as an immutable type?

There doesn't seem to be a built-in function for this (the isimmutable function works on values, not types), but the following function should work:

isimmutable_type{T}(::Type{T}) = T <: Tuple || !T.mutable 

[Jameson]

that should be:
isimmutable(T::DataType) = !T.mutable

You could add more cases for the other Types (Union, TypeVar, and TypeConstructor), although whether you need that in practice would be determined by your use case. Tuple isn't a special case type since mid-2015 (during v0.4-dev)

[Steven G. Johnson]

On Sunday, August 21, 2016 at 12:59:12 AM UTC-4, Jameson wrote:

that should be:
isimmutable(T::DataType) = !T.mutable

You could add more cases for the other Types (Union, TypeVar, and TypeConstructor), although whether you need that in practice would be determined by your use case. Tuple isn't a special case type since mid-2015 (during v0.4-dev)

The implementation of isimmutable in Base should probably be updated then.  https://github.com/JuliaLang/julia/blob/c74b7d0161b85ea78cbe21609fe37da3b72d39a1/base/reflection.jl#L118