cannot resize array with shared data

[andrew cooke]

Hi,

I'm trying to understand the bug report at https://github.com/andrewcooke/CRC.jl/issues/4 which has

julia> using CRC

julia> crc32c = crc(CRC_32_C)
handler (generic function with 3 methods)

julia> a = b"hello world";

julia> crc32c(a)
0xc99465aa

julia> resize!(a, 5)
ERROR: cannot resize array with shared data
 in resize! at ./array.jl:545


but I can find no docs on what "shared data" for arrays means.  All I have found is the C code at https://github.com/JuliaLang/julia/blob/master/src/array.c which, tbh, is a little opaque (what is a->how? how can i reset a->shared?)

It sounds like the array code is doing something like reference counting and refusing to be mutable when there are references?  How do I decrement the count?

Is there any documentation on sharing of arrays?

My guess is that a caller could work round this by making a copy, but that seems inefficient.  Especially since my code simply iterates through the data and then "no longer cares".

Sorry, this email is very vague.  Given the lack of docs (nothing about sharing in the manual i can see) I wonder if I've stumbled over some internal detail that should not be exposed to users?

Any help appreciated,
Andrew

[Kevin Squire]

Hi Andrew, 

A very similar issue came up in https://github.com/JuliaLang/julia/issues/4592, which was addressed by 

https://github.com/JuliaLang/julia/commit/2570b4a60e5ab5c438c57ed9a3eafa5c881bc39a

If you look at the fix, you'll see that the error in your code comes from array_try_unshare or jl_array_grow_end.  In either case, unsharing/resizing doesn't work for arrays sharing data with "how == 3", which means that the data being resized is actually just a pointer to the original data (defined here).

In your case, what is happening is that a string is created ("hello world"), and the variable a simply gets a pointer to the underlying string data (the b_str macro does just that), so it's sharing data (with a string that isn't referenced anymore), and therefore can't be resized.

At a minimum, it would be great to have this information somewhere in the docs.  A more informative error might also be useful.  Would you mind opening up an issue (and/or if you're so motivated, a pull request)?

Cheers,

   Kevin

[andrew cooke]

hi,

thanks for replying, but i didn't understand much of what you said.

i think you're saying that my code (the CRC library) is calling array_try_unshare or jl_array_grow_end?  i don't see either of those in my code at https://github.com/andrewcooke/CRC.jl/blob/master/src/CRC.jl

what i do see is a call to reinterpret().  that seems to cause this problem.

so i guess my question is - is there any way to specify that the reinterpret has a finite scope?  by the time that this error occurs the reinterpreted array is no longer used.

i'm happy to raise an issue, but i'm still not sure what the issue is.  it seems like julia is trying too hard to rotect me from something that isn't happening?

here's my demo of the issue with reinterpret:

julia> a = b"1234"
4-element Array{UInt8,1}:
 0x31
 0x32
 0x33
 0x34

julia> resize!(a, 2)
2-element Array{UInt8,1}:
 0x31
 0x32

julia> resize!(a, 4)
4-element Array{UInt8,1}:
 0x31
 0x32
 0x00
 0x34

julia> reinterpret(Uint32, a);

julia> resize!(a, 2)

ERROR: cannot resize array with shared data
 in resize! at ./array.jl:545

thanks,
andrew

[Kevin Squire]

Hi Andrew,

I saw your issue (cross link: https://github.com/JuliaLang/julia/issues/10354), and remembered that I hadn't replied again here.  Sorry about that.  At this point, you might have figured out most of this, but just for completeness:

thanks for replying, but i didn't understand much of what you said.

i think you're saying that my code (the CRC library) is calling array_try_unshare or jl_array_grow_end?  i don't see either of those in my code at https://github.com/andrewcooke/CRC.jl/blob/master/src/CRC.jl

When you call resize! (here), it can call jl_array_grow_end (here, which you found), which is where that error comes from.  

 

what i do see is a call to reinterpret().  that seems to cause this problem.

More specifically (you may have found this), reinterpret (here) calls jl_reshape_array (here), which sets a->how = 3, which means that the internal structure representing an array in Julia holds a pointer to the array that actually owns the data (here).

There may or may not have been a good reason for arrays to be designed like this, but without digging into the array.c code more, it's hard (for me) to say if this is easy to change.

Cheers,

   Kevin