request for feature: modify field in immutable object in a container

[vav...@uwaterloo.ca]

I would like to request the following language feature: a function or macro to modify a field of an immutable inside a container.  Consider:

immutable T

  fielda::Int

  fieldb::Int

  fieldc::Int

end

function modify_fieldc!(x::Array{T,1}, sub::Int, newc::Int)

   x[sub] = T(x[sub].fielda, x[sub].fieldb, newc)

end

This function modifies one field of an immutable object that sits inside a container.  The above construct, namely:

   x[sub] = T(x[sub].field1, x[sub].field2, ... , newval, ... x[sub].fieldn)

occurs rather frequently in my code.  It is not very readable and is also fragile in the case that I modify my code and put more fields in T later. It would be much nicer if there were a universal function like this:

   modifyField!(x, sub, fieldc, newc)

Note that I declared T to be 'immutable' rather than 'type' for performance reasons-- I prefer the data in the array x to be packed in memory rather than accessed with pointers.  If T were a 'type' then obviously the problem would go away.

Maybe it is already possible to write a function or macro for this purpose in the existing language?

Thanks,

Steve Vavasis

[Mauro]

Something along these lines is being implemented/discussed in
https://github.com/JuliaLang/julia/pull/6122
https://github.com/JuliaLang/julia/issues/5333

Meanwhile, below function should work. Note that I did not write
copyandmodify! as it makes a new instance and does not modify it in
place. Which isn't really possible for an immutable.

function copyandmodify{T}(pp::T, di)
di = !isa(di, Associative) ? Dict(di) : di
ns = names(pp)
args = Array(Any, length(ns))
for (i,n) in enumerate(ns)
args[i] = get(di, n, getfield(pp, n))
end
T(args...)
end
copyandmodify{T}(pp::T; kws...) = copyandmodify(pp, kws)

# examples
immutable A{I}
a::I
b::I
c::Float64
end
a1 = A(5, 6, 5.)
a2 = copyandmodify(a1, b=7)
a3 = copyandmodify(a1, ((:a,7),)) # not using keywords is probably faster

immutable B{I}
a::I
b::I
c::Vector{I}
end
b1 = B(5, 6, [5,5])
b2 = copyandmodify(deepcopy(b1), b=7) # use deepcopy to not share array c

[Simon Danisch]

You can actually modify immutables in an array directly. I actually implemented this in GLAbstraction for my own usage.

I'm implementing FixedSizeArrays.jl at the moment, which will include this feature (In the feature list I call it "setindex!/getindex for arrays of FSA (e.g. easy access to single fields)")

This only works for immutables as it needs the tightly packed memory layout of the array... Sadly, immutability doesn't occur in the type hierarchy, so the setindex! that I'm talking about can't be written to work for any immutable.

So this means, you either need to inherit from FixedSizeArrays to use this, or copy the code (when it's transfered to fixedsizearrays).

Maybe we can open an issue at Julia, to see if we can get something like this into base.

[Simon Danisch]

Here is some code:

https://gist.github.com/SimonDanisch/805fee6b676107e378ae#file-setindex-jl

Am Sonntag, 8. Februar 2015 21:35:20 UTC+1 schrieb vav...@uwaterloo.ca:

[Mauro]

So, this actually modifies an immutable, right? This doesn't seem
right, aren't they supposed to be immutable?

On Mon, 2015-02-09 at 14:29, Simon Danisch <sdan...@gmail.com> wrote:
> You can actually modify immutables in an array directly. I actually
> implemented this in GLAbstraction for my own usage.
> I'm implementing FixedSizeArrays.jl

> <https://github.com/SimonDanisch/FixedSizeArrays.jl> at the moment, which

[Simon Danisch]

Someone else has to answer this. I suppose it's not that bad. The immutables are getting copied into the heap allocated memory from the array, which should be fairly okay to mutate?!

But I seriously don't know the details on this. From a semantic viewpoint it's definitely not okay!

[Kevin Squire]

Hi Steve,

I have a function in VideoIO.jl which does this: https://github.com/kmsquire/VideoIO.jl/blob/master/src/util.jl#L6-L14

When wrapping libav/ffmpeg libraries, some of the structs have dozens of fields, yet I found that needed to declare them as immutable for C compatibility. At the same time, I needed to modify a few values occasionally.

I have used it without consequences yet, but I don't know that there aren't any.  I would reason that modifying a field of an immutable is equivalent to overwriting the whole thing with mostly the same values, but I'm pretty sure the compiler doesn't recognize that.

Cheers,

   Kevin

[Michael Francis]

My 2c but is sounds like Julia needs a struct keyword which is C packed but does not enforce immutability. Immutable should means just that and allows the compiler more options for optimization. 

[Kevin Squire]

My understanding is that normal (mutable) types should be C-layout compatible soon (and maybe already are in v0.4--I forget the status of that work).

Although, I'm less sure of the future of packed arrays of types. 

Cheers,

   Kevin 

[Jameson Nash]

All types (immutable and mutable) have been c-compatible since v0.1.

I rewrote the manual page on interacting with c and fortran last night in my jn/ccall3 branch PR. If anyone can review and comment on that text, it would be much appreciated. I'm not sure if readthedocs allows you to preview pull request changes. In a local git repo checkout, you can do 'cd doc && make html'

I've wanted to make something like Simons gist above what unsafe_store does, given a symbol, but hadn't gotten around to implementing it yet.

[Simon Danisch]

But arrays of type are still not densely packed, right?
I tried it with the newest master and it doesn't look like it.

Am Sonntag, 8. Februar 2015 21:35:20 UTC+1 schrieb vav...@uwaterloo.ca:

[Tobias Knopp]

No I think its still only pointers to the type objects that are stored in the array. But note that what Jameson wrote still holds. In ccall you can assume that the type has C struct layout. No need to go for an immutable there.

Tobi

[Kevin Squire]

I haven't had the chance to check recently (and am not sure when I will be able to), but when wrapping libav/ffmpeg, I found that passing mutable objects (of types) to C-code produced a segmentation fault, but that passing immutables worked (with no other changes).  I believe that was after Jameson's changes, and had just assumed that the layout wasn't compatible still.  

Anyway, when I get the chance, I'll try again, and create an issue if it still has problems.

Kevin

[Simon Danisch]

While we're at it...

Why does this work:

reinterpret(Float32, Int32(0))

@assert isbits(ImmutableType1) && isbits(ImmutableType2) && (sizeof(ImmutableType1) == sizeof(ImmutableType2)) 

reinterpret(ImmutableType1, Array(ImmutableType2, 42))

but this does not:

reinterpret(ImmutableType1, ImmutableType2(...))

Am Sonntag, 8. Februar 2015 21:35:20 UTC+1 schrieb vav...@uwaterloo.ca:

[Jameson Nash]

Just because their layout is the same doesn't mean you can use the wrong one when declaring a data-structure. They still have a very nearly 1-to-1 correspondence with C in that respect too.

[Jameson Nash]

That last one requires a different sequence of llvm instructions than the other examples (it's the difference between isbits and bitstype). Not hard, but likely unimplemented.

[Kevin Squire]

Just because their layout is the same doesn't mean you can use the wrong one when declaring a data-structure. They still have a very nearly 1-to-1 correspondence with C in that respect too.

I don't quite understand what this means.  When would passing a pointer to an immutable vs a pointer to a type produce different results from called C code?

Kevin

[ele...@gmail.com]

On Tuesday, February 10, 2015 at 1:25:15 AM UTC+11, Simon Danisch wrote:

Someone else has to answer this. I suppose it's not that bad. The immutables are getting copied into the heap allocated memory from the array, which should be fairly okay to mutate?!

But I seriously don't know the details on this. From a semantic viewpoint it's definitely not okay!

WARNING: if you use unsafe practices to mutate an immutable object you enter the realms of undefined behaviour where the compiler/runtime can do anything it likes from sending your details to Scammers-R-Us to eating your shorts.  Seriously if anything in the compiler optimisations, the runtime or garbage collector is changed to depend on immutables being, well, immutable, then it will cause unknown issues.  And there is no guarantee that this hasn't happened already, just your specific code hasn't hit it ... yet.

Things that may be an issue include:

1. Julia now has a generational GC, does it assume immutable intergenerational pointers cannot change and doesn't rescan for them?

2. Does the compiler generate the correct GC write barriers for unsafe operations and does it ignore immutables since they can't change?

3. Do optimisations use copies of immutables as implied in the manual, so you don't mutate what you think you mutate?

4. Does the generated code *not* use copies where it normally might, since it doesn't matter if the object is immutable, so again you don't mutate what you expected to?

1. and 2. require the GC to run during the lifetime of the mutated immutable, maybe you have been lucky and that hasn't happened ... yet.

3. and 4. the behaviour depends on what other use is made of the immutable, again maybe you have been lucky.

As for passing immutables to C, well was always going to be risky, C just doesn't understand immutable.  If as suggested by Jameson mutables are layout compatible with C then the faster the habit of using immutables for that dies the better.

Train is approaching my stop, so thats enough for this rant.

Cheers

Lex

[Jameson Nash]

On Mon Feb 09 2015 at 7:59:20 PM <ele...@gmail.com> wrote:

On Tuesday, February 10, 2015 at 1:25:15 AM UTC+11, Simon Danisch wrote:

Someone else has to answer this. I suppose it's not that bad. The immutables are getting copied into the heap allocated memory from the array, which should be fairly okay to mutate?!

But I seriously don't know the details on this. From a semantic viewpoint it's definitely not okay!

WARNING: if you use unsafe practices to mutate an immutable object you enter the realms of undefined behaviour where the compiler/runtime can do anything it likes from sending your details to Scammers-R-Us to eating your shorts.  Seriously if anything in the compiler optimisations, the runtime or garbage collector is changed to depend on immutables being, well, immutable, then it will cause unknown issues.  And there is no guarantee that this hasn't happened already, just your specific code hasn't hit it ... yet.

Semantically, modifying a field of the immutable in the array is the same as reading the whole thing, creating a new immutable with one field modified, and writing it back to the array. The only thing you are likely missing by hacking unsafe_store in this manner is a missing TBAA annotation. That's fairly unlikely to matter here though (more about this below).

Things that may be an issue include:

1. Julia now has a generational GC, does it assume immutable intergenerational pointers cannot change and doesn't rescan for them?

If your type contains pointers to other non-isbits types, you are probably just as well off using a type. They are stored the same. Only `isbits` types get the special inline-storage behavior.

 

2. Does the compiler generate the correct GC write barriers for unsafe operations and does it ignore immutables since they can't change?

No, it doesn't generate write barriers during calls to unsafe_store. Although if you are writing bits data inside the immutable, it hardly matters, since it doesn't need a write barrier to do that.

 

3. Do optimisations use copies of immutables as implied in the manual, so you don't mutate what you think you mutate?

Quite frequently. But if you mutate the array data, you mutate the array data, no questions asked. What you definitely shouldn't do is grab a pointer to your immutable (via pointer_from_objref) and try to mutate that (but trying to grab a pointer to an immutable should become an error in a future version of Julia anyways).

 

4. Does the generated code *not* use copies where it normally might, since it doesn't matter if the object is immutable, so again you don't mutate what you expected to?

This is possible. But it would require LLVM to decide that the array slot itself couldn't have been re-assigned. Any intervening function call is typically enough to block optimization on assuming the array data hasn't changed. But TBAA (type-based alias analysis) is probably more likely to bite you here, since LLVM won't consider the offset store to affect the immutable type it lives inside.

1. and 2. require the GC to run during the lifetime of the mutated immutable, maybe you have been lucky and that hasn't happened ... yet.

3. and 4. the behaviour depends on what other use is made of the immutable, again maybe you have been lucky.

As for passing immutables to C, well was always going to be risky, C just doesn't understand immutable.  If as suggested by Jameson mutables are layout compatible with C then the faster the habit of using immutables for that dies the better.

There's nothing risky about it. Types have the same layout and the same behavior when getting passed to C regardless of whether they are declared mutable or immutable. The ccall code hardly even checks the `.mutable` field of the type (and that code hopefully will be deprecated soon, since it doesn't really benefit the user).

Often, however, `immutable` provides a better representation of a C-struct and is therefore necessary for compatibility (unless the only usages of the type are by-pointer).

The updated documentation I mentioned above is part of my ccall-enhancement pull request (which makes calling c functions that require or return by-value structs actually work correctly). It can be read at:

https://github.com/JuliaLang/julia/blob/jn/ccall3/doc/manual/calling-c-and-fortran-code.rst.

Regards.

[Michael Francis]

It does sound like immutable has been conflated with packed data structures. That seems unfortunate. It would be nice if the properties of a type could be selected at create time rather than with specific reserved words. Something like the following.

type Buffer <: Immutable,Packed
a::Int64
end

Though my preference would be to remove the reserved keywords all together.

Buffer = type(
a::Int64
; flags = immutable | packed
)

Which is just a function which happens to register a type and has a few named args.

As an aside, the standard lib breaks immutability with char arrays for fast ASCII string joins, though that usage should be safe as it is in the scope of the allocating method.

[Jameson Nash]

That's a pretty reasonable idea. There's even a few more attributes (coming from llvm and the platform ABI), that I would add to the list (such as float-compatible, complex-compatible, unsigned/signed).

However, it's unlikely to happen because it's also a lot more work on everyone. The more types of types that get introduced, the more interactions they start to have and it can get a bit hard to manage for both code authors and compiler writers. Supporting all the same set of operations as C is a partial goal, but supporting them with exactly the same semantics is not.

I expect at some future point, immutable will become synonymous with inline and `const` annotations will be allowed on individual fields of mutable types. This is similar to what you are suggestion, although it comes at it from a slightly different angle.

(packed is usually something else – however, the ability to specify arbitrary offsets for fields in a type is also something that will likely be implemented in the future)

> As an aside, the standard lib breaks immutability with char arrays for fast ASCII string joins, though that usage should be safe as it is in the scope of the allocating method.

I've occasionally tried to convert such code when I come across it to create an array first, then to turn that into a String. That has the same allocation semantics, but doesn't ignore the the intended immutability of strings. Although, as you pointed out, in practice, it doesn't matter here.

[Michael Francis]

Totally agree packed has a strong meaning of fixed offsets et al. Probably a bad choice of worlds.

Part of my thought for going to a more functional interface is that the properties could be closed in the definitions. Hence we could have

Foo = Immutable( ... )

Which wraps the properties of the type away from the end user. Personally I'd rather get away from reserved words in the parser and have more natively in julia. A person writing a c interface might use a type 'cstruct' which has appropriate properties (and constraints ) on the types which may be used in it. The average user wouldn't see nor use this. It would allow clean extension though, for example a gpu api which only allows float not double could define its own type creator.

You can do this today with macros, but that seems like a patch round not exposing directly to julia in the first place.

[ele...@gmail.com]

On Tuesday, February 10, 2015 at 1:10:33 PM UTC+11, Jameson wrote:

On Mon Feb 09 2015 at 7:59:20 PM <ele...@gmail.com> wrote:

On Tuesday, February 10, 2015 at 1:25:15 AM UTC+11, Simon Danisch wrote:

Someone else has to answer this. I suppose it's not that bad. The immutables are getting copied into the heap allocated memory from the array, which should be fairly okay to mutate?!

But I seriously don't know the details on this. From a semantic viewpoint it's definitely not okay!

WARNING: if you use unsafe practices to mutate an immutable object you enter the realms of undefined behaviour where the compiler/runtime can do anything it likes from sending your details to Scammers-R-Us to eating your shorts.  Seriously if anything in the compiler optimisations, the runtime or garbage collector is changed to depend on immutables being, well, immutable, then it will cause unknown issues.  And there is no guarantee that this hasn't happened already, just your specific code hasn't hit it ... yet.

Semantically, modifying a field of the immutable in the array is the same as reading the whole thing, creating a new immutable with one field modified, and writing it back to the array. The only thing you are likely missing by hacking unsafe_store in this manner is a missing TBAA annotation. That's fairly unlikely to matter here though (more about this below).

This depends on what is the immutable object.  I am assuming that its the memory inside the array, not some heap object pointed to by the array.  If its the latter ignore most of my rant, but I understand (from your excellent documentation referenced below) its the former for at least some immutable structs.  If its the memory inside the array then copying another immutable with different field values over it changes the immutable, its only existence is in the array slot, so effectively arrays of immutables are immutable, except for push pop etc.

So a user who makes a local copy of a field of an immutable is stuffed (well, its immutable so the field can't change right?), so is the compiler which hoists x = immutable.field out of a loop, as is LLVM if it understands immutable and keeps field values in registers.  As you note AA is supposed to fix this, but if its missing then the problem remains, and the user copy isn't subject to AA correction anyway.

 

Things that may be an issue include:

1. Julia now has a generational GC, does it assume immutable intergenerational pointers cannot change and doesn't rescan for them?

If your type contains pointers to other non-isbits types, you are probably just as well off using a type. They are stored the same. Only `isbits` types get the special inline-storage behavior.

And then you don't mutate an immutable, but it would be good to document this restriction.

 

 

2. Does the compiler generate the correct GC write barriers for unsafe operations and does it ignore immutables since they can't change?

No, it doesn't generate write barriers during calls to unsafe_store. Although if you are writing bits data inside the immutable, it hardly matters, since it doesn't need a write barrier to do that.

Yes, I was thinking of pointers, but your exclusion above cures that.

 

 

3. Do optimisations use copies of immutables as implied in the manual, so you don't mutate what you think you mutate?

Quite frequently. But if you mutate the array data, you mutate the array data, no questions asked. What you definitely shouldn't do is grab a pointer to your immutable (via pointer_from_objref) and try to mutate that (but trying to grab a pointer to an immutable should become an error in a future version of Julia anyways).

Again cached copies should be invalidated somehow.

 

 

4. Does the generated code *not* use copies where it normally might, since it doesn't matter if the object is immutable, so again you don't mutate what you expected to?

This is possible. But it would require LLVM to decide that the array slot itself couldn't have been re-assigned. Any intervening function call is typically enough to block optimization on assuming the array data hasn't changed. But TBAA (type-based alias analysis) is probably more likely to bite you here, since LLVM won't consider the offset store to affect the immutable type it lives inside.

Yes, which leads to the above problems.

 

1. and 2. require the GC to run during the lifetime of the mutated immutable, maybe you have been lucky and that hasn't happened ... yet.

3. and 4. the behaviour depends on what other use is made of the immutable, again maybe you have been lucky.

As for passing immutables to C, well was always going to be risky, C just doesn't understand immutable.  If as suggested by Jameson mutables are layout compatible with C then the faster the habit of using immutables for that dies the better.

There's nothing risky about it. Types have the same layout and the same behavior when getting passed to C regardless of whether they are declared mutable or immutable. The ccall code hardly even checks the `.mutable` field of the type (and that code hopefully will be deprecated soon, since it doesn't really benefit the user).

Its risky in that a mutation inside the C library won't be detected by AA either.

 

Often, however, `immutable` provides a better representation of a C-struct and is therefore necessary for compatibility (unless the only usages of the type are by-pointer).

Thats probably the problematic design decision.  Hindsight (that wonderful thing) might suggest a C compatible definition for types that are passed to C (somewhat like the nebulous "standard layout" structs in C++11).

 

The updated documentation I mentioned above is part of my ccall-enhancement pull request (which makes calling c functions that require or return by-value structs actually work correctly). It can be read at:

https://github.com/JuliaLang/julia/blob/jn/ccall3/doc/manual/calling-c-and-fortran-code.rst.

Nice docs.

Cheers

Lex

[Jameson Nash]

On Tue Feb 10 2015 at 12:16:40 AM <ele...@gmail.com> wrote:

On Tuesday, February 10, 2015 at 1:10:33 PM UTC+11, Jameson wrote:

On Mon Feb 09 2015 at 7:59:20 PM <ele...@gmail.com> wrote:

On Tuesday, February 10, 2015 at 1:25:15 AM UTC+11, Simon Danisch wrote:

Someone else has to answer this. I suppose it's not that bad. The immutables are getting copied into the heap allocated memory from the array, which should be fairly okay to mutate?!

But I seriously don't know the details on this. From a semantic viewpoint it's definitely not okay!

WARNING: if you use unsafe practices to mutate an immutable object you enter the realms of undefined behaviour where the compiler/runtime can do anything it likes from sending your details to Scammers-R-Us to eating your shorts.  Seriously if anything in the compiler optimisations, the runtime or garbage collector is changed to depend on immutables being, well, immutable, then it will cause unknown issues.  And there is no guarantee that this hasn't happened already, just your specific code hasn't hit it ... yet.

Semantically, modifying a field of the immutable in the array is the same as reading the whole thing, creating a new immutable with one field modified, and writing it back to the array. The only thing you are likely missing by hacking unsafe_store in this manner is a missing TBAA annotation. That's fairly unlikely to matter here though (more about this below).

This depends on what is the immutable object.  I am assuming that its the memory inside the array, not some heap object pointed to by the array.  If its the latter ignore most of my rant, but I understand (from your excellent documentation referenced below) its the former for at least some immutable structs.  If its the memory inside the array then copying another immutable with different field values over it changes the immutable, its only existence is in the array slot, so effectively arrays of immutables are immutable, except for push pop etc.

So a user who makes a local copy of a field of an immutable is stuffed (well, its immutable so the field can't change right?), so is the compiler which hoists x = immutable.field out of a loop, as is LLVM if it understands immutable and keeps field values in registers.  As you note AA is supposed to fix this, but if its missing then the problem remains, and the user copy isn't subject to AA correction anyway.

Right, it's an immutable so the field can't change. For an isbits type, the value in the array is completely independent of the value extracted from the array. llvm will make sure of that by making any sort of copy it needs to in order to prove this (unless you trick the TBAA pass, by say, assigning to a pointer at an offset from the original)

Things that may be an issue include:

1. Julia now has a generational GC, does it assume immutable intergenerational pointers cannot change and doesn't rescan for them?

If your type contains pointers to other non-isbits types, you are probably just as well off using a type. They are stored the same. Only `isbits` types get the special inline-storage behavior.

And then you don't mutate an immutable, but it would be good to document this restriction.

I don't expect that this distinction is necessarily permanent, it's just the current implementation.

 

2. Does the compiler generate the correct GC write barriers for unsafe operations and does it ignore immutables since they can't change?

No, it doesn't generate write barriers during calls to unsafe_store. Although if you are writing bits data inside the immutable, it hardly matters, since it doesn't need a write barrier to do that.

Yes, I was thinking of pointers, but your exclusion above cures that.

 

3. Do optimisations use copies of immutables as implied in the manual, so you don't mutate what you think you mutate?

Quite frequently. But if you mutate the array data, you mutate the array data, no questions asked. What you definitely shouldn't do is grab a pointer to your immutable (via pointer_from_objref) and try to mutate that (but trying to grab a pointer to an immutable should become an error in a future version of Julia anyways).

Again cached copies should be invalidated somehow.

There aren't really any cached copies aside from: the shared box you can get a reference to via pointer_to_objref (and which should never ever mutate), and the copy that exists in memory (which is handled by LLVM, and invalidated based on its TBAA pass, as needed)

 

4. Does the generated code *not* use copies where it normally might, since it doesn't matter if the object is immutable, so again you don't mutate what you expected to?

This is possible. But it would require LLVM to decide that the array slot itself couldn't have been re-assigned. Any intervening function call is typically enough to block optimization on assuming the array data hasn't changed. But TBAA (type-based alias analysis) is probably more likely to bite you here, since LLVM won't consider the offset store to affect the immutable type it lives inside.

Yes, which leads to the above problems.

 

1. and 2. require the GC to run during the lifetime of the mutated immutable, maybe you have been lucky and that hasn't happened ... yet.

3. and 4. the behaviour depends on what other use is made of the immutable, again maybe you have been lucky.

As for passing immutables to C, well was always going to be risky, C just doesn't understand immutable.  If as suggested by Jameson mutables are layout compatible with C then the faster the habit of using immutables for that dies the better.

There's nothing risky about it. Types have the same layout and the same behavior when getting passed to C regardless of whether they are declared mutable or immutable. The ccall code hardly even checks the `.mutable` field of the type (and that code hopefully will be deprecated soon, since it doesn't really benefit the user).

Its risky in that a mutation inside the C library won't be detected by AA either.

The ccall interface handles that concern for you (by passing a mutable copy rather than the immutable object).

 

Often, however, `immutable` provides a better representation of a C-struct and is therefore necessary for compatibility (unless the only usages of the type are by-pointer).

Thats probably the problematic design decision.  Hindsight (that wonderful thing) might suggest a C compatible definition for types that are passed to C (somewhat like the nebulous "standard layout" structs in C++11).

Not necessarily. The current design was arrived at after considering copying C struct semantics much more directly. Julia has a goal of making C-interop low-cost and easy, but that doesn't require copying C's behaviors outright. Rather, it requires defining a translation between the two concepts. Julia chose the later route.

[Mauro]

Thanks Lex, Jameson and Michael for this interesting discussion. I read
it a few times but still cannot quite follow:

Is the take-home that it is ok to mutate immutables? No repercussions
from Julia itself, just confused library users? This is not what the
manual suggests:
http://docs.julialang.org/en/latest/manual/types/#immutable-composite-types

Or is it just ok to mutate immutables stored in a mutable container?

Or is the take-home that for library interface code it's ok to mutate
immutables because there is no other way to mirror C-structs?

[Stefan Karpinski]

The problem with introducing structs / value types – which would be necessary for inline arrays of mutable values – is that they have completely different semantics from reference types, which would both complicate the language and torpedo Julia's current ability to write generic code that works across all kinds of values. This thread goes has a good discussion of why that's bad for generic code. That said, this is a recurring problem and I'd like to come up with a usable solution that doesn't introduce a whole new kind of object semantics to the language.

[Jameson Nash]

The take away should be that you can't mutate an immutable any more than you can mutate an integer. If you have an array [1,2,3] and you assign 4 to the 2, you don't change the value of 2, you change what value exists in the second position of the array.

As Stefan just mentioned, we do need to add some mechanisms for more easily creating them from existing ones.

[Mauro]

@Stefan, thanks that was a good read. I liked the example.

> The take away should be that you can't mutate an immutable any more than
> you can mutate an integer. If you have an array [1,2,3] and you assign 4 to
> the 2, you don't change the value of 2, you change what value exists in the
> second position of the array.

This applies to isbits immutables, right? Then the only surprise could
be when two bindings point to the same immutable:

julia> immutable A; a::Int end

julia> a = A(1)
A(1)

julia> c = a
A(1)

julia> mutate_immutable!!(a, 5)

julia> c
A(5)

Which couldn't happen with an Int.

[Jameson Nash]

I'm not sure what you are saying with that code. There is no possible way to define the function mutate_immutable!! such that it modified a or c.

You example works the same with Ints, but might be easier to see:

a=1
c=a
mutate_immutable!(a,5)
c === a === 1 # true

function mutate_immutable(x,y) ... end

[Michael Francis]

 

Re-reading the thread I believe what is being asked for is essentially a copy on write semantic for replacing an immutable item in a collection with a slightly modified original, not updating the immutable. So this isn't mutation at all ? 

In the case of an in-lined object it is safe to be an in-place update, in the case of a referenced immutable (non bits types) a new copy would have to be made with the modification and the reference updated.  

Otherwise I don't see how you can trust that there are not additional references to the same immutable as per @Mauro. 

When you give people pointers ... immutability runs away... good job we don't have string interning like Java

function change( x, y  )

  unsafe_copy!(pointer(x), pointer(y),3)

end

x = "hello"

y = "world"

change( x,y )

x == "worlo"

[Mauro]

> I'm not sure what you are saying with that code. There is no possible way
> to define the function mutate_immutable!! such that it modified a or c.

This does it:

immutable A; a::Int end

a = A(1)
c = a
A.mutable = true
a.a = 99
A.mutable = false
@show a, c #(A(99),A(99))
a===c===A(99) # true

I don't think a trick like this can be pulled with an Int. Although
putting it into a function errors. I also tried the pointer trick Simon
was using but couldn't get it to work.

[vav...@uwaterloo.ca]

I have now implemented a solution that follows Mauro's basic outline.  The difference is that the heavy lifting in my code is done at compile-time instead of run-time so that modifyField! will have good performance even in an inner loop.  Please see my announcement soon to be posted on julia-users, and please direct followups to that announcement.

Thanks,

Steve Vavasis

[Scott Jones]

On Tuesday, February 10, 2015 at 3:10:33 AM UTC+1, Jameson wrote:

On Mon Feb 09 2015 at 7:59:20 PM <ele...@gmail.com> wrote:

On Tuesday, February 10, 2015 at 1:25:15 AM UTC+11, Simon Danisch wrote:

Someone else has to answer this. I suppose it's not that bad. The immutables are getting copied into the heap allocated memory from the array, which should be fairly okay to mutate?!

But I seriously don't know the details on this. From a semantic viewpoint it's definitely not okay!

WARNING: if you use unsafe practices to mutate an immutable object you enter the realms of undefined behaviour where the compiler/runtime can do anything it likes from sending your details to Scammers-R-Us to eating your shorts.  Seriously if anything in the compiler optimisations, the runtime or garbage collector is changed to depend on immutables being, well, immutable, then it will cause unknown issues.  And there is no guarantee that this hasn't happened already, just your specific code hasn't hit it ... yet.

Semantically, modifying a field of the immutable in the array is the same as reading the whole thing, creating a new immutable with one field modified, and writing it back to the array. The only thing you are likely missing by hacking unsafe_store in this manner is a missing TBAA annotation. That's fairly unlikely to matter here though (more about this below).

Things that may be an issue include:

1. Julia now has a generational GC, does it assume immutable intergenerational pointers cannot change and doesn't rescan for them?

If your type contains pointers to other non-isbits types, you are probably just as well off using a type. They are stored the same. Only `isbits` types get the special inline-storage behavior.

 

2. Does the compiler generate the correct GC write barriers for unsafe operations and does it ignore immutables since they can't change?

No, it doesn't generate write barriers during calls to unsafe_store. Although if you are writing bits data inside the immutable, it hardly matters, since it doesn't need a write barrier to do that.

 

3. Do optimisations use copies of immutables as implied in the manual, so you don't mutate what you think you mutate?

Quite frequently. But if you mutate the array data, you mutate the array data, no questions asked. What you definitely shouldn't do is grab a pointer to your immutable (via pointer_from_objref) and try to mutate that (but trying to grab a pointer to an immutable should become an error in a future version of Julia anyways).

 

4. Does the generated code *not* use copies where it normally might, since it doesn't matter if the object is immutable, so again you don't mutate what you expected to?

This is possible. But it would require LLVM to decide that the array slot itself couldn't have been re-assigned. Any intervening function call is typically enough to block optimization on assuming the array data hasn't changed. But TBAA (type-based alias analysis) is probably more likely to bite you here, since LLVM won't consider the offset store to affect the immutable type it lives inside.

1. and 2. require the GC to run during the lifetime of the mutated immutable, maybe you have been lucky and that hasn't happened ... yet.

3. and 4. the behaviour depends on what other use is made of the immutable, again maybe you have been lucky.

As for passing immutables to C, well was always going to be risky, C just doesn't understand immutable.  If as suggested by Jameson mutables are layout compatible with C then the faster the habit of using immutables for that dies the better.

There's nothing risky about it. Types have the same layout and the same behavior when getting passed to C regardless of whether they are declared mutable or immutable. The ccall code hardly even checks the `.mutable` field of the type (and that code hopefully will be deprecated soon, since it doesn't really benefit the user).

Often, however, `immutable` provides a better representation of a C-struct and is therefore necessary for compatibility (unless the only usages of the type are by-pointer).

The updated documentation I mentioned above is part of my ccall-enhancement pull request (which makes calling c functions that require or return by-value structs actually work correctly). It can be read at:

https://github.com/JuliaLang/julia/blob/jn/ccall3/doc/manual/calling-c-and-fortran-code.rst.

I got a 404 error for that... can you please update the link?  What has happened in the meantime to your PR (what is the #?)

Thanks, Scott 

[Scott Jones]

Alignment (or at least, rounding up the size of a structure to a multiple of the requested alignment) I think is also needed... it's been around for ages, and is now part of the C11 standard.

Being able to set such attributes would be very useful for C/C++ interfaces.

[Isaiah Norton]

Alignment (or at least, rounding up the size of a structure to a multiple of the requested alignment)

 

See discussion here: https://github.com/JuliaLang/julia/issues/2555

I got a 404 error for that... can you please update the link?  

https://github.com/JuliaLang/julia/pull/7906 

[Scott Jones]

Thanks for the pointers!  BTW, is there anything to make C/C++ bitfields easy to deal with in Julia?  I haven't found anything yet :-(