Rust ownership model into Go (not a feature request, just a question)

[Gerard]

This is not a request! It's just a question.

What would be required to implement the Rust ownership model into Go? Syntax not much I think. Obviously a different compiler (probably LLVM) and new filename extensions. Also the functions that change a parameter should have this parameter prefixed with the keyword var. Also the C FFI would be changed and the Go FFI changed as well, since the code generates non-GC code. "Green threads", the key feature of Go, needs to be altered as well, possibly even removed. But what else?

Again, this is just a question because I am curious.

[Egon]

What would be the reason for including the ownership model? And how much of the ownership model? AFAIK the Rust ownership model only deals with what you can do with the values and how you can pass them around.

I suspect you could do a some part of the ownership checking as a static analyzer without changing the language or compiler. Probably you could add some constraints as comments (e.g. this value shouldn't be modified after this point).

+ Egon

[Gerard]

The reason is to not have GC so that it would be more "low level" and you don't need a runtime.

[Roberto Zanotto]

You have a runtime for goroutine scheduling and for other things anyway. If you want you can take away GC form Go, maybe reflection also, then replace goroutines with system threads and maybe you don't have a runtime anymore. But those are really valuable things that make Go fun to write, easy to read and to maintain. If you want to go more low level you can use C or Rust, I bet that managing memory manually or having to make the compiler happy with memory ownership rules is not as fun and productive as having an efficient and concurrent GC.

Also, the Go compiler and runtime are written in Go. What project are you working at that needs to be lower level than that?

[Gerard]

That's not the thing and i am aware of that. The question is just what is required to implement the ownership model?

[Ian Lance Taylor]

On Sat, Aug 22, 2015 at 5:25 AM, Gerard <gvds...@gmail.com> wrote:
>
> What would be required to implement the Rust ownership model into Go? Syntax
> not much I think. Obviously a different compiler (probably LLVM) and new
> filename extensions. Also the functions that change a parameter should have
> this parameter prefixed with the keyword var. Also the C FFI would be
> changed and the Go FFI changed as well, since the code generates non-GC
> code. "Green threads", the key feature of Go, needs to be altered as well,
> possibly even removed. But what else?

It would be a radical change to the language. The point of the Rust
ownership model is that the type system permits you to express various
properties of the value, such as who owns it. Go's type system is
very simple, and is unable to express anything even remotely like
that. Changing Go to adopt Rust's type system requires a lot more
than sprinkling "var" in a few places.

Ian

[Gerard]

Yes of course it would be radical. It wouldn't be Go anymore. That is very clear.

[David Chase]

I think this is a really reasonable question, and if I knew the answer I would have done it myself.  I don't know Rust, but I have looked at some uses of linear and affine types, and did work on a language design (Fortress) where this issue came up several times.  The way I would approach this problem for Go (or any other language) would be to see how much mileage I could get from comment-style annotations without changing the existing uncommented language.  That gives you a lot more "test data" to work with and the opportunity to show how the analysis could help "real programs".

You won't necessarily get rid of a garbage collector right away, but this information is useful for other things besides garbage collection.  People reason about this sort of thing all the time to be sure that their programs are correct, maybe scribble a few comments in the margin, and that's the end of it -- no automated verification, no further use of information except by other humans who may read it, may understand it, and might keep it up to date as the program changes.  The problem we wanted to solve for Fortress that kept stumping me (lack of time/imagination/skill) was splitting and recombining ownership in the face of recursive subdivision, and expressing how that works through interfaces (i.e., "this is abstract, but I promise you that its implementation is independent from any other, and by the way, this property splits and recombines").  The tricky base case is handling update of non-overlapping bit-fields within a word (i.e., fundamental properties of compare-and-swap or load-linked-store-conditional -- the "independence" is only true at a particular moment, but that is good enough).

I don't see how this necessarily requires giving up Green Threads or even modifying them.

[Roberto Zanotto]

I'm assuming you refer to my comment. I was reasoning about "why would you want to do that?" and OP clearly said "so that you don't need a runtime". My point was that *if that is the goal* GC and memory management are not the only things to reconsider, you would have to redesign the language from scratch (goroutine scheduler included).

[Gerard]

I don't think that the language semantics would change nor the language as such. But the implementation, yes, and the libraries as well.

[Giulio Iotti]

On Saturday, August 22, 2015 at 7:13:12 PM UTC+3, Roberto Zanotto wrote:

I'm assuming you refer to my comment. I was reasoning about "why would you want to do that?" and OP clearly said "so that you don't need a runtime". My point was that *if that is the goal* GC and memory management are not the only things to reconsider, you would have to redesign the language from scratch (goroutine scheduler included).

I also don't understand all this hatred for the runtime. Unlike many think, it's not a runtime as in managed code. It's like a library whose calls are transparently added by the compiler.

For that matter "int main()" in C is not the real entry point; one could say there is a runtime in C too. Go runtime just happens to do more things.

-- 

Giulio Iotti

https://twitter.com/dullboy

[Steven Blenkinsop]

Here's a common mistake highlighted in Effective Go:

func Serve(queue chan *Request) {
for req := range queue {
sem <- 1
go func() {
process(req) // Buggy; see explanation below.
<-sem
}()
}
}

This would be impossible under Rust's ownership/borrowing model. In Rust, it would be illegal to update req after it was captured by the goroutine. The type system would force you to make a copy of req and move that into the closure. In Go, this is something you have to keep in mind as a gotcha. Ideally, you'd use the race detector and kick yourself as you fix this, but there are other cases this kind of bug can occur that can't be caught by the race detector, such as capturing the loop variable in a deferred closure. Going through lists of gotchas in Go often makes you realize how much ownership and borrowing can help. On the other hand, it does significantly complicate the language.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Henrik Johansson]

Rusts concept of ownership is indeed powerful but it is a bit messy to wrap your head around. Maybe it gets easier with time, maybe some of the Servo devs are lurking on the list? I for one likes both the languages.

[crbrox]

As an aside comment, this particular case is detected by go vet, for 'go' or 'defer'. It could be detected before building.

[Steven Blenkinsop]

Yeah, go vet is a nifty tool for catching some of the more basic mistakes. I was thinking about whether it would be possible to extend the compiler's escape analysis to track aliasing, mutation, and mutual exclusion in order to enable static analysis which catches these things more broadly. You'd most like need a single-threaded mutex-like type to allow a lot of single-threaded code to pass analysis without needing a major architectural overhaul, though.

[Henry Adi Sumarto]

The one problem with Rust's memory model is the concept of lifetime. It adds a considerable complexity and yet -without it- the model will fail.

I don't know how Go's GC is implemented, but I think it is possible to build a smarter compiler that will automatically insert codes to free memory when it is obvious. The GC then only need to check the dubious ones. This semi-GC approach will lighten the workload of the GC.

[Dave Cheney]

The compiler already does that, it's called escape analysis.

[Michael Jones]

I knew that was coming. One question, are there cases of "escapes to heap" because we don't know what happens in called functions when we could know with bottom up instrumentation, such that the memory could be put on a delete now, or delete ASAP list?

I asked this a few years ago and the answer is that we're not that smart yet. Maybe we're close enough now.

On Aug 22, 2015 7:59 PM, "Dave Cheney" <da...@cheney.net> wrote:

The compiler already does that, it's called escape analysis.

[Axel Wagner]

Gerard <gvds...@gmail.com> writes:
> What would be required to implement the Rust ownership model into Go?

I think, the fastest way would be to get rid of go and use rust
instead. You would need to get rid of a lot of the stuff that makes go
fun to write and add all the stuff that (imo) makes rust not fun to
write. Rust is certainly the more sane starting point to get a rust/go
hybrid of this form.

> Syntax not much I think.

I disagree. You'd need to add some syntax for mutability
vs. non-mutability, for borrows and, most importantly for lifetimes (even
if it's syntax that's put into comments as suggested in another
message).

> Obviously a different compiler (probably LLVM)

I don't think so, necessarily. I mean, yes, in the sense that you can't
put it into the officially blessed go compilers, but you could fork and
modify gc to implement all of this.

> Also the C FFI would be changed and the Go FFI changed as well, since
> the code generates non-GC code.

I don't think the GC has any major influence on the FFI. Apart from "you
shouldn't pass go-allocated values to C". You might get rid of that
restriction with this (by also dealing with what happens to the stack,
because you can't rely on C code to grow it or otherwise handle the
relatively small go stacks)

> "Green threads", the key feature of Go, needs to be altered as well,
> possibly even removed.

No, not at all. There is a perceivable flavor of rust with green-threads
(in the form of libgreen) which got thrown out not because of ownership
or GC, but because rust emphasizes a native-speed and -feel FFI. If the
goal is to get rid of most of the runtime, then yes, you can't put
green threads in, but if, as originally proposed, you only want to
implement the ownership model, green threads can stay.

> But what else?

I think, that's already more than enough :) As said above, I think rust
would be the saner starting point, as you would need to primarily remove
stuff from it, instead of bolting stuff on. And as you anyway would lose
a lot of, if not most of the stuff that makes go worthwile to write over
rust, I don't see why you'd want to start with go, or take go into
account at all. Why not just use rust which already has all the features
you want? To understand and answer the question properly, one would need
to know more about your motivation for it, i.e. what's the precise
difference between what you are proposing and just using rust and what
is the difference between go and what you are proposing? Particular in
regards to the latter there seems to be some confusion, i.e. do you only
want to implement the ownership model or do you want to make the runtime
smaller?

[Gerard]

Obviously you are right. Rust is however completely different and compared with Go it's highly complex. The thing is, the ownership model is relatively easy to understand and, from a user perspective, easy to use. They have done a great job implementing it.

[Axel Wagner]

Gerard <gvds...@gmail.com> writes:
> Obviously you are right. Rust is however completely different and compared
> with Go it's highly complex.

I agree.

> The thing is, the ownership model is relatively easy to understand
> and, from a user perspective, easy to use.

I disagree.

I regard the ownership model of rust as the only thing preventing me
from using it, because it is *such a chore* to use. i.e. the complexity
of rust (as in, the cognitive overhead of using it) is about 80%
the ownership model for me.

So again, I think you should make clearer what your goals and
motivations are. Because to me it looks like you want to take the worst
part of rust and bolt it onto go, sacrificing the best part of go (its
simplicity and ease of use) in the process. And I don't understand why
you would rather reimplement complicated stuff for go, than rip out
stuff from rust, which sounds much easier to me. I'd think that if you
use rust without generics and algebraic datatypes and re-add libgreen
you have roughly the feature set of "go with the rust ownership model"
as a language (yes, there are still differences, but I would consider
them reatively minor). And you can probably start using that today.

[Joubin Houshyar]

A restricted application of the idea to channels is probably worth a look and should not radically change the language.

Ian

[r...@golang.org]

There seems to be tension between splices and the backing store they use/share and ownership. I would think the the debate with the Rust compiler about this would be long and frustrating. 

 

On Saturday, August 22, 2015 at 8:25:57 AM UTC-4, Gerard wrote:

[Ian Lance Taylor]

On Sat, Aug 22, 2015 at 8:49 PM, Michael Jones <m...@wearality.com> wrote:
>
> I knew that was coming. One question, are there cases of "escapes to heap"
> because we don't know what happens in called functions when we could know
> with bottom up instrumentation, such that the memory could be put on a
> delete now, or delete ASAP list?

I'm not sure this is quite what you are asking, but we do now have
bottom up instrumentation. When you compile a Go package, the export
information records information for each function that takes a pointer
(slice, etc.) parameter, recording whether the parameter escapes to
the heap, or escapes to the result variables. That information is
then used by other packages that import that one. So we build up the
escape analysis information bottom up.

Ian

[Ian Lance Taylor]

We could implement a very restricted subset without a major change to
the language: if you have a pointer in a local variable, and you send
that pointer on a channel, then we could set the local variable to
nil. It might be interesting to see how many programs that would
break.

Of course, this wouldn't change anything if you send a global variable
on a channel, if you send a heap pointer on a channel (c <- h.p), etc.

Ian

[Michael Jones]

Yes, thank you!

—
Michael Jones, CEO • mic...@wearality.com • +1 650 656-6989
Wearality Corporation • 289 S. San Antonio Road • Los Altos, CA 94022

[Joubin Houshyar]

Not sure post-send nil assignment sans tagging the address would really get us that subset. Copy is happening anyway so why not just basic copy-on-send semantics?

Crazy idea that just popped in my head is to treat address spaces as GC authority zones with boundary conditions defined by Chan IO. So somewhere between where we are now and Erlang's micro-heaps (which are process centric). 

[Brendan Tracey]

We could implement a very restricted subset without a major change to
the language: if you have a pointer in a local variable, and you send
that pointer on a channel, then we could set the local variable to
nil.  It might be interesting to see how many programs that would
break.

If the pointer is a "read-only" variable, then there's no problem with sending it over a channel multiple times. I also write a lot of code where the parallelism is over slice subsets, i.e. of the form

for i := 0; i < len(x)/size; i++ {
      work <- x[i*size:(i+1)*size]
}

(obviously the real code deals with rounding issues).

I'm just pointing out that I think a lot of my code would break with that semantics.

[Joubin Houshyar]

I ended thinking about this last night and have something tentatively called Communicable Stack Objects (CSO as complement to CSP). 

No syntactic changes and no type system changes. Naturally semantics and runtime changes. Introduces a new memory space giving set {Heap, Stack, and Communicable Stack} memory objects. Heaps are micro-segmented and 1:1 to goroutine; stack is the same stack; CSO is the underlying shared memory with 'stewardship' of MO changing between the messaging layer and the particular goroutine in whose stack the CSO 'visits' per -> / <- ops. CSOs transition to SO on access after -> and return on stack.  Should hugely simplify GC to boot.

I'll write it up when I get a chance.

Ian

[Gerard]

On Sunday, August 23, 2015 at 2:14:56 PM UTC+2, Axel Wagner wrote:

Gerard <gvds...@gmail.com> writes:
> Obviously you are right. Rust is however completely different and compared
> with Go it's highly complex.

I agree.

> The thing is, the ownership model is relatively easy to understand
> and, from a user perspective, easy to use.

I disagree.

I regard the ownership model of rust as the only thing preventing me
from using it, because it is *such a chore* to use. i.e. the complexity
of rust (as in, the cognitive overhead of using it) is about 80%
the ownership model for me.

I still find the ownership model easy to use. It takes a bit of understanding but then it just works.
 

So again, I think you should make clearer what your goals and
motivations are. Because to me it looks like you want to take the worst
part of rust and bolt it onto go, sacrificing the best part of go (its
simplicity and ease of use) in the process. And I don't understand why
you would rather reimplement complicated stuff for go, than rip out
stuff from rust, which sounds much easier to me. I'd think that if you

use rust without generics and algebraic datatypes and re-add libgreen
you have roughly the feature set of "go with the rust ownership model"
as a language (yes, there are still differences, but I would consider
them reatively minor). And you can probably start using that today.

Starting with Rust is also an approach. I haven't thought of that. Btw, I like generics. They solve real, although minor, issues. But as I said before, Rust is complicated compared with Go. I think they would like to be more formal and also want to implement functional language features into the language. There is nothing wrong with that, but the end result is that there are lots of things that don't make sense, again compared with Go. I like that line ending without ' ; ' but if you do use ' ; ' it should be used everywhere. That's how I look at it. Ok, here are three complaints, the module system incl name spaces notation of Rust sucks, the testing functionality is too flexible and the iterator functionality requires too much mental baggage.

What I do like about Rust is the (functional language) immutability by default. You know which functions can alter data. Only using a pointer the way Go does it, leaves that open. And the part of which I did start this thread with, the ownership model. This approach of dealing with resources is revolutionary. I know the Go designers are not gonna implement this because of obvious reasons and I also don't think it's fair to request this functionality, and that's why I asked what is required to implement it. I think that "bolting it onto Go", for instance with comments isn't the right approach. Therefore it is too fundamental. I however don't think the language semantics of Go as such would be heavily influenced with the ownership model.

[Henry Adi Sumarto]

The problem is that you can't just take Rust's ownership concept. In order for the memory model to work correctly, you also need to implement borrowing and lifetime as well. These are the three pillars of Rust's memory model and you need three of them for the model to work. Out of the three, lifetime is the most problematic and it affects the way you should structure and design your program.

Then, in order for the memory model to work with concurrency, you may need a bunch of Rust's other features as well, such as explicit mutability, etc. Having done all that, you will probably end up with a language that is more or less similar to Rust.

[Joubin Houshyar]

draft is up. https://github.com/alphazero/csp-cso-chipmunk-  

- interesting that the end-user can be told the white-lie of "everything is copied on sends"

- seems to favor canonical REST interactions. (this was a surprise)

- concurrent work on composites seems to favor data-flows.

- some interesting gc and distribution (think OTP) bits not addressed

A modal-reference type (builtin/capability-model) would comprehensively address globals, pointers, etc. Also would allow for lifting the hood ala unsafe and permit more idiomatic forms around composites. 

critical feedback is welcomed.

[Roberto Zanotto]

Sorry then, I made a wrong assumption :)

It still holds that if the goal is: "don't need a runtime", GC is not the only thing, you have to rape the language pretty badly.

On Saturday, August 22, 2015 at 5:32:37 PM UTC+2, Gerard wrote:

That's not the thing and i am aware of that. The question is just what is required to implement the ownership model?

On Saturday, August 22, 2015 at 5:23:36 PM UTC+2, Roberto Zanotto wrote:

You have a runtime for goroutine scheduling and for other things anyway. If you want you can take away GC form Go, maybe reflection also, then replace goroutines with system threads and maybe you don't have a runtime anymore. But those are really valuable things that make Go fun to write, easy to read and to maintain. If you want to go more low level you can use C or Rust, I bet that managing memory manually or having to make the compiler happy with memory ownership rules is not as fun and productive as having an efficient and concurrent GC.

Also, the Go compiler and runtime are written in Go. What project are you working at that needs to be lower level than that?

On Saturday, August 22, 2015 at 5:07:21 PM UTC+2, Gerard wrote:

The reason is to not have GC so that it would be more "low level" and you don't need a runtime.

On Saturday, August 22, 2015 at 3:43:52 PM UTC+2, Egon wrote:

On Saturday, 22 August 2015 15:25:57 UTC+3, Gerard wrote:

This is not a request! It's just a question.

What would be required to implement the Rust ownership model into Go? Syntax not much I think. Obviously a different compiler (probably LLVM) and new filename extensions. Also the functions that change a parameter should have this parameter prefixed with the keyword var. Also the C FFI would be changed and the Go FFI changed as well, since the code generates non-GC code. "Green threads", the key feature of Go, needs to be altered as well, possibly even removed. But what else?

Again, this is just a question because I am curious.

What would be the reason for including the ownership model? And how much of the ownership model? AFAIK the Rust ownership model only deals with what you can do with the values and how you can pass them around.

I suspect you could do a some part of the ownership checking as a static analyzer without changing the language or compiler. Probably you could add some constraints as comments (e.g. this value shouldn't be modified after this point).

+ Egon