Proposal: read-only slices

2,846 views
Skip to first unread message

Brad Fitzpatrick

unread,
May 13, 2013, 8:01:57 PM5/13/13
to golang-dev

Brad Fitzpatrick

unread,
May 13, 2013, 8:19:59 PM5/13/13
to Tylor Arndt, golang-dev
I knew the community would help me pick a color.

I picked the smallest (least offensive?) option: a period isn't too many pixels. 

I don't really care, though.



On Mon, May 13, 2013 at 5:16 PM, <voidl...@gmail.com> wrote:
At first read, I really like everything but the syntax's use of the period/dot and I am eager for something like this to exist in Go.

In my opinion the period is too inconspicuous-
Given: t := make([]T, 10)
Here are a few ideas:

var vt[]#T = t
var vt[]$T = t
var vt[]:T = t
var vt[]~T = t
 
--
 
---
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

minux

unread,
May 13, 2013, 8:36:37 PM5/13/13
to Brad Fitzpatrick, golang-dev
So this is the (part of?) revised contribution process that you mentioned in another post?
"The Go Language Change Proposal"

Will readonly slices support slicing? (i think it should, but it's not mentioned anywhere in
the docs)
i'm wondering if string could be treated as a [].byte?
that is, at universal block, we have a declaration:
type string [].byte

how does the readonly slices interact with the reflect system?
will it introduce a new Kind or just a readonly flag for the slice kind (so as to pave the way
for future extensions like readonly map)?

Brad Fitzpatrick

unread,
May 13, 2013, 8:42:10 PM5/13/13
to minux, golang-dev
On Mon, May 13, 2013 at 5:36 PM, minux <minu...@gmail.com> wrote:

On Tue, May 14, 2013 at 8:01 AM, Brad Fitzpatrick <brad...@golang.org> wrote:
So this is the (part of?) revised contribution process that you mentioned in another post?
"The Go Language Change Proposal"

No, unrelated.
 
Will readonly slices support slicing? (i think it should, but it's not mentioned anywhere in
the docs)

Yes. Updated doc.
 
i'm wondering if string could be treated as a [].byte?
that is, at universal block, we have a declaration:
type string [].byte

No.

A string has an additional guarantee: that nobody, ever, will mutate those bytes.  The doc explicitly says that [].byte just means you can't.  But somebody else might right now, or some time later.

Nobody wants to get rid of "string".
 
how does the readonly slices interact with the reflect system?
will it introduce a new Kind or just a readonly flag for the slice kind (so as to pave the way
for future extensions like readonly map)?

I'm open to suggestions.  You can imagine reflect.Value.Cap panicing if it's a read-only slice, for instance.  I don't know whether keeping the same Kind would cause problems, if the representation changes.

Alternatively, we could say that the representation of a read-only slice is the same as a slice, and cap() still works, but the cap(v) == len(v) always.


mortdeus

unread,
May 13, 2013, 8:58:31 PM5/13/13
to golan...@googlegroups.com
I think `[[]]T` is best. Its easy to type in editors, especially with editors that auto close braces. Adding it to the language grammar will be trivial. It also works well with the variable length and multidimensional array syntax.

http://play.golang.org/p/ioYulis11h    


On Monday, May 13, 2013 7:01:57 PM UTC-5, Brad Fitzpatrick wrote:

Ian Lance Taylor

unread,
May 13, 2013, 8:59:46 PM5/13/13
to Brad Fitzpatrick, minux, golang-dev
On Mon, May 13, 2013 at 5:42 PM, Brad Fitzpatrick <brad...@golang.org> wrote:
>>
>> how does the readonly slices interact with the reflect system?
>> will it introduce a new Kind or just a readonly flag for the slice kind
>> (so as to pave the way
>> for future extensions like readonly map)?
>
>
> I'm open to suggestions. You can imagine reflect.Value.Cap panicing if it's
> a read-only slice, for instance. I don't know whether keeping the same Kind
> would cause problems, if the representation changes.
>
> Alternatively, we could say that the representation of a read-only slice is
> the same as a slice, and cap() still works, but the cap(v) == len(v) always.

One way to view this question is: are we introducing a new type, or
are we introducing a special case of an interface type? In one sense
you are proposing a polymorphic interface type that supports the
operations
Len() int
At(int) T
Slice(int, int) [].T
If we view this as an interface type, then reflect should return the
original type. If not, then it could be a new Kind, or it could be
Slice with a new field.

Either way we need to think about converting a value out of the type
and back to []byte and/or string? Is that ever possible? Does it
always involve a copy? If we adopt the interface approach, should we
support x.([]byte) and x.(string)?

Ian

mortdeus

unread,
May 13, 2013, 9:03:19 PM5/13/13
to golan...@googlegroups.com
Also the syntax works if we wanted to create a read only slice from a r/w slice. ro := rw[[1:4]] 

Brad Fitzpatrick

unread,
May 13, 2013, 9:04:58 PM5/13/13
to Ian Lance Taylor, minux, golang-dev
On Mon, May 13, 2013 at 5:59 PM, Ian Lance Taylor <ia...@golang.org> wrote:
On Mon, May 13, 2013 at 5:42 PM, Brad Fitzpatrick <brad...@golang.org> wrote:
>>
>> how does the readonly slices interact with the reflect system?
>> will it introduce a new Kind or just a readonly flag for the slice kind
>> (so as to pave the way
>> for future extensions like readonly map)?
>
>
> I'm open to suggestions.  You can imagine reflect.Value.Cap panicing if it's
> a read-only slice, for instance.  I don't know whether keeping the same Kind
> would cause problems, if the representation changes.
>
> Alternatively, we could say that the representation of a read-only slice is
> the same as a slice, and cap() still works, but the cap(v) == len(v) always.

One way to view this question is: are we introducing a new type, or
are we introducing a special case of an interface type?

I wouldn't make it an interface.

a) that'd have a larger representation, right?
b) I don't want my callees to get access to the underlying []byte with a x.([]byte).
 
 In one sense
you are proposing a polymorphic interface type that supports the
operations
    Len() int
    At(int) T
    Slice(int, int) [].T
If we view this as an interface type, then reflect should return the
original type.  If not, then it could be a new Kind, or it could be
Slice with a new field.

In what sense do you view our current send-only channels?  Aside: I'd love a method operator world, but we're not there.
 
Either way we need to think about converting a value out of the type
and back to []byte and/or string?  Is that ever possible?  Does it
always involve a copy?  If we adopt the interface approach, should we
support x.([]byte) and x.(string)?

I would be fine with s, ok := view.(string), but that means we need to know it was originally a string, which suggests a more interface-y representation, or representation tricks.

I never want to see a cheap view.([]byte).

I'd be totally happy with string([].byte) always making a copy, just like string([]byte)

Brad Fitzpatrick

unread,
May 13, 2013, 9:05:26 PM5/13/13
to mortdeus, golang-dev
Let's please ignore syntax for now.

It's very subjective, and subjectively, I think [[]] is disgusting.


Brad Fitzpatrick

unread,
May 13, 2013, 9:19:53 PM5/13/13
to Ian Lance Taylor, minux, golang-dev
On Mon, May 13, 2013 at 5:59 PM, Ian Lance Taylor <ia...@golang.org> wrote:

Either way we need to think about converting a value out of the type
and back to []byte and/or string?  Is that ever possible?  Does it
always involve a copy? 

You could also imagine that it always makes a copy by default, just like today:

     b := []byte("stuff")
     var s string = string(b)   // calls runtime.slicebytetostring

But instead, slicebytetostring only conditionally makes a copy, depending on whether &b[0] is from a string (GC information or whether it's in a memory region dedicated to storing string contents)

Brad Fitzpatrick

unread,
May 13, 2013, 9:22:24 PM5/13/13
to Tylor Arndt, golang-dev
Inline image 1


On Mon, May 13, 2013 at 6:17 PM, <voidl...@gmail.com> wrote:
For what its worth, I think I like var vt[]:T = t best.
It is (subjectively) aesthetically clean and in Go has connotations related to slicing [x : y].


On Monday, May 13, 2013 7:16:33 PM UTC-5, voidl...@gmail.com wrote:
At first read, I really like everything but the syntax's use of the period/dot and I am eager for something like this to exist in Go.

In my opinion the period is too inconspicuous-
Given: t := make([]T, 10)
Here are a few ideas:

var vt[]#T = t
var vt[]$T = t
var vt[]:T = t
var vt[]~T = t
 

On Monday, May 13, 2013 7:01:57 PM UTC-5, Brad Fitzpatrick wrote:
ya_vote.jpg

Brad Fitzpatrick

unread,
May 13, 2013, 9:25:17 PM5/13/13
to Tylor Arndt, golang-dev
Sorry, I know that's condescending, but I'd like to keep this focused for now.

Syntax is a huge distraction, and I'm sure it will be a big enough topic later.

Let's figure out how this would work first.
ya_vote.jpg

mortdeus

unread,
May 13, 2013, 9:36:35 PM5/13/13
to golan...@googlegroups.com
If we introduce Immutability for slices we might as well introduce a universal immutability mechanism for all types.     


On Monday, May 13, 2013 7:01:57 PM UTC-5, Brad Fitzpatrick wrote:

Gustavo Niemeyer

unread,
May 13, 2013, 9:38:53 PM5/13/13
to Brad Fitzpatrick, golang-dev
Thanks Brad, that looks interesting. A few comments:

- The comparison with channels seems distracting
- Behavior when mutating the slice? Undefined, or defined with memory barriers?
- Should the printf family of functions change their formatting argument?
- Does it support + in a similar way to string?
- append?
- ~[]byte seems more clear, and reads properly ("view of slice of T",
not "slice of view of T").

Will send further comments if I can think of other questions.


gustavo @ http://niemeyer.net

Brad Fitzpatrick

unread,
May 13, 2013, 9:41:48 PM5/13/13
to Gustavo Niemeyer, golang-dev
On Mon, May 13, 2013 at 6:38 PM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
On Mon, May 13, 2013 at 9:01 PM, Brad Fitzpatrick <brad...@golang.org> wrote:
> Design doc for adding read-only slices to Go:
>
> https://docs.google.com/a/golang.org/document/d/1UKu_do3FRvfeN5Bb1RxLohV-zBOJWTzX0E8ZU1bkqX0/edit#heading=h.2wzvdd6vdi83
>
> Comments welcome.

Thanks Brad, that looks interesting. A few comments:

- The comparison with channels seems distracting

It exists to say that there's precendent.  We already have restricted types.
 
- Behavior when mutating the slice? Undefined, or defined with memory barriers?

Undefined. Like I said in the doc, we already permit data races with []byte. This is no different.
 
- Should the printf family of functions change their formatting argument?

We're not changing the standard library in Go 1.x.
 
- Does it support + in a similar way to string?

No. Will clarify. It's more of a slice than a string.
 
- append?

It's not mutable, so no.

Andrew Gerrand

unread,
May 13, 2013, 9:41:39 PM5/13/13
to mortdeus, golang-dev

On 13 May 2013 18:36, mortdeus <mort...@gocos2d.org> wrote:
If we introduce Immutability for slices we might as well introduce a universal immutability mechanism for all types.     

It may be worth considering immutability for maps, although I haven't personally felt the need for them.

I don't like the idea of consty pointers, though.

Andrew Gerrand

unread,
May 13, 2013, 9:42:46 PM5/13/13
to Gustavo Niemeyer, Brad Fitzpatrick, golang-dev

On 13 May 2013 18:38, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
- The comparison with channels seems distracting

I don't think it's a distraction. It provides an important comparison with regard to conversions (eg, you can assign a bidirectional chan to a unidirectional one).

mortdeus

unread,
May 13, 2013, 9:46:25 PM5/13/13
to golan...@googlegroups.com
Another thing to consider is if we have an immutable slice for types with methods. Whats the rules for methods for pointers? What if one of the slice elements modifies itself by referencing itself via a globally scoped pointer. Do we create a snapshot copies of the slice, and if so how can we update immutable slice to reflect changes made to the original mutable slice?

These are the kinds of things that need to be thought about.

Andrew Gerrand

unread,
May 13, 2013, 9:49:18 PM5/13/13
to mortdeus, golang-dev
On 13 May 2013 18:46, mortdeus <mort...@gocos2d.org> wrote:
Another thing to consider is if we have an immutable slice for types with methods. Whats the rules for methods for pointers? What if one of the slice elements modifies itself by referencing itself via a globally scoped pointer. Do we create a snapshot copies of the slice, and if so how can we update immutable slice to reflect changes made to the original mutable slice?

These are the kinds of things that need to be thought about.

It's not an issue because type T[] and T.[] are different types. They don't share any methods. All the usual conversion rules apply.

Andrew

Brad Fitzpatrick

unread,
May 13, 2013, 9:50:34 PM5/13/13
to mortdeus, golang-dev
On Mon, May 13, 2013 at 6:46 PM, mortdeus <mort...@gocos2d.org> wrote:
Another thing to consider is if we have an immutable slice for types with methods. Whats the rules for methods for pointers?

Then the receiver is a pointer to a read-only slice, and it's still not allowed to mutate its receiver.
 
What if one of the slice elements modifies itself by referencing itself via a globally scoped pointer.

We still let you have a data race, as described in the doc.
 
Do we create a snapshot copies of the slice,
 
No.

Gustavo Niemeyer

unread,
May 13, 2013, 9:55:03 PM5/13/13
to Brad Fitzpatrick, golang-dev
On Mon, May 13, 2013 at 10:41 PM, Brad Fitzpatrick <brad...@golang.org> wrote:
> On Mon, May 13, 2013 at 6:38 PM, Gustavo Niemeyer <gus...@niemeyer.net>
>> - Behavior when mutating the slice? Undefined, or defined with memory
>> barriers?
>
> Undefined. Like I said in the doc, we already permit data races with []byte.
> This is no different.

The behavior of []byte is defined with explicit synchronization. If
you say undefined, then this is different.

>> - append?
>
> It's not mutable, so no.

I suppose it should still be mentioned, at least to define the
cross-compatibility with []byte appending.


gustavo @ http://niemeyer.net

Brad Fitzpatrick

unread,
May 13, 2013, 9:57:54 PM5/13/13
to Gustavo Niemeyer, golang-dev
On Mon, May 13, 2013 at 6:55 PM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
On Mon, May 13, 2013 at 10:41 PM, Brad Fitzpatrick <brad...@golang.org> wrote:
> On Mon, May 13, 2013 at 6:38 PM, Gustavo Niemeyer <gus...@niemeyer.net>
>> - Behavior when mutating the slice? Undefined, or defined with memory
>> barriers?
>
> Undefined. Like I said in the doc, we already permit data races with []byte.
> This is no different.

The behavior of []byte is defined with explicit synchronization. If
you say undefined, then this is different.

It's no different than a []byte:

-- we're not inserting atomics for you when you set/get a slice
-- we let you have buggy code with data races
-- we let you use synchronization correctly if you do want concurrent read/write
 
>> - append?
>
> It's not mutable, so no.

I suppose it should still be mentioned, at least to define the
cross-compatibility with []byte appending.


Will do.
 

David Symonds

unread,
May 13, 2013, 11:05:25 PM5/13/13
to Brad Fitzpatrick, golang-dev
I think append should work. If a read-only slice is the first argument
it'll always reallocate, and it works as normal if it's the second
argument (expanded with ...).

Ian Lance Taylor

unread,
May 13, 2013, 11:27:20 PM5/13/13
to Brad Fitzpatrick, minux, golang-dev
On Mon, May 13, 2013 at 6:04 PM, Brad Fitzpatrick <brad...@golang.org> wrote:
>>
>> One way to view this question is: are we introducing a new type, or
>> are we introducing a special case of an interface type?
>
>
> I wouldn't make it an interface.
>
> a) that'd have a larger representation, right?

Right, because they would also record the original type.


>> In one sense
>> you are proposing a polymorphic interface type that supports the
>> operations
>> Len() int
>> At(int) T
>> Slice(int, int) [].T
>> If we view this as an interface type, then reflect should return the
>> original type. If not, then it could be a new Kind, or it could be
>> Slice with a new field.
>
>
> In what sense do you view our current send-only channels?

Channels feel a bit different to me: they remain channels, but
converting to a send-only or receive-only channel eliminates certain
operations. Converting []byte to [].byte changes the nature of the
value, since it no longer has a capacity. Converting string to
[].byte changes the meaning of the string significantly--e.g., range
and slice operations act differently.

As far as reflection goes, a chanType has a channel direction field
that indicates the directions that the channel supports (SendDir,
RecvDir, BothDir). We could add a field to sliceType.

Ian

Dan Kortschak

unread,
May 13, 2013, 11:41:51 PM5/13/13
to Ian Lance Taylor, Brad Fitzpatrick, minux, golang-dev
On Mon, 2013-05-13 at 20:27 -0700, Ian Lance Taylor wrote:
> Converting []byte to [].byte changes the nature of the
> value, since it no longer has a capacity.

Why in principle should a [].byte not have a capacity? The elements of
the view are not writable, but why should a holder of the [].byte not be
able to expand the length to the capacity? They can make no change to
the underlying data, all they can do is see it. The capacity means
something slightly different here, but it still seems to have a meaning.

Dan

mortdeus

unread,
May 13, 2013, 11:46:55 PM5/13/13
to golan...@googlegroups.com, Ian Lance Taylor, Brad Fitzpatrick, minux
cap([].byte) == len([].byte)

Gustavo Niemeyer

unread,
May 13, 2013, 11:51:34 PM5/13/13
to Dan Kortschak, Ian Lance Taylor, Brad Fitzpatrick, minux, golang-dev
Pretty consistently, in the vast majority of cases where a slice is
extended one mutates its content. While someone can break that rule,
it sounds like a view not being extensible is a sane and practical
promise.


gustavo @ http://niemeyer.net

Ian Lance Taylor

unread,
May 14, 2013, 12:09:26 AM5/14/13
to mortdeus, golan...@googlegroups.com
On Mon, May 13, 2013 at 6:46 PM, mortdeus <mort...@gocos2d.org> wrote:
> Another thing to consider is if we have an immutable slice for types with
> methods. Whats the rules for methods for pointers? What if one of the slice
> elements modifies itself by referencing itself via a globally scoped
> pointer. Do we create a snapshot copies of the slice, and if so how can we
> update immutable slice to reflect changes made to the original mutable
> slice?

We don't make a snapshot copy. I don't see how that could be possible.

In Brad's proposal, a value of type [].T is not immutable. It simply
does not support any modification operations. As the doc says, this
is similar to a send-only channel. The existence of a value V of chan
<- int doesn't mean that there is no way to receive values from the
channel; it only means that you can't receive a value from V.

This proposal is analogous to a const char * pointer in C/C++. The
array to which such a pointer points may change, but it may not be
changed through that pointer.


This is perhaps something of a drawback to this proposal. It provides
a read-only view of a slice, but it in no way ensures that the slice
does not change. This has some utility as documentation: a function
or method that takes a parameter of type [].T is certain to not modify
the elements of that slice (barring the use of unsafe). And of course
we get the ability to pass a string value to a parameter of type
[].byte. But this is not the same as immutability. It is only the
inability to change the elements of a slice through a particular view
of that slice.

The proposal suffers from the strstr problem of C. The C function
strstr may be written as either "char *strstr(const char *, const char
*);" (the standard way) or as "char *strstr(char *, const char *);" or
as "const char *strchr(const char *, const char *);". The first
requires an unsafe operation in the implementation of strstr: a
conversion from const char * to char *; it also provides an
underhanded way to convert from "const char *" to "char *". The
second means that strstr may not be used to search inside a string
literal. The third means that the caller of strstr may need to
perform the unsafe operation. The equivalent problem in Go shows up
in bytes.Split. We can use [].byte as the first parameter, but then
we have to return [][].byte which may not be what the caller needs.
Or we can use []byte as the first parameter, but then we can't pass in
a string literal or a [].byte, although the function does not need to
modify the slice it is passed.

I suspect that these kinds of issues may be inevitable when trying to
provide a read-only version of a type in a language like Go, or C,
that does not support function overloading and does not support any
sort of polymorphism (e.g., func Split(T {[]byte, [].byte}, [].byte) T
which is intended to mean that the first parameter is either []byte or
[].byte and the result parameter has the same type as the first
parameter).


Note that there is another approach, which is to not make read-only
variants of types, but to instead have read-only values. The way this
works is to say that a value of type []T may happen to be immutable,
meaning that the values in the slice may not be modified. We permit
the free assignment of string to []byte, but the resulting value is
immutable. Any attempt to modify it fails, at compile time if
possible but otherwise at runtime. We implement this by, say,
negating the length field. Any attempt to access a value in the slice
uses the absolute value of the length field in the bounds check. Any
attempt to change a value in the slice uses the length field
unchanged, thus failing the bounds check. This approach loses the
documentation aspect of Write([].byte). The advantage is that the
type system does not become more complex. The disadvantage is that
more errors are detected only at runtime.

Ian

Daniel Theophanes

unread,
May 14, 2013, 12:16:29 AM5/14/13
to golan...@googlegroups.com
I'm sure performance could be improved with this.

It feels just different enough to feel confusing. I find the idea of a read-only map interesting. I think you've pulled a small thread on a much larger concept. Or tried to overly generalize a much smaller one (convert []byte <-> string w/o copy).

I don't like introducing such a specialized type.
Another option to creating a new read-only type is to create a region that can lock the mutability of a slice or un-lock the mutability of a string.

bb := make([]byte, 10)
lock(bb)
defer unlock(bb)
s := string(bb[:3]) // No copy.
bb[2] = 'a' // Compile error or panic.

However, I'm sure you've thought through all these options already.
-Daniel

Ian Lance Taylor

unread,
May 14, 2013, 12:28:02 AM5/14/13
to Dan Kortschak, Brad Fitzpatrick, minux, golang-dev
That's true, it could work that way. I think one of Brad's goals is
the copy-free conversion of string to [].byte, and of course a string
does not have a capacity. But I suppose we could implement that by
setting the capacity of the [].byte to the length of the string.
Still, it's hard to see any particular use for a capacity for [].byte.

Ian

Dan Kortschak

unread,
May 14, 2013, 12:37:55 AM5/14/13
to Ian Lance Taylor, Brad Fitzpatrick, minux, golang-dev
On Mon, 2013-05-13 at 21:28 -0700, Ian Lance Taylor wrote:
> Still, it's hard to see any particular use for a capacity for [].byte.

Yes, I was struggling to find one. None appeared that were convincing.

Dan

Lucio De Re

unread,
May 14, 2013, 12:57:35 AM5/14/13
to golang-dev
I was going to wait for the .1 version of the proposal before reading
it and the discussion so far has been quite instructive. Still, what
follows may be a little off the mark.

Firstly, I agree with Ian's view that having immutable underlying data
is more constructive than having a mechanism to pick an immutable view
of potentially mutable underlying data. The copy free conversion of a
string to a slice seems small gain for a new language construct and
the reverse operation, the copy free conversion of a [].byte to a
string does not make much sense as it would allow the creation of
mutable strings, From an embedded-system perspective, immutability
ought to be a property of memory elements, not their representation;
we may want to follow that path instead.

I wonder if the above means that once we create a view ([].byte) of a
memory region, we effectively lock that region from alteration from
anywhere? If then we extend the len() of that view, we necessarily
extend (or reduce) the read-only boundary, etc. We could then have a
view operator similar, but not necessarily visually similar to slicing
to perform such an operation on any memory area and perhaps we ought
to be looking in that direction? The compiler would not pick this up,
but the language constructs to intercept erroneous accesses to the
locked region at runtime could be added to Go and would be useful to
some developers. It may also be an interesting approach to locking.

As for using channels as an example of prior art, if assigning a
bidirectional channel to a unidirectional one effectively blocks
access to one direction, this is an artificial situation, in an ideal
situation, one would explicitly request to change the properties of
the channel instead. I hope we're not using this misfeature as
encouragemt for future development.

Lastly, in my opinion the possibility of races occurring when
manipulating complex data elements needs to be addressed, if necessary
by adding language constructs to protect critical regions. But using
the existence of such possibilities should not be deemed as permission
to continue the trend, on the contrary, in a situation where race
conditions can be eliminated or at least alleviated (having read-only
data reduces the risk of races in some respects), one ought to
consider any opportunity to eliminate race conditions entirely.

Mine are probably not the best educated comments, I'm sure, I hope
they are helpful.

Lucio.

Brad Fitzpatrick

unread,
May 14, 2013, 1:35:09 AM5/14/13
to Ian Lance Taylor, minux, golang-dev
On Mon, May 13, 2013 at 8:27 PM, Ian Lance Taylor <ia...@golang.org> wrote:
>>>>  In one sense
>> you are proposing a polymorphic interface type that supports the
>> operations
>>     Len() int
>>     At(int) T
>>     Slice(int, int) [].T
>> If we view this as an interface type, then reflect should return the
>> original type.  If not, then it could be a new Kind, or it could be
>> Slice with a new field.
>
>
> In what sense do you view our current send-only channels?

Channels feel a bit different to me: they remain channels, but
converting to a send-only or receive-only channel eliminates certain
operations.  Converting []byte to [].byte changes the nature of the
value, since it no longer has a capacity.

Okay, but what if it still had a capacity, in its representation and/or capability (via cap(v)), but just one you couldn't usefully use, since you couldn't append onto it?

Then it's just like channels: the same representation, but the type system is restricting the available operations. That's all I care about.

I don't care about cap() so much (I just see it as unnecessary and useless on [].T, so could go), but I do care about capabilities.  I would love an operator method world where there were interfaces like SliceReader[T] with Len() int and At(int) T, and SliceWriter[T] adding Addr(int) *T.  But in lieu of that, and before generics, I like being able to restrict access with something like [].T.
 
Converting string to
[].byte changes the meaning of the string significantly--e.g., range
and slice operations act differently.

But we can make these copy-free too:

   var v [].byte = ...
   for i, s := range string(v) { ... }
 
   var s string = ...[
   var i, s := range []byte(s) { ... }


I would naturally expect that a range over a [].byte acts like a range over a []byte instead of a string, so I don't view that as a deficiency.

Brad Fitzpatrick

unread,
May 14, 2013, 1:37:44 AM5/14/13
to Daniel Theophanes, golang-dev
On Mon, May 13, 2013 at 9:16 PM, Daniel Theophanes <kard...@gmail.com> wrote:
I'm sure performance could be improved with this.

It feels just different enough to feel confusing. I find the idea of a read-only map interesting. I think you've pulled a small thread on a much larger concept. Or tried to overly generalize a much smaller one (convert []byte <-> string w/o copy).

I don't like introducing such a specialized type.
Another option to creating a new read-only type is to create a region that can lock the mutability of a slice or un-lock the mutability of a string.

bb := make([]byte, 10)
lock(bb)
defer unlock(bb)
s := string(bb[:3]) // No copy.
bb[2] = 'a' // Compile error or panic.

However, I'm sure you've thought through all these options already.

That is not the path to performance.

That seems extremely complex in both the compiler and runtime.  I think it would end up slower, and more confusing.

I'd rather use mprotect and unsafe if I wanted to play those games.

Brad Fitzpatrick

unread,
May 14, 2013, 1:52:57 AM5/14/13
to Ian Lance Taylor, Dan Kortschak, minux, golang-dev
On Mon, May 13, 2013 at 9:28 PM, Ian Lance Taylor <ia...@golang.org> wrote:
On Mon, May 13, 2013 at 8:41 PM, Dan Kortschak
<dan.ko...@adelaide.edu.au> wrote:
> On Mon, 2013-05-13 at 20:27 -0700, Ian Lance Taylor wrote:
>> Converting []byte to [].byte changes the nature of the
>> value, since it no longer has a capacity.
>
> Why in principle should a [].byte not have a capacity? The elements of
> the view are not writable, but why should a holder of the [].byte not be
> able to expand the length to the capacity? They can make no change to
> the underlying data, all they can do is see it. The capacity means
> something slightly different here, but it still seems to have a meaning.

That's true, it could work that way.  I think one of Brad's goals is
the copy-free conversion of string to [].byte,

Yes. This is the main part I care about.

and of course a string
does not have a capacity.  But I suppose we could implement that by
setting the capacity of the [].byte to the length of the string.
Still, it's hard to see any particular use for a capacity for [].byte.

I have no opinion on whether a [].T should have an accessible or underlying capacity.  It seems useless, so I'm inclined to reduce size and say that it's just Ptr+Len, like a string.

But eliminating cap([].T) isn't something I feel strongly about.

I feel more strongly about the ability to change slice capacities (https://code.google.com/p/go/issues/detail?id=1642), which would also apply here, if we decide that [].T has cap.

Christoph Hack

unread,
May 14, 2013, 1:54:36 AM5/14/13
to golan...@googlegroups.com, Daniel Theophanes
Your immutable slice proposal sounds very similar to strings in general. Therefore I was thinking if it's a good idea to change the string type to be an alias for [].byte in the first place (similar to rune and byte in the current specification). Have you thought about that?

Brad Fitzpatrick

unread,
May 14, 2013, 1:56:42 AM5/14/13
to Christoph Hack, golang-dev, Daniel Theophanes
On Mon, May 13, 2013 at 10:54 PM, Christoph Hack <chri...@tux21b.org> wrote:
Your immutable slice proposal sounds very similar to strings in general. Therefore I was thinking if it's a good idea to change the string type to be an alias for [].byte in the first place (similar to rune and byte in the current specification). Have you thought about that?

Jan Mercl

unread,
May 14, 2013, 3:56:23 AM5/14/13
to Brad Fitzpatrick, golang-dev
On Tue, May 14, 2013 at 2:01 AM, Brad Fitzpatrick <brad...@golang.org> wrote:
> Comments welcome.

I think the proposal is pushing the legitimately desired
optimizations, belonging to the compiler, into the language instead.
That's why I don't like it.

-j

Daniel Morsing

unread,
May 14, 2013, 4:37:11 AM5/14/13
to Brad Fitzpatrick, golang-dev
On Tue, May 14, 2013 at 2:01 AM, Brad Fitzpatrick <brad...@golang.org> wrote:
If this change is made, why would you ever use strings in function parameters?

It seems to me that strings and read-only slices are very close to
each other semantically, and it will probably introduce the same sort
of confusion that already exists between make, new and composite
literals.

I'd rather put effort into making strings generate less garbage while
keeping their current semantics. That will probably include the
compiler doing some analysis, but another idea that I haven't seen
floated yet is changing the runtime representation such that strings
converted from slices are copy on write whenever the slice data is
changed. You could probably smuggle some of the mechanism in via the
bounds check.

Regards,
Daniel Morsing

>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

David Crawshaw

unread,
May 14, 2013, 8:29:01 AM5/14/13
to Brad Fitzpatrick, golang-dev
I've been reading through some of my code trying to work out what
benefit I would get from the extra cognitive load of [].byte. So far
it doesn't seem worth it.

A common problem in my code is when I see a []byte, I don't know if
some other reference will change it under me. The string type helps
readability by solving that problem.

In theory, [].byte removes the risk of action-at-a-distance when
calling API functions that take []byte, but API functions are well
documented so I find the benefit there minimal.

On Mon, May 13, 2013 at 8:01 PM, Brad Fitzpatrick <brad...@golang.org> wrote:
> Design doc for adding read-only slices to Go:
>
> https://docs.google.com/a/golang.org/document/d/1UKu_do3FRvfeN5Bb1RxLohV-zBOJWTzX0E8ZU1bkqX0/edit#heading=h.2wzvdd6vdi83
>
> Comments welcome.
>

Robin

unread,
May 14, 2013, 11:23:23 AM5/14/13
to Jan Mercl, Brad Fitzpatrick, golang-dev
I feel the same. Minimizing []byte to string and string to []byte
conversions is fundamental to performance, but this optimization should
belong to the compiler.

For instance, the compiler could analyse the function below and use an
internal [].byte representation of a, since no write is performed.

``
func f(a []byte) bool {
for _, b := range a {
if b == 'x' {
return true
}
}
}
``

I just don't think this belongs in the language.

Also as Daniel Morsing mentioned it would be interesting to evaluate a
copy on write solution. What drawbacks and advantages would it provide.

The beauty of Go is in it's simplicity. I'd much rather have a simple
language and push some of the complexity into the compiler during the
various optimization phases.

Regards /Robin

Brad Fitzpatrick

unread,
May 14, 2013, 9:42:36 AM5/14/13
to Daniel Morsing, golang-dev
On Tue, May 14, 2013 at 1:37 AM, Daniel Morsing <daniel....@gmail.com> wrote:
On Tue, May 14, 2013 at 2:01 AM, Brad Fitzpatrick <brad...@golang.org> wrote:
If this change is made, why would you ever use strings in function parameters?

If you wanted the promise that strings give you.  See https://code.google.com/p/go/issues/detail?id=5376

Namely, a string gives you a promise that it's a valid and unchanging reference for life.

It seems to me that strings and read-only slices are very close to
each other semantically, and it will probably introduce the same sort
of confusion that already exists between make, new and composite
literals.

Pointers confuse users too, though.
 
I'd rather put effort into making strings generate less garbage while
keeping their current semantics. That will probably include the
compiler doing some analysis,

I've filed bugs for several of these (e.g. https://code.google.com/p/go/issues/detail?id=2205), but some in particular are viewed as changing the language too much to do as an optimization, either because it's too surprising that one implementation is significantly faster than another, or because it's not entirely an equivalent transformation. For instance, the compiler optimization describd in https://code.google.com/p/go/issues/detail?id=2632 was deemed too risky because it meant the compiler removed a guarantee from the language that the strconv package (or whatever else package) might have relied on: that you could do two passes over the data and get the same bytes.  Saying [].byte or []byte means you don't rely on that property.
 
but another idea that I haven't seen
floated yet is changing the runtime representation such that strings
converted from slices are copy on write whenever the slice data is
changed. You could probably smuggle some of the mechanism in via the
bounds check.

That seems unlikely to fly, considering the referencing-counting-vs-GC story.

Brad Fitzpatrick

unread,
May 14, 2013, 10:01:55 AM5/14/13
to David Crawshaw, golang-dev
On Tue, May 14, 2013 at 5:29 AM, David Crawshaw <craw...@google.com> wrote:
I've been reading through some of my code trying to work out what
benefit I would get from the extra cognitive load of [].byte. So far
it doesn't seem worth it.

Think of it like a normal []byte, but with fewer rights.  You don't get confused by seeing a lowercase type or lowercase func.

It will actually simplify docs like http://godoc.org/code.google.com/p/leveldb-go/leveldb/db#Iterator which fairly says the validity period, but sadly has to tell you your access rules.
 
A common problem in my code is when I see a []byte, I don't know if
some other reference will change it under me. The string type helps
readability by solving that problem.

A string is still good if you want to retain it and/or need a guarantee of immutability.

Right now there's no clean way to write a function which just wants to read some bytes and not hold on to them and not allocate in the case of some callers having []byte and some callers having string (which have two incompatible promises).

Put another way, we have:

type []T interface {
    Len() int
    Cap() int
    At(i int) T
    Addr(i int) *T
}

so:

type []byte interface {
    Len() int
    Cap() int
    At(i int) byte
    Addr(i int) *byte
}

type string interface {
    Len() int
    At(i int) byte
    ImmutableValidHandleForLife()
}

But we don't have a common subset of both of those:

type ??? interface {
    Len() int
    At(i int) byte
}

So whenever you read a function signature:

func Foo(b []byte)
func Bar(s string)

You don't know what capabilities Foo or Bar actually wants. 

It's as if the language had io.ReadCloser and io.ReadSeeker, but no io.Reader, so everybody takes an io.ReadSeeker but documents:

// ReadAll reads from r until an error or EOF and returns the data it read.
// Oh FYI, we take a ReadSeeker but don't ever seek on it. Don't worry. But if all you have
// is something that can Read, be sure to convert it into a ReadSeeker first somehow.
func ReadAll(r io.ReadSeeker) ([]byte, error)


Niklas Schnelle

unread,
May 14, 2013, 10:19:46 AM5/14/13
to golan...@googlegroups.com
Could you explain what the reasoning is to add a special read only slice type over a general const reference concept? Espcieally since a broader const concept allows for example
to have (r cont *UserStruct) Foo() methods that make it clear that the method can not alter the struct pointed to by r. As I see it const referencing makes at least as much
sense for pointers to structs and map references as it makes for slices?

So we could have all of the following instead of only read only slices:
var s const *SomeStruct;
var m const map[string]int = someothermap;
var p const *unit32;
and so on

Brad Fitzpatrick

unread,
May 14, 2013, 10:23:05 AM5/14/13
to Niklas Schnelle, golang-dev
const means too many things in C and C++.

That word is forbidden for being too broad and vague.



Aaron France

unread,
May 14, 2013, 10:25:34 AM5/14/13
to Brad Fitzpatrick, Niklas Schnelle, golang-dev
Hi,

So we will define const as "immutable". Why is [].byte so special?


--
 
---
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/Y7j4B2r_eDw/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

Brad Fitzpatrick

unread,
May 14, 2013, 10:26:55 AM5/14/13
to Aaron France, Niklas Schnelle, golang-dev
Bytes and strings are very common.

Same reason it's special in append and copy: http://golang.org/ref/spec#Appending_and_copying_slices

roger peppe

unread,
May 14, 2013, 10:57:10 AM5/14/13
to Brad Fitzpatrick, golang-dev
i don't think you'll be able to change any of the signatures of the
functions in the strings or bytes packages, otherwise you'll
break code that relies on the specific types of the functions.

e.g.
var f func(string, string) int = strings.Count


On 14 May 2013 01:01, Brad Fitzpatrick <brad...@golang.org> wrote:
> Design doc for adding read-only slices to Go:
>
> https://docs.google.com/a/golang.org/document/d/1UKu_do3FRvfeN5Bb1RxLohV-zBOJWTzX0E8ZU1bkqX0/edit#heading=h.2wzvdd6vdi83
>
> Comments welcome.
>

Brad Fitzpatrick

unread,
May 14, 2013, 10:59:51 AM5/14/13
to roger peppe, golang-dev
On Tue, May 14, 2013 at 7:57 AM, roger peppe <rogp...@gmail.com> wrote:
i don't think you'll be able to change any of the signatures of the
functions in the strings or bytes packages, otherwise you'll
break code that relies on the specific types of the functions.

e.g.
var f func(string, string) int = strings.Count

Yes. As mentioned in the doc.

roger peppe

unread,
May 14, 2013, 11:05:12 AM5/14/13
to Brad Fitzpatrick, golang-dev
ah, sorry, i missed that bit.

Daniel Theophanes

unread,
May 14, 2013, 11:22:40 AM5/14/13
to Brad Fitzpatrick, golang-dev
Something like that would add complexities to the compiler and
depending on the specification, the runtime. But I would prefer it to
be there then in the user code base: "Oh, this package uses a
read-only slice, but this user uses a normal slice uhhh, ok, compiler,
I'll play your games and convert." They are too similar and it would
happen. It would not be the end of the world, but I can't help but
think it would be a pain point.

I don't think there is a functional difference between:
* read-only slices
* using unsafe to convert between bytes and strings
* "locking" a byte normal slice (similar to how a channel can be
declared send or receive only).

In each case you are telling the compiler, "hold on, I know what I'm
doing here". In each case it is fairly easy to change the backing
store from a different pointer. We just don't like using "unsafe" in
normal code.

I can't see the cost being worth the benefit for introducing a
read-only slice type. I'd rather introduce a concept that a normal
slice can be "locked". In other words, what you proposed, but without
the additional type.

-Daniel

Niklas Schnelle

unread,
May 14, 2013, 11:33:49 AM5/14/13
to Brad Fitzpatrick, golang-dev, Niklas Schnelle

Then s/ const / ro /g for read only. This could be very helpful in writing stricter code in general. After all it is pretty close to the const pointer concept in c/c++ that has been proven to be well worth the effort in many cases.

Brad Fitzpatrick

unread,
May 14, 2013, 11:37:37 AM5/14/13
to Daniel Theophanes, golang-dev
On Tue, May 14, 2013 at 8:22 AM, Daniel Theophanes <kard...@gmail.com> wrote:
Something like that would add complexities to the compiler and
depending on the specification, the runtime. But I would prefer it to
be there then in the user code base:  "Oh, this package uses a
read-only slice, but this user uses a normal slice uhhh, ok, compiler,
I'll play your games and convert."

You can't convert from [].T to []T, except using unsafe.

I don't think there is a functional difference between:
 * read-only slices
 * using unsafe to convert between bytes and strings

Not having to use unsafe to do normal things and not corrupting memory is a big feature.

Every time I tell somebody to use unsafe to make a fake StringHeader, I cry.

Daniel Theophanes

unread,
May 14, 2013, 11:42:36 AM5/14/13
to Brad Fitzpatrick, golang-dev
> You can't convert from [].T to []T, except using unsafe.

Yes, in that direction you would have to copy to "convert".

> Every time I tell somebody to use unsafe to make a fake StringHeader, I cry.

I agree, I don't like that either.

Brad Fitzpatrick

unread,
May 14, 2013, 11:46:42 AM5/14/13
to Daniel Theophanes, golang-dev
On Tue, May 14, 2013 at 8:42 AM, Daniel Theophanes <kard...@gmail.com> wrote:
> You can't convert from [].T to []T, except using unsafe.

Yes, in that direction you would have to copy to "convert".

Which is great, because it's explicit and you see the cost.

And the other direction is visually light-weight because it's free:

var ts []T = ...
var v [].T = ts

No conversion necessary  Just assignment.


Maxim Khitrov

unread,
May 14, 2013, 12:34:56 PM5/14/13
to Brad Fitzpatrick, Daniel Theophanes, golang-dev
But the problem is that for a long time you'll have old functions that
take []byte, which is treated and documented as read-only, while new
ones take [].byte, which has its read-only status enforced by the
compiler. If the latter wants to call the former, it will have to
either make a copy or resort to unsafe hacks. For example:

// MyFunc doesn't modify b.
func MyFunc(b [].byte) bool {
return bytes.Contains(b, []byte("go")) // Not in Go1
}

I like the idea of cheap conversion between []byte and string, in one
form or another, but I don't think a new type is the answer. Declaring
a function parameter as a string also adds some information about the
expected content. It usually means "this value is text in UTF-8
encoding, not data." Read-only slices would blur this distinction.

Brad Fitzpatrick

unread,
May 14, 2013, 12:50:14 PM5/14/13
to Maxim Khitrov, Daniel Theophanes, golang-dev
On Tue, May 14, 2013 at 9:34 AM, Maxim Khitrov <m...@mxcrypt.com> wrote:
On Tue, May 14, 2013 at 11:46 AM, Brad Fitzpatrick <brad...@golang.org> wrote:
> On Tue, May 14, 2013 at 8:42 AM, Daniel Theophanes <kard...@gmail.com>
> wrote:
>>
>> > You can't convert from [].T to []T, except using unsafe.
>>
>> Yes, in that direction you would have to copy to "convert".
>
>
> Which is great, because it's explicit and you see the cost.
>
> And the other direction is visually light-weight because it's free:
>
> var ts []T = ...
> var v [].T = ts
>
> No conversion necessary  Just assignment.

But the problem is that for a long time you'll have old functions that
take []byte, which is treated and documented as read-only, while new
ones take [].byte, which has its read-only status enforced by the
compiler. If the latter wants to call the former, it will have to
either make a copy or resort to unsafe hacks. For example:

// MyFunc doesn't modify b.
func MyFunc(b [].byte) bool {
        return bytes.Contains(b, []byte("go")) // Not in Go1
}

Yes, that is unfortunate.  I instead imagine a world where there are overlapping standard library versions, even within a package, with each file declaring its preferred standard library version.  v2's v1 compatibility standard library could be implemented in terms of v2's.

As opposed to a full-scale cutover ala Python 3 / Perl 6 where everybody converts all at once!  (hah)

I like the idea of cheap conversion between []byte and string, in one
form or another, but I don't think a new type is the answer.

What is?
 
Declaring
a function parameter as a string also adds some information about the
expected content. It usually means "this value is text in UTF-8
encoding, not data."

In my experience, that is rarely the case. And similarly, I've never wanted to range over a string's UTF-8 codepoints.

Gustavo Niemeyer

unread,
May 14, 2013, 12:59:24 PM5/14/13
to Maxim Khitrov, Brad Fitzpatrick, Daniel Theophanes, golang-dev
On Tue, May 14, 2013 at 1:34 PM, Maxim Khitrov <m...@mxcrypt.com> wrote:
> // MyFunc doesn't modify b.
> func MyFunc(b [].byte) bool {
> return bytes.Contains(b, []byte("go")) // Not in Go1
> }

It is true that while people don't start using it, there will be a
period while certain things cannot be used without conversion. That
said, this is no different than the current situation with
[]byte/string. The point is that eventually we'll be in a better
situation across the board, when people do use the feature that
addresses exactly that problem.

> It usually means "this value is text in UTF-8
> encoding, not data." Read-only slices would blur this distinction.

This distinction is already blurred. The string type does not mean that.


gustavo @ http://niemeyer.net

Alex Belanger

unread,
May 14, 2013, 3:43:12 PM5/14/13
to Gustavo Niemeyer, Maxim Khitrov, Brad Fitzpatrick, Daniel Theophanes, golang-dev
Would it be possible to simply add to the string package a method that returns a []byte to the underlaying string in memory?
I keep hearing the problem is with copying the data. Isn't a slice just a pointer with a `cap` and `len` ?

We could fake it.

I don't understand the motivation for a new type when the language can still support it well.

Brad Fitzpatrick

unread,
May 14, 2013, 3:45:27 PM5/14/13
to Alex Belanger, Gustavo Niemeyer, Maxim Khitrov, Daniel Theophanes, golang-dev
Arguably you also don't understand the current language, so it's hard to understand change proposals.

Ingo Oeser

unread,
May 14, 2013, 6:31:33 PM5/14/13
to golan...@googlegroups.com
I like the basic idea, but would really limit it to byte slices. Maybe even call it byteview and have a separate type like string for it.

A package byteviews could implement then the read-only portions of strings and bytes package.

Instead of providing read-only slices in general and even for slices of pointers, providing byteviews to everything that contains neither padding nor pointers might be useful.

Russ Cox

unread,
May 28, 2013, 4:46:31 PM5/28/13
to Brad Fitzpatrick, golang-dev
On Mon, May 13, 2013 at 8:01 PM, Brad Fitzpatrick <brad...@golang.org> wrote:
[Pretty version of this reply here.]

The main part of this proposal is the introduction of the idea of a read-only slice. 

The restrictions on cap(x) and on x[i:j] not extending x beyond len(x) seem not fundamental to the idea of a read-only slice. They are either inherited from string (but we could define cap(x) = len(x) when converting a string to a read-only slice) or they are intended as performance hacks. Either way, I am going to ignore those, so we can focus on the implications of read-only slices.

The restriction on &x[i] seems to me a serious handicap. Go has pointers, and one of the important roles of pointers is to allow introducing cheap shorthand variables, such as p := &x[i] followed by multiple uses of p. I am going to ignore this restriction too. One way to ignore it is to introduce read-only pointers and read-only maps, so that all the reference types can be made read-only.

That is, I am assuming the following (and nothing else):
 * new type readonly []T
 * new type readonly map[K]V
 * new type readonly *T
 * implicit conversion from []T to readonly []T is allowed
 * implicit conversion from string to readonly []byte is allowed
 * if x is readonly []T, then x[i] = v and append(x, …) are disallowed
 * if x is readonly map[K]V, then x[i] = v and delete(x, i) are disallowed
 * if x is readonly *T, then *x = v is disallowed

Although this introduces more new types than the proposal, having the new types (particularly readonly *byte) means breaking less existing code, so the evaluation should, if anything, lean toward favoring the proposal. (The new read-only pointer does prompt some questions about receiver types and method sets, but I'll ignore those.)

I implemented this: hg clone -U ro2 code.google.com/r/rsc-go-readonly.

Documentation clarity

It certainly helps users of a library to see documentation that makes clear which argument slices may be modified by a call. For example, this pair of functions in math/big have different semantics, now called out by the annotation:

func (z *Int) SetBits(abs []Word) *Int
func (z *Int) SetBytes(buf readonly []byte) *Int

In package bufio, a Scanner’s SplitFunc may edit its input, noted by the lack of a readonly qualifier. But in contrast, in package io, Writer’s Write can be defined to take a readonly []byte instead of a []byte.

These are improvements.

Duplication and triplication

The main linguistic motivation is to be able to write functions that operate equally well on []byte and string. The change delivers on this promise for functions that use strings only as input.

For example, in unicode/utf8,
func DecodeRune(s []byte) (r rune, size int)
func DecodeRuneInString(s string) (r rune, size int)
are replaced by
func DecodeRune(s readonly []byte) (r rune, size int)

Similarly, many functions common to packages bytes and strings can be merged, so that (pardon the syntax):
func bytes.Index(s, sep []byte) int
func strings.Index(s, sep string) int
are replaced by
func strings.Index(s, sep readonly []byte) int
Even better, code that before said bytes.Index(s, []byte(“foo”)) can now use bytes.Index(s, “foo”). Global variables that exist only to avoid repeated []byte conversions can be retired.

As another example, since io.Writer now defines Write to take a readonly []byte, so io.WriteString is gone: w.Write(“hello, world”) is legal.

These are certainly improvements. However, there are some places where the change does not deliver on its promise to eliminate duplication, and there are places where the change actually requires more duplication.

Duplicated functions returning subslices of their input cannot be merged. For example,
func bytes.TrimSpace(s []byte) []byte
func strings.TrimSpace(s string) string
cannot be replaced by
func strings.TrimSpace(s readonly []byte) readonly []byte
because often the caller really does need a []byte, for future mutation, or string, for use with string operations like ==, <, >, or +. A readonly []byte is not good enough, so the duplication must be preserved. 

Part of the proposal was to convert parsing functions, such as strconv.Atoi or time.Parse, to accept readonly []byte instead of string. If such a parser wants to trim spaces from its input, it cannot call bytes.TrimSpace, nor can it call strings.TrimSpace. There would have to be a third package, robytes.TrimSpace, full of slicing functions for readonly []byte. Instead of needing two copies of some functions, we now need three. Fundamentally, it is impossible to write a general function that wraps a slice operation. This was true before, but the problem is exacerbated by having three types now instead of two.

This triplication would also apply to the functions in package regexp, which are often used during parsing.

One possibility would be to assume some solution that defines ==, <, >, and + on readonly []byte and then adopt the convention that slicing operations always return readonly []byte. I started down that road but quickly ran into a problem: if Split returns readonly []byte, that result cannot be passed to bytes.Join or strings.Join, so we’re back to needing a third function robytes.Join if we want to preserve the property that Split and Join work as inverses.

The “slicing returns readonly []byte” convention does not tell us what Join should return. Perhaps:
func bytes.Join(x [][]byte, sep readonly []byte) []byte
func strings.Join(x []string, sep readonly []byte) string
func robytes.Join(x []readonly []byte, sep readonly []byte) []byte
?

Note that [][]byte and []string do not convert implicitly to []readonly []byte, just as []byte cannot convert implicitly to []interface{}, so these functions must have different input types as well as different output types.

Immutability and memory allocation

Another motivation was performance: many functions are written to take a string argument so that string constants can be passed, but by changing them to take a readonly []byte, they would accept a []byte as well, saving the caller a conversion. This does work in many cases: the most obvious case is Write methods, already mentioned above.

However, changing functions taking string to functions taking readonly []byte does cause possibly unnecessary allocations on the error path. For example, os.Open might be changed to take a readonly []byte so that constructed []byte paths can be passed directly. However, it returns an error implementation recording the file name for use in error messages. When the input is of type string, recording the string is essentially free. However, when the input is of type readonly []byte, the implementation cannot assume the data is globally immutable, so it must make a copy to store in the error result. This results in unnecessary copying when passing in a string, unless the implementation uses a special representation for readonly []byte to optimize the reverse conversion. Parsers like strconv.Atoi are subject to the same problem.

The fundamental difference between string and readonly []byte is that a string’s contents are globally immutable, while a readonly []byte’s contents are only locally immutable. For example, bufio.ReadSlice might return a readonly []byte to protect the content of its internal buffer, but the visible contents still change at the next read operation. 

The need to make copies to protect against non-local mutability arises in contexts other than error results. It happens any time a function needs to save an argument, such as for use in a cache or as a map key.

The immutability of strings also enables an optimization in Map, Replace, and ToXxx: if the input string is not changing, the original can be returned instead of a copy. Because global immutability is no longer guaranteed, the no-copy optimization cannot be used in an implementation taking a readonly []byte.

Loss of generality

I did not push the conversion far enough to run into this, but there are a few common methods in the Go tree that involve returning slices. At the least:
Bytes() []byte
Peek(n int) []byte
Different implementations might choose to make the result readonly or not. Packages testing for the interface will only find it if the readonly bits match. 
x, ok := x.(interface{ Peek(int) []byte })
will not find the readonly form, and vice versa. To the extent that such bifurcation happens, the libraries become less powerful.

I did see this in some tests. Before, bytes.TrimSpace and bytes.ToUpper both had the same signature:
func TrimSpace(x []byte) []byte
func ToUpper(x []byte) []byte
but now they have different signatures:
func TrimSpace(x []byte) []byte
func ToUpper(x readonly []byte) []byte
One can no longer use the same function variable for both: function variables become less general and therefore less powerful. TrimSpace/ToUpper is not likely to come up in practice outside testing, but other examples may.

Another example of loss of generality is interfaces with some “read-only” and some “read-write” methods, such as this code from package sort:

type Interface interface {
Less(i, j int) bool
Len() int
Swap(i, j int)
}

func Sort(data Interface) bool {
… code using Less, Len, and Swap …
}

func IsSorted(data Interface) bool {
… code using only Less and Len …
}

type IntSlice []int
func (x IntSlice) Less(i, j int) bool { return x[i] < x[j] }
func (x IntSlice) Len() int { return len(x) }
func (x IntSlice) Swap(i, j int) { x[i], x[j] = x[j], x[i] }

func Ints(a []int) { // invoked as sort.Ints
Sort(IntSlice(a))
}

func IntsAreSorted(a []int) bool {
return IsSorted(IntSlice(a))
}

If we introduce read-only slices, it makes sense to change IntsAreSorted to take a readonly []int. However, the readonly []int cannot be converted to the non-read-only IntSlice. Instead, a new ReadOnlyIntSlice must be defined, adding duplication, and either ReadOnlyIntSlice.Swap will need to panic, or IsSorted will need to be changed to take a new ReadOnlyInterface so that ReadOnlyIntSlice need not implement Swap, adding either a magic panic or more duplication.

Loss of string semantics

Another problem that comes up with blindly rewriting string arguments to have type readonly []byte is that the latter type has fewer operations available. Parsers in particular often use == or switch on substrings. A readonly []byte type would probably have to be accompanied by language changes to allow string-like comparisons on that type (if not on slices generally), as well as a language change to allow selecting the kind of iteration in a range loop (iterate bytes or iterate runes). That's not a long-term show-stopper, but it's worth noting.

Conclusions

The readonly []byte (or [].byte) proposal has definite benefits in documentation clarity and removal of string/[]byte duplication. However, it cannot remove all the duplication, and in some important cases it triggers triplication instead. It causes specialization mismatches between code written using readonly and code not written using readonly. And although eliminating memory allocations was an explicit goal, the use of readonly []byte removes from unified code the ability to depend on immutability of arguments, possibly causing additional memory allocations.

Should we add readonly to Go 2?

No, I don’t believe we should add readonly []byte to the language in this form. It does solve some problems, but it introduces at least as many new problems. Perhaps a new design will come along that eliminates all the duplication and avoids the new allocations, but assuming not, I think we should keep going with the current type system.

If we were designing Go from scratch today, it might be interesting to define string as an alias for readonly []byte, so that there would only be two types, one a subtype of the other; global immutability would not be a guarantee, and no code would be written with that assumption. However, we cannot easily escape the fact of Go’s history: much code exists today that depends on the global immutability of strings. We cannot reasonably introduce this mutable definition even in Go 2: the task of updating existing programs will be too onerous.

Should we add readonly to Go 1.x?

No, I don't believe we should add readonly to Go 1.x unless we know it will fit nicely with the (presumably compatibility-breaking) readonly in Go 2. We don't.

Russ

Dave Cheney

unread,
May 28, 2013, 6:16:23 PM5/28/13
to Russ Cox, Brad Fitzpatrick, golang-dev
Thank you for your detailed and considered response. 


--

Nigel Tao

unread,
May 28, 2013, 11:22:21 PM5/28/13
to Russ Cox, Brad Fitzpatrick, golang-dev
On Wed, May 29, 2013 at 6:46 AM, Russ Cox <r...@golang.org> wrote:
> Duplicated functions returning subslices of their input cannot be merged.
> For example,
> func bytes.TrimSpace(s []byte) []byte
> func strings.TrimSpace(s string) string
> cannot be replaced by
> func strings.TrimSpace(s readonly []byte) readonly []byte

Such functions could conceivably return (low, high int) instead and
the caller could re-slice. This obviously wouldn't be as nice to use,
though, barring syntactic sugar.


> However, changing functions taking string to functions taking readonly
> []byte does cause possibly unnecessary allocations on the error path.

We could introduce an immutable built-in function that upgrades
(local) readonly-ness to (global) immutability. Implementation-wise,
immutable(x) wouldn't allocate/copy if x was already immutable (e.g.
for slice-typed x, if x pointed to a certain area of memory).


> If we were designing Go from scratch today, it might be interesting to
> define string as an alias for readonly []byte,

A possible migration path is to make str an alias for readonly []byte
and remove the string type for Go 2 ("literals" would have type str).
Go 1 code would not compile instead of behaving subtly differently at
runtime. Go 1 behavior (except for for/range) could be kept by using
immutable(str(b)). Can gofix help here?


But all that's just thinking out loud. I don't have a concrete
proposal, and agree that we shouldn't change Go 1.x until we know what
we want for Go 2.

Kevin Gillette

unread,
May 29, 2013, 6:52:23 PM5/29/13
to golan...@googlegroups.com
I agree with @jan and @robin in that the perceived motivation for this is at least partly encroaching on what today's escape analysis, or that in the near future could certainly do to minimize the number of []byte <-> string copies; from what I've gleaned, the missing piece of this puzzle is compiler code that actually uses the results of existing analysis to prevent copies. Moreover, there'd be some places where static analysis would do a better job than the proposal, even with the proposal's own types, since _semantically_ you can't pass a [].byte into a string without a copy. Because of this, I'm only considering the aspects of the proposal that are not concerned with solving the perceived garbage problem.

Also, there's something to be said for hiding data behind functions and channels for readability' sake: Go currently has a simple "if you can see it, you can modify it" expectation that this proposal would violate, since it'd allow data to (at least under some circumstances) be safely used as its own API. The proposal does not use alternative syntax for accessing roslices, which is a good thing, though it does mean that readonly and mutable slices are only distinguishable at the declaration site, decreasing readability. Right now, exported package level variables can be modified by any importers; except in special circumstances (like an exported var of a non-exported type), this proposal wouldn't prevent anyone from replacing the referenced slice outright -- it'd just prevent modification of the original.  Semantically, there wouldn't be much difference if that global var was the only reference, yet many programmers may see this proposal and assume it applies to the slice value in addition to the backing array.

There's also a notable potential for abuse here. bufio.Reader's ReadSlice method doesn't return a copy of the internal buffer, but that's not a problem, since the Reader will blindly overwrite those contents on the next method call; in designing such an API, many would choose to return a [].byte instead of []byte for this same method, but that would not add any security in this case, and would be stealing away a scratch buffer that the application could temporarily use for its own purposes (instead forcing the application to create more garbage if it needed to manipulate the results).

The reduction of library surface area is positive, but unless we could completely eliminate either the bytes or the strings package (not that we could before 2.x anyway), it seems better not to try, and instead decrease footprint by making all applicable strings functions offload onto bytes.

@rog, regarding `var f func(string, string) int = strings.Count`, some, but not all, of this could be addressed by (controversially) modifying assignability rules such that a function type's assignability is transitively based on the assignability of its parameter types (in this case, making all assignability relationships like aliases). For example:

func F(string) string {}
func G([].byte) [].byte {}
var (
f1 func(string) [].byte = F // OK: return string assignable to [].byte
f2 func([].byte) string = F // NO: param [].byte not assignable to string
g1 func(string) [].byte = G // OK: param string assignable to [].byte
g2 func([].byte) string = G // NO: return [].byte not assignable to string
)

A positive point: it would also mean that `func(io.ReadSeeker)` is assignable to `func(io.Reader)`. In any case, this certainly brings readability down, though. A negative point: for maximum ease of use, with this we may start to see APIs like `func([].byte) string`, since it could take a string or []byte, yet be assigned into either a string or [].byte.

The issue @rsc mentioned with converting to `bytes.TrimSpace([].byte) [].byte` could be remedied by allowing a language exception for the caller of such a function: if the caller passes a []T (which is converted into a [].T), then the caller can assign the result back into a []T if the function result references the same memory as the parameter.  This raises a few issues: 1) the compiler would have to verify that result is derived from the input (which may not be trivial), 2) there'd be no syntax to indicate that a function can be used in this "temporarily immutable" way, requiring additional documentation guidelines instead, and 3) type inference would be problematic: should it be inferred as []T because the input happened to be []T, or inferred as [].T because it's documented that way?

@david: if roslices are forced to have cap == len (or no cap), then I agree that append should allow an roslice as the first parameter. Otherwise, it'd be problematic when cap != len; should we panic at runtime, or prevent appends to roslices, or just always force an allocation? In either case, the append question is just a convenience (and shouldn't be a blocker), as `append(append([]T(nil), src...), extra...)` is roughly equivalent to `append(src, extra...)`.

That said, this proposal should be entirely distinct from shrinking the cap of a slice: there are still many applications to exposing many cap == len slices that were all formed from a single allocation. I suggest the following clarifications:
  • Even if not deemed useful, [].T should have a cap, and it corresponds to the same cap a corresponding mutable slice would have (limiting cap is orthogonal to this). We can always take cap away or make it fixed later after using this more easily than we can give it back.
  • Slicing a [].T derives a [].T, and all the normal slice-slicing rules apply.
  • Assigning the string expression s[3:7] to a [].byte is equivalent to [].byte(s[3:7]), not [].byte(s)[3:7], thus extra code would be required to ever get a cap != len slice derived from a string.
# Regarding "types":

@minux, @andrew, @david, @mortdeus: there seems to be a lot of interpretations toward both sides on whether [].T and []T should be distinct types, or whether the immutability should be a property of a type; you can be sure that there'll be a lot of confusion over this issue if and when it ends up in the spec Unless there's an excellent reason to do otherwise, the more liberal interpretation (which allows those in doubt to do what they'd expect) is a favorable choice, the liberal choice being that roslices are not type-distinct from their mutable counterparts. To this end (and if for no other reason, consistency's sake), I feel the following clarifications would be beneficial:
  • [].T is a []T, except that it cannot be used to modify the underlying data.
  • Methods for a slice type can accept a readonly receiver, just as a type can already take value and pointer receivers. This would mean that the syntax for specifying a slice type would need to be independent of its type.
  • Internally these would likely need to be distinct types, though reflect could have an IsReadonly method rather than a separate kind, with CanSet and friends unconditionally returning false for element values.
# Relationship to strings:

While this proposal would make string and [].byte very similar, I do agree that they should be separate types, since this proposal isn't concerned with slice equality and comparability. Other than that, it would be useful for some of the semantics of strings and roslices to be unified:
  • strings have different for-range behavior than slices, though there are already cases where it is desirable to range over the runes in a []byte without needing to either convert into string or pull in encoding/utf8. Perhaps a parallel proposal (probably for the fabled Go 2) for configurable byte and rune iteration of string, []byte, and []rune could be devised.
  • "strings cannot be changed by anyone ever..." Part of this proposal should be compiler support for intelligently handling readonly literals. With roslices declared this way, there's never any opportunity for mutable access, therefore they should be linked into the rodata segment or equivalent. Readonly map literals could get an optimal memory layout.
# Transitive immutability:

I agree with @iant that taking the address of an roslice's element is not unreasonable. The proposal also didn't address slices of composites, for example `[].[2]int` or `[].[]byte`. If [].T were truly a distinct type from []T, then the only slice types it'd truly make readonly would be slices of scalar types (such as []int). Based on a purely structural readonly property, as some have suggested, the following would unfortunately be valid:

var x = [].[2]int{{1,2}}
x[0][0] = 3

On the other hand, if the immutable property of a type extended to all deep accesses, preventing any kind of assignment or use of methods that could modify data, or assigning any descendant pointer or non-readonly reference value of the roslice to another variable, then roslices could be usefully used for access mediation. Still, this approach feels too limited (and too much spec-bloat for what it achieves) -- offering immutability to all composite types in general would accomplish a lot more with a cleaner specification. In such a case, any access to an immutable type yields another immutable type (which is responsible for immutability-at-depth), and no write access is allowed to the contents of an immutable value (though an immutably-typed variable can be given another value).

# Tack-on proposal:

Extend non-copy convertibility to cover read-only slice views of arbitrary compatibily aligned data. For example, anything would be non-copy convertible to a [].byte, since everything aligns to a multiple of one byte. This would make some uses of unsafe obsolete. It would also allow at least read-only views in the following case:

type Int32 int32
var x []int32 = []Int32{1,2,3}

As an exception, int, uint, and uintptr shouldn't be allowed as the underlying recipient element type unless the underlying source element type is also one of those types, or is platform-independently a word multiple. For example:

[].int([]int32(...))              // not allowed
[].int([][2]int32(...))           // allowed
[].struct{x, y int16}([]int(...)) // allowed

Even if it were restricted to the recipients being either [].byte or having the same underlying structure, it'd still be quite useful.

Keith Randall

unread,
May 29, 2013, 7:21:42 PM5/29/13
to Kevin Gillette, golang-dev
On Wed, May 29, 2013 at 3:52 PM, Kevin Gillette <extempor...@gmail.com> wrote:
I agree with @jan and @robin in that the perceived motivation for this is at least partly encroaching on what today's escape analysis, or that in the near future could certainly do to minimize the number of []byte <-> string copies; from what I've gleaned, the missing piece of this puzzle is compiler code that actually uses the results of existing analysis to prevent copies. Moreover, there'd be some places where static analysis would do a better job than the proposal, even with the proposal's own types, since _semantically_ you can't pass a [].byte into a string without a copy. Because of this, I'm only considering the aspects of the proposal that are not concerned with solving the perceived garbage problem.

Also, there's something to be said for hiding data behind functions and channels for readability' sake: Go currently has a simple "if you can see it, you can modify it" expectation that this proposal would violate, since it'd allow data to (at least under some circumstances) be safely used as its own API. The proposal does not use alternative syntax for accessing roslices, which is a good thing, though it does mean that readonly and mutable slices are only distinguishable at the declaration site, decreasing readability. Right now, exported package level variables can be modified by any importers; except in special circumstances (like an exported var of a non-exported type), this proposal wouldn't prevent anyone from replacing the referenced slice outright -- it'd just prevent modification of the original.  Semantically, there wouldn't be much difference if that global var was the only reference, yet many programmers may see this proposal and assume it applies to the slice value in addition to the backing array.

There's also a notable potential for abuse here. bufio.Reader's ReadSlice method doesn't return a copy of the internal buffer, but that's not a problem, since the Reader will blindly overwrite those contents on the next method call; in designing such an API, many would choose to return a [].byte instead of []byte for this same method, but that would not add any security in this case, and would be stealing away a scratch buffer that the application could temporarily use for its own purposes (instead forcing the application to create more garbage if it needed to manipulate the results).

The reduction of library surface area is positive, but unless we could completely eliminate either the bytes or the strings package (not that we could before 2.x anyway), it seems better not to try, and instead decrease footprint by making all applicable strings functions offload onto bytes.

@rog, regarding `var f func(string, string) int = strings.Count`, some, but not all, of this could be addressed by (controversially) modifying assignability rules such that a function type's assignability is transitively based on the assignability of its parameter types (in this case, making all assignability relationships like aliases). For example:

func F(string) string {}
func G([].byte) [].byte {}
var (
f1 func(string) [].byte = F // OK: return string assignable to [].byte
f2 func([].byte) string = F // NO: param [].byte not assignable to string
g1 func(string) [].byte = G // OK: param string assignable to [].byte
g2 func([].byte) string = G // NO: return [].byte not assignable to string
)

A positive point: it would also mean that `func(io.ReadSeeker)` is assignable to `func(io.Reader)`. In any case, this certainly brings readability down, though. A negative point: for maximum ease of use, with this we may start to see APIs like `func([].byte) string`, since it could take a string or []byte, yet be assigned into either a string or [].byte.

The issue @rsc mentioned with converting to `bytes.TrimSpace([].byte) [].byte` could be remedied by allowing a language exception for the caller of such a function: if the caller passes a []T (which is converted into a [].T), then the caller can assign the result back into a []T if the function result references the same memory as the parameter.  This raises a few issues: 1) the compiler would have to verify that result is derived from the input (which may not be trivial), 2) there'd be no syntax to indicate that a function can be used in this "temporarily immutable" way, requiring additional documentation guidelines instead, and 3) type inference would be problematic: should it be inferred as []T because the input happened to be []T, or inferred as [].T because it's documented that way?


This sounds like it might work.  If you have a [].byte you can upconvert it to []byte if you have a pointer to the underlying modifiable []byte.  So you might write:

in bytes:
  bytes.TrimSpace([].byte) [].byte

in your code:
  var buf []byte = ...
  buf = promote(buf, bytes.TrimSpace(buf))
  ...buf is trimmed and still writeable..

It is easy to detect at runtime whether the first argument to promote dominates the storage used by the second argument (and presumably panic if it doesn't).  Some nice syntax might even be buf = buf[bytes.TrimSpace(buf)], where indexing a slice A by a (readonly?) slice B means trimming A to the subset of elements referred to by B, or panic if they don't subset.

The downside, this only works if TrimSpace is documented to return a subset of its input.  If it occasionally returns some other buffer it won't work.

Or maybe if the slices don't subset, promote does a copy instead of panicing?
 

--

Russ Cox

unread,
May 29, 2013, 8:10:20 PM5/29/13
to Kevin Gillette, golang-dev
On Wed, May 29, 2013 at 6:52 PM, Kevin Gillette <extempor...@gmail.com> wrote:
A positive point: it would also mean that `func(io.ReadSeeker)` is assignable to `func(io.Reader)`.

You can only do this if they have compatible memory layouts. Today, they don't. Well, actually those two might but Writer and ReadWriter do not, because Write is method[0] in the first and method[1] in the second (the method list today is sorted by name). You'd want to keep the constant time (and very cheap) method dispatch in any changes.
 
The issue @rsc mentioned with converting to `bytes.TrimSpace([].byte) [].byte` could be remedied by allowing a language exception for the caller of such a function: if the caller passes a []T (which is converted into a [].T), then the caller can assign the result back into a []T if the function result references the same memory as the parameter.  

This is pretty clumsy, as you noted. Really you need some way to express this properly in the type system if you go down this road.

@david: if roslices are forced to have cap == len (or no cap), ...

Just like the syntax, this is not even worth discussing. It's noise compared to the rest of the issues.

Otherwise, it'd be problematic when cap != len; should we panic at runtime, or prevent appends to roslices, or just always force an allocation? In either case, the append question is just a convenience (and shouldn't be a blocker), as `append(append([]T(nil), src...), extra...)` is roughly equivalent to `append(src, extra...)`.

This is arguably noise too, but since we're talking about it: I gave this some thought while implementing  and convinced myself that append of a read-only slice should be disallowed, for two reasons that have nothing to do with len/cap. First, the result of the append today has the type of the first argument. Given var x []readonly byte, it is obviously wrong for append(x, 1) to have type []readonly byte as well. Addressing this requires putting a special case into the append definition. Second, and worse, I am worried about loops like:

    var x []readonly int
    for i := 0; i < n; i++ {
        x = append(x, i)
    }

If you remove the "readonly", this loop generates only a linear amount of garbage, because of append's multiplicative growth. However, with the "readonly" keyword, each append must make a copy, because its argument is a readonly slice, resulting in quadratic amounts of garbage. I would rather the compiler reject the code.

# Transitive immutability:

If you go down this road things get very confused very fast, because of cyclic data and intended references to large mutable structures. I suggest not doing that. Immutability has to be specific to a single piece of data. Also, a plea to use the terms this way: "read-only" means that the current piece of code cannot change it (locally immutable). "Immutable" means nothing can (globally immutable).

# Tack-on proposal:

Extend non-copy convertibility to cover read-only slice views of arbitrary compatibily aligned data. For example, anything would be non-copy convertible to a [].byte, since everything aligns to a multiple of one byte.

This is the way Go used to work, more or less. We decided it was too fragile, because a conversion might work accidentally due to details of the representation that were not intended to be maintained. For example, at the time if you had

type T1 struct {x int}
type T2 struct {x int}

you could convert freely between T1 and T2, or between *T1 and *T2, but that introduces a coupling that the authors of T1 and/or T2 may not have intended. For large-scale programs, the current rule is more robust.

I agree that if we did introduce readonly (not planned, I must emphasize) it would probably imply some changes along these lines, but more likely only for the new annotations.

I am fairly convinced that any serious attempt to do this would have to expose both readonly and immutable, as separate concepts. The wounds in the type system would be deep and require intensive care to heal properly.

Russ

j...@google.com

unread,
Oct 10, 2013, 7:12:28 AM10/10/13
to golan...@googlegroups.com
Re-opening this old wound to explore a direction hinted at in the discussion but never pursued.

This is not a proposal for Go 1.x. It is unlikely even for Go 2. Think of it as a bit of proto-design for a successor to Go. (I'm posting here because I can't find the go-succ-nuts group.)

The intent is to handle the duplication or triplication forced on some functions like TrimSpace, as mentioned in detail in Russ's analysis. We'd naively define TrimSpace as

  func TrimSpace([].byte) [].byte

but this would prevent callers who pass in a []byte from using their own memory when it came back from TrimSpace.

My proposal is to introduce a marker in return types that means "this is the same type as this formal parameter." For the purposes of this message, I'll reserve the word "like" for this. The syntax is Eiffel's. I'll also adopt Eiffel's name for this feature: anchored types.

With anchored return types, we could write TrimSpace as

func TrimSpace(b [].byte)  like b

If you call TrimSpace with a []byte, it returns a []byte; if you call it with a [].byte, it returns a [].byte.

This proposal doesn't help with functions like

  func Join([][]byte, []byte) []byte

We might want to say that the return type is like the element type of the first formal, but we can't in my proposal. Such functions are relatively rare.

Some minor points before getting to the larger questions:

- "like" can only occur as a return type, and must refer to a formal parameter that has a readonly annotation.

- Named return values with "like" types must be treated as readonly.

- A return value that is like f, where f is of type readonly T, can accept both T and readonly T. For example, given the definition of TrimSpace above, a return statement in the function could have an expression of  either [].byte or []byte. That is really not anything new: it is implied by the free conversion from T to readonly T.

- As always, for the purposes of assignment to variables of function type and matching interface methods, function signatures must match exactly. You can only assign TrimSpace to a variable of type "func([].byte) like b".

Now to the main questions:

Is it possible to check a call involving an anchored type? And if so, how? I believe it is, but my type-fu isn't strong enough to prove it. I think anchored types are just a special case of generic types, and type inference for generic types is pretty well understood. A function signature like TrimSpace's would be treated as the type alpha -> alpha, where alpha is a type variable, and type checking proceeds by proving that alpha can be assigned consistently.

Will this involve compiling multiple versions of functions? No, not if T and readonly T have the same representation.

Does this make Go's type system just a little too complicated for what is supposed to be a simple, easy-to-understand language? Answer: yes. Yes it does. That is why this is not a proposal for Go. (See what I did there?)

Kevin Gillette

unread,
Oct 14, 2013, 12:49:58 AM10/14/13
to golan...@googlegroups.com, j...@google.com
I believe much of the impetus to explore this topic in the first place was to solve what was perceived to be a practical need; I don't think there's any fundamental reason to provide read-only slices on the language level in isolation from such practical matters.

There's also a separate concern: as previously proposed, the read-only property of read-only slices only really serves to bridge the gap between strings and byte slices, such that there is an explicit, mechanical process by which copying can be avoided. The read-only property is not intended to emulate the "const" keyword from C-based languages. Because its orthogonality would be rather limited, it could be described as a "shallow" language addition.

Furthermore, the practical aspects of read-only slices could be achieved without language changes at all, via the improvement of the compilers' escape analysis facilities. With such changes, existing code, unmodified, would be able to convert between string and byte-slice types without copying, which is considerably better than requiring code changes to take advantage of a language-bound optimization. When there has been the option of making explicit in the language now what the compiler or runtime could deterministicaly do later, almost (if not actually) universally it has been chosen that Go take the optimistic path and leave it for the compiler to handle later.

draw...@gmail.com

unread,
Nov 9, 2013, 4:50:33 PM11/9/13
to golan...@googlegroups.com
Though this is vey old thread, I like to add my opinions to push this feature to be added to Go ASAP.

I really badly want this feature. Currently, there's no simple way to return some immutable data from a function, so I need to (1) write full immutable wrappers for all types (2) return only new copies. This can eliminate needs for this unnecessary overhead.

I also strongly support read-only map for same reason.

For notations, I think there must be a clearly recognizable, unique and dedicated syntax and type. For better code readability, better tooling support. Identifying mutability of in/out parameters is critical. Function signature must be served as a quick documentation.

For specific syntax, I don't agree dot. Because it's ambiguous. It's somewhat like const in C/C++. I still don't understand why type assertion is using dot syntax. Also don't agree to : colons for same reason. If I have to choose, maybe I would choose # sharp sign, ! explanation mark or plain english keyword *readonly* (as like rsc's proposal).

    var arr1 = #[]int          // Readonly array.
    var map1 = #[int]int    // Readonly map

    var arr1 = ![]int          // Readonly array.
    var map1 = ![int]int    // Readonly map

    var arr1 = readonly []int          
    var map1 = readonly [int]int    

minux

unread,
Nov 9, 2013, 5:38:22 PM11/9/13
to draw...@gmail.com, golang-dev
On Sat, Nov 9, 2013 at 4:50 PM, <draw...@gmail.com> wrote:
Though this is vey old thread, I like to add my opinions to push this feature to be added to Go ASAP.
Did you read Russ' very detailed analysis of this proposal (he even did a POC implementation)?

There are multiple drawbacks.

Hoon H.

unread,
Nov 9, 2013, 6:16:59 PM11/9/13
to minux, Hoon H., golang-dev
I didn't read it fully, so now I read it completely.

And I don't see any real drawback from the text except legacy compatibility. And IMO, all the drawbacks are only about string type - which is very exceptional type for const-correctness.

Personally I think real const correctness will simplify overall program complexity when program goes bigger.
However, if Go team already made a decision, I don't have any more interest on this.

Brad Fitzpatrick

unread,
Nov 11, 2013, 5:14:42 PM11/11/13
to Hoon H., minux, golang-dev
Yes, if/when generics happen, read-only types should be considered at the same time.  Because like you said, most of the negatively from this proposal is around its effects on the standard library.  Generics would warrant a re-think of the standard library as well.



Marçal Juan Llaó

unread,
Jan 6, 2015, 7:51:35 AM1/6/15
to golan...@googlegroups.com, draw...@gmail.com, minu...@gmail.com
Sorry to answer in this old thread but I wasn't sure to open a new dedicated one on a vaguely idea I had today, here I go:

Scoped read only variables.

This is a general flag for a variable, to help developers doing less bugs and for performance optimizations as well.

For example:

package main

import (
"fmt"
)

type Foo struct {
Name string
}

func NewFoo() *Foo {
return &Foo{Name: "hi"} readonly
}

func (f *Foo) SetName(newName string) {
// More complicated stuff...
f.Name = newName
}

func main() {
a := NewFoo()

fmt.Println(a.Name) // Valid
a.SetName("pepito") // Valid
a.Name = "grillo" // Invalid
}

This is intended to unnecessary create a method for "a.ReadName()" but don't let programmers modify the value directly. The idea is to have a dynamic scoped constant variable/struct.

This also can be used in other scenarios, but as it was a quick idea I had this morning, I prefer go's experts tell what you think... :D

Thanks for your time! Thanks!

El dilluns 11 de novembre de 2013 23:14:42 UTC+1, Brad Fitzpatrick va escriure:

Jonathan Amsterdam

unread,
Jan 6, 2015, 7:59:51 AM1/6/15
to Marçal Juan Llaó, golan...@googlegroups.com, draw...@gmail.com, minu...@gmail.com
This looks like a compile-time feature but it actually requires a runtime check. Making the variable unexported and using an accessor function will actually be much faster, since there is no runtime check. In fact, since the function will be inlined, it will be as fast as accessing the field directly.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/Y7j4B2r_eDw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Robert Griesemer

unread,
Jan 6, 2015, 1:14:15 PM1/6/15
to Marçal Juan Llaó, golang-dev, draw...@gmail.com, minux ma
Please use the golang-nuts mailing list for these discussions; golang-dev should be used for active and planned development.

Your suggestion would be a language change (possibly backward compatible, but still a language change), and we're done with significant changes like this for the time being. Thanks.

- gri

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Marçal Juan Llaó

unread,
Jan 6, 2015, 5:15:46 PM1/6/15
to golan...@googlegroups.com, mar...@gmail.com, draw...@gmail.com, minu...@gmail.com
Ok, thanks. Keep doing a great job!

El dimarts 6 de gener de 2015 19:14:15 UTC+1, gri va escriure:
Reply all
Reply to author
Forward
0 new messages