Proposal for a new statement

[David DENG]

An assigment statement that remove the reference in the source.

For example, the operator is <=(and :<=), then the following statement:

x <= y

equals

x, y = y, <nil>

where <nil> is the zero value for y.

This is like the move action in really life.(common assignment is a copy action)

Using this statement can avoid many memory leak. e.g. removing the last element in a slice, and return the removed one can be written as:

x :<= s[len(s) - 1]

s = s[:len(s) - 1]

return x

s[len(s) -1] is automatically assigned with a zero value.

Let's discuss this feature.

David

[Miek Gieben]

[ Quoting <david...@gmail.com> in "[go-nuts] Proposal for a new statem..." ]

> Let's discuss this feature.

Let's not...

Regards,

--
Miek Gieben http://miek.nl

[Jan Mercl]

On Wed, Jan 9, 2013 at 3:42 PM, David DENG <david...@gmail.com> wrote:
> Using this statement can avoid many memory leak. e.g. removing the last
> element in a slice, and return the removed one can be written as:
>
> x :<= s[len(s) - 1]
> s = s[:len(s) - 1]
> return x
>
> s[len(s) -1] is automatically assigned with a zero value.

Note that generally when using the slicing operator, it may be
completely valid and reasonable code when the
"outside-of-the-new-slice" values are kept. Where one really wants to
"unpin" the element (or to what it points to), the explicit way is
simple enough to not warrant a new-extra-special-magic operator, IMHO,

> Let's discuss this feature.

-1 from me.

-j

[steve wang]

I think there is no memory leak happens there because after the slice is shortened the element that once was located at the tail is no longer accessable and will be recycled automatically by GC whether or not you clear it before.

[David DENG]

No. I wrote code to test it. http://play.golang.org/p/zoe_phL_Rn (comment LINE 24 to see the difference)

The GC cannot do that. Since I may resize the slice back to its capacity.

David

[steve wang]

It actually acts like what you said. 

And I found this in the specification which was missed by me before:

for slices, the upper bound is the capacity rather than the length.

Thanks your for pointing out my mistake.

[Joey Geralnik]

No comment on the creation of a new operator, but <= is already taken.

--Joey

--
 
 

[bryanturley]

On Wednesday, January 9, 2013 10:48:26 AM UTC-6, steve wang wrote:

It actually acts like what you said. 

And I found this in the specification which was missed by me before:

for slices, the upper bound is the capacity rather than the length.

Thanks your for pointing out my mistake.

It still isn't a memory leak, since it will be freed when the slice is freed.
Just in some limbo.

[steve wang]

It acts like memory leak in terms of the intention of the programmer who tries to treat a slice like a stack.

[Kevin Gillette]

Has this been a frequent need for you? Is there any public code written in Go that you can point to where this would have helped considerably?

Aside from questions of need or the value gained proportional to cost, this is honestly one the cleanest/simplest syntax proposals I've seen in a long time.

I think :<= is much less valuable than <=, since, except in very few cases (when do you need to zero a single element in a slice, anyway?), declaring a variable to remove the reference from another seems like a name management problem. If we went further to suggest that <= could "undeclare" the right-side identifier (allowing the number of in-scope names to gradually decrease, even though it'd probably not have any other compiler effect), then :<= would be unmanageable in most scoping scenarios (such as when deleting an outer-scope identifier from an inner conditional scope).

In terms of memory leaks, zeroing a variable can only help if the zero value happens to be nil (reference or pointer types), or if a very large portion of a slice is unused (such as pinning 10mb of cap to store 10kb of data). In almost all cases dealing with more than one element in a slice, <= will not be an efficient solution; the better approaches will generally be to make a copy of the slice containing just the used length and removing references to the old slice, or as has been discussed elsewhere, providing a mechanism for the cap to be manually shrunk; this has GC implications in that if no slices refer to an orphaned cap, it could be reclaimed if large enough, but in all cases not considered to contain live pointers/references.

Anyway, I like the idea of using <= somehow in a statement. Joey: this use of <= could be mutually exclusive with the use of the same token in expressions, though we'd have to change the semantics of a statement that nobody uses, `x <= y ;` so that instead of generating the error "x <= y not used", it'd actually do something.  I think there may be other, higher demand cases, for an additional assignment form, such as in error handling, but I'll not hijack this post for such purposes.

[Kyle Lemons]

While I understand that some are worried about the garbage collection implications of the failure to zero a slice element before shortening the slice, in practice I don't think this proposal would improve the situation.  If you already know enough to use your "move" statement (I submit to you that <= is a non-starter, having meaning already) in this context, then you also know enough to assign zero in the same scenario.  It would not help in the cases where a programmer accidentally makes this mistake in a (relatively rare, I think) context in which it makes a difference. 

[Kevin Gillette]

"non-starter" ?

[bryanturley]

On Wednesday, January 9, 2013 2:38:22 PM UTC-6, Kevin Gillette wrote:

"non-starter" ?

I think he means overloading <= is a bad idea, which I would agree with.
There is all of unicode to draw symbols from why not use them?
Though I don't see any real benefit in this particular operations case.

[Kevin Gillette]

As soon as you define a token that doesn't consist solely of printable ascii characters, you'll fracture the Go community. I strongly assert that even the most bizarre sequence of ascii characters (or making no changes at all) is better than exploring the range of unicode chars. After all, why was >= possibly chosen instead of ≥, or why != instead of ≠ ?

[Jan Mercl]

On Wed, Jan 9, 2013 at 10:25 PM, Kevin Gillette
<extempor...@gmail.com> wrote:
> As soon as you define a token that doesn't consist solely of printable ascii
> characters, you'll fracture the Go community.

Go's lexical grammar already accepts several different token types
containing non ASCII characters.

-j

[Kevin Gillette]

If you mean _allowing_ identifiers to contain 'letter' codepoints, that's an entirely different matter, since that benefits developers by giving the ability to express much of their code in their own native language, or to use greek letters for variables in algorithms, and so on. If you mean allowing operators and other tokens to take alternative ascii and non-ascii forms (if Go has this, I was unaware of it) is somewhat moot, since the ascii option will almost invariably be used, and unless you're really trying hard to win over the academicians and theorists, it's probably not worth even a line of code to support "proper" mathematical symbols as operators.

However, these capabilities are all best used with discretion (application code or libraries meant for internal or non-international use, sure, otherwise? no) -- times when library developers have used non-ascii in exported identifiers, they have often come under criticism, or have independently realized the general difficulty involved in using an API with non-ascii runes, and have voluntarily renamed their exported identifiers (though I've seen parameter names retain non-ascii symbols -- since those do not need to be keyed in, those have considerably less opportunity to be detrimental. Some of the difficulties are: there are many symbols which look identical or nearly identical, but which have different code points due to different semantics, and a great many developers would have to resort to copy-and-paste for rune input, since for many, ascii accounts for well over 99.9% of the code they need to type, and thus lack familiarity with efficient unicode input.

[Patrick Mylund Nielsen]

That may be, but making them part of the language builtin operators would be insanely annoying. People have different keyboards, can't use unicode, etc. Imagine if Foo.Bar was Foo·Bar like in the runtime.

I can't think of a language, even a math-heavy one described in >50% greek that actually uses non-ascii characters in its main syntax.

-j

--

[bryanturley]

On Wednesday, January 9, 2013 3:57:26 PM UTC-6, Patrick Mylund Nielsen wrote:

That may be, but making them part of the language builtin operators would be insanely annoying.

I agree.  But what is more annoying overloaded operators or the occasional oddball non-keyboard glyph?

cout << "Hello" << x << "\n"; // +1 on the rage meter everytime i see it

cout ←"Hello" ←x ← "\n"; // hard to type but I don't need blood pressure medicine now

The worst part is trying to explain to someone who has coded c++ for 10ish years what bit shifting is...

Also I disliked that whole paradigm and used printf() when I was compelled to write c++ code so don't think I am advocating that ;)

I know we are all used to typing != but ≠ is actually what we mean.
There may be a more appropriate way to say == as well.
We are slaves to our keyboards, and I am fine with that.

[Kevin Gillette]

I much, much prefer "overloading", and consider that opinion to be in service of the public good. That said, yes, many cases of overloading have been done in poor taste, such as just about anything that ended up in C++ (whether with << or ←, I would find the io mechanism in C++ pretty dreadful anyway). That acknowledgement, however, does not imply that all instances of overloading are necessarily distasteful.

[Patrick Mylund Nielsen]

Reading, yes, definitely, unicode/the real symbols make more sense. But even if you write 10x less than you read this is frustrating IMO. Personally, I'd have to find a similar symbol somewhere (and wouldn't be able to search for one), copy, and paste it every time. I don't have the mental acuity to remember a large collection of alt+code combos.

--
 
 

[Dan Kortschak]

Not in support of the proposal, but what about '<~' for the syntax?

[Kyle Lemons]

Yes, "non-starter"; by that I mean that I believe the proposal would never be accepted unless that were changed.

There are lots of things that aren't taken:

a[0] -> x

move(x, a[0])

for instance.

Not that I'm really in favor of the proposal.

--
 
 

[Dave Cheney]

Here is my counter proposal:

Use the language you have today, learn its strengths and its foibles.
Promote your love for it widely from the hilltops, hold meetups, give
presentations, and most importantly; write some code.

Futzing about with new operators is pointless as they will obviously
not be part of any Go 1.x series, and without strong support and
adoption there may not be a Go 2.x series.

> --
>
>

[David DENG]

Yeah, limbo or something, in a GC-based program, no memory-leak at all.

What I mean is if the slice is held for a long time, e.g. a global singleton, some elements are not gc-ed for a very long time.

David

On Thursday, January 10, 2013 1:34:12 AM UTC+8, bryanturley wrote:

[David DENG]

<= can be easily distiguished from the comparator, since assignment is not an expression in go. Or << which is not used in go yet. This is quite discussible.

David

[Kevin Gillette]

<< isn't used in Go yet? http://play.golang.org/p/mmJ_rI-o9O

[David DENG]

I think you may write code like:

t <= s[10:len(s)]

which zeros all elements from 10 in the slice. And some times clearing one element also make sense, if that element is a very big one. e.g. and interface, 2d-slice, etc.

Or a build-in function like copy, named 'move' or something.

move(x, y)

move(t, s[10:len(s))

btw: I was missing a fill function before.

David

[bryanturley]

On Wednesday, January 9, 2013 6:43:02 PM UTC-6, Kevin Gillette wrote:

<< isn't used in Go yet? http://play.golang.org/p/mmJ_rI-o9O

This conversation just got really confused.

Got to +1 what Dave Cheney said above.

[Scott Pakin]

On Wednesday, January 9, 2013 2:57:26 PM UTC-7, Patrick Mylund Nielsen wrote:

I can't think of a language, even a math-heavy one described in >50% greek that actually uses non-ascii characters in its main syntax.

APL and APL2?

[Patrick Mylund Nielsen]

Yes, Kamil Kisiel also pointed out APL. It left my mind. Nevertheless, it's a good example: it requires a separate keyboard/layout, and often annoys/confuses newcomers. Neither of those are really things Go should force everyone to do.

--
 
 

[Patrick Mylund Nielsen]

A separate input for Go (by default) is not feasible.

On Wed, Jan 9, 2013 at 10:10 PM, Alexander Usov <a.s....@gmail.com> wrote:

Wolfram Mathematica.

End-users normally interact with via notebook interface, and it handles math input really well.

--
 
 

[si guy]

Is it safe/possible to use unsafe to shrink the cap of a slice after deleting/releasing the last element(s)? Will the gc clean up the mess?

[Jan Mercl]

This thread is one of those, where the SSMLP ("shrinking slice memory
leak problem") is discussed and/or mentioned or even used as an
argument for a language change proposal. Let me ignore that I don't
agree with using the term 'leak' in this context. I would like to
point out that SSMLP is _only a special case_ of a more general
situation, from with every GC language may suffer: Entities reachable
&& never used anymore (past some point), but in ways which the
compiler cannot easily (or at all) detect. This applies to any entity
- a variable; an array/slice element; a struct field; anything having
a pointer referring to it; ... (what I maybe forgot to include here).

However, the special case of slices is the more complicated one. No
slice knows if there are or are not some other slice(s) with the same
backing array. That's why, in the first approximation, the GC _cannot_
assume elements of the backing array above slice's length are
unreachable. The GC in this approach can collect, if applicable, the
elements of the backing array iff the backing array per se is getting
collected.

-j

[Kevin Gillette]

Certainly slides would not know about other slices, but since the GC is mark and sweep, presumably it would detect all slice headers pointing to a backing array. The question becomes: 1) does the current GC only find/inspect the pointer in the slice and compare it to some allocator metadata about the backing array, 2) or does it/could it easily inspect the slice cap to determine the part of the backing array used by all slices (which would be a max function across discovered caps)?

I expect #2 is not done because caps cannot currently be shrunk without unsafe or other out-of-spec means, yet a shrinking cap can be simulated with strings, and both slices and heads can shrink the used head of the backing array. Certainly most cases of head/tail shrinking would not produce large enough, aligned regions that could be reused or otherwise reclaimed, unless you're dealing with big data, yet the GC could defer scanning backing arrays for pointers until all non-array regions have been marked, ignoring any pointers in unused portions of each array.

Though hardly groundbreaking, I figure compile time information for a precise GC could put an allocated array into 3 categories which could further increase the efficiency of the collection phase: non-ptr, ptr, and potentially self-referential ptr.

[Kyle Lemons]

On Thu, Jan 10, 2013 at 10:27 AM, Kevin Gillette <extempor...@gmail.com> wrote:

Certainly slides would not know about other slices, but since the GC is mark and sweep, presumably it would detect all slice headers pointing to a backing array.

You're allowed to re-slice a slice into its capacity.

x := make([]int, 0, 10)

x = x[:cap(x)]

thus, it could suddenly become reachable again.

 

The question becomes: 1) does the current GC only find/inspect the pointer in the slice and compare it to some allocator metadata about the backing array, 2) or does it/could it easily inspect the slice cap to determine the part of the backing array used by all slices (which would be a max function across discovered caps)?

I expect #2 is not done because caps cannot currently be shrunk without unsafe or other out-of-spec means, yet a shrinking cap can be simulated with strings, and both slices and heads can shrink the used head of the backing array. Certainly most cases of head/tail shrinking would not produce large enough, aligned regions that could be reused or otherwise reclaimed, unless you're dealing with big data, yet the GC could defer scanning backing arrays for pointers until all non-array regions have been marked, ignoring any pointers in unused portions of each array.

Though hardly groundbreaking, I figure compile time information for a precise GC could put an allocated array into 3 categories which could further increase the efficiency of the collection phase: non-ptr, ptr, and potentially self-referential ptr.

--

[Kevin Gillette]

Of course -- I'm referring to shrinking the capacity of a slice (thus you could not slice past the current cap into what used to be the cap).

[Dan Kortschak]

Surely this involves a partial free of an object. How do you do this?

[Kevin Gillette]

If you consider the backing array a single, indivisible object, then I suppose so. I would have expected the book keeping for the original allocation to be modifiable.

[Kyle Lemons]

On Thu, Jan 10, 2013 at 1:56 PM, Kevin Gillette <extempor...@gmail.com> wrote:

If you consider the backing array a single, indivisible object, then I suppose so. I would have expected the book keeping for the original allocation to be modifiable.

I would consider it incredibly surprising behavior if the capacity of my slice changed.  How can you tell that it won't be used later?  This is very common:

x := make([]T, 0, len(someMap))

for k := range someMap {

  x = append(x, someMap)
}

 

On Thursday, January 10, 2013 1:13:23 PM UTC-7, kortschak wrote:

Surely this involves a partial free of an object. How do you do this?

On 11/01/2013, at 6:31 AM, "Kevin Gillette" <extempor...@gmail.com> wrote:

> Of course -- I'm referring to shrinking the capacity of a slice (thus you could not slice past the current cap into what used to be the cap).

--
 
 

[Dan Kortschak]

A quick read of malloc.goc shows that objects are allocated as single objects through mallocgc.

[Kevin Gillette]

Even though one slice's capacity shrank, no other slices pointing to the same backing array would have their capacities shrank. The cap of a slice would not "shrink out from under you" so to speak, since presumably you'd be the one doing the shrinking, and only for that specific slice.

[Marcelo E. Magallon]

On Wed, Jan 09, 2013 at 04:23:50PM -0600, Patrick Mylund Nielsen wrote:

> Reading, yes, definitely, unicode/the real symbols make more
> sense. But even if you write 10x less than you read this is
> frustrating IMO. Personally, I'd have to find a similar symbol
> somewhere (and wouldn't be able to search for one), copy, and
> paste it every time. I don't have the mental acuity to remember
> a large collection of alt+code combos.

This is highly off-topic, but a small tip...

If you are using X.org (any Linux distribution, any BSD, etc),
you can define a key as a "compose" key. After that, you can
press [compose] <- to get ← (→ ≤ ≥ ≠ …). Most of the default
combinations are pretty intuitive.

If that doesn't work for you, in Vim you can define digraphs
(not to be confused with the C preprocessor's digraphs), so for
example you can press C-k <- and you also get ←.

I'm pretty sure there are similar solutions for other
environments and operating systems. I wouldn't know them,
though.

Marcelo

[minux]

On Fri, Jan 11, 2013 at 4:13 AM, Dan Kortschak <dan.ko...@adelaide.edu.au> wrote:

Surely this involves a partial free of an object. How do you do this?

I think the real solution for this problem is adding the ability to set (shrink only) capacity

of slice. This is issue 1642. And there are other use cases for this feature (one notable

example is an arena allocator for byte slices which doesn't want the client to extend

the capacity to write into backing arrays of slices immediately following the current slice).

Once we support that, maybe we can make the partial free of an slice's backing array

possible in this case (general partial free of objects is of course possible, but very difficult

make correct).

[roger peppe]

On 11 January 2013 05:35, minux <minu...@gmail.com> wrote:
>
> On Fri, Jan 11, 2013 at 4:13 AM, Dan Kortschak
> <dan.ko...@adelaide.edu.au> wrote:
>>
>> Surely this involves a partial free of an object. How do you do this?
>
> I think the real solution for this problem is adding the ability to set
> (shrink only) capacity
> of slice. This is issue 1642. And there are other use cases for this feature
> (one notable
> example is an arena allocator for byte slices which doesn't want the client
> to extend
> the capacity to write into backing arrays of slices immediately following
> the current slice).

i raised that issue, and it would definitely be useful, but there are down-sides
too - currently you can always tell if two slices alias one another, but
if you allow capacity checking, this becomes harder. i suppose one could
add a new built-in function to help that particular case, though.

[Kevin Gillette]

Strictly speaking, you can still check if two slices *overlap* even in the presence of shrinkable slices -- while it would no longer be as simple as checking the last element in the cap of each slice for equality, it is still rather simple and still involves only a constant number of steps. This may be a good thing to put in reflect (it would be quite fast since, while a different element you're should guarantee non-overlap, no other reflection should be needed aside from the slice header and the element size).

If there's any concern about no longer being able to tell if two slices share the same backing array, it's semantically moot in almost every way, in that two non-overlapping slices have the same relative properties whether they share an array or not. It could matter if you need very tight memory control, but just as it is now, it would be trivial to do your own book keeping to make such determinations.

[roger peppe]

On 11 January 2013 10:40, Kevin Gillette <extempor...@gmail.com> wrote:
> Strictly speaking, you can still check if two slices *overlap* even in the presence of shrinkable slices -- while it would no longer be as simple as checking the last element in the cap of each slice for equality, it is still rather simple and still involves only a constant number of steps. This may be a good thing to put in reflect (it would be quite fast since, while a different element you're should guarantee non-overlap, no other reflection should be needed aside from the slice header and the element size).

Yes, you're right, it could be fast enough, assuming we could make sure the
compiler didn't allocate for the interface conversions.

Here's a naive version (that does allocate) FWIW.
http://play.golang.org/p/t-bD7bpvB4

It's nice to see that putting a slice in an interface{} doesn't always allocate.
I hadn't realised the compiler was that clever.

> If there's any concern about no longer being able to tell if two slices share the same backing array, it's semantically moot in almost every way, in that two non-overlapping slices have the same relative properties whether they share an array or not.

Agreed.