half-finished proposal for notnull pointers and exceptions

[NoiseEHC]

Hi Russ (and members of this mailing list of course)!


Here is the missing ring.go file I have promised to you. :)


In fact this is a rewrite of a formal proposal which I was starting to
write in 2009.11.16... It was originally named as “proposal for
exception handling and notnull pointers” but in the meantime it seemed
better finishing the notnullness of pointers so here it is.


It can seem a little bit long but do not forget that the point of it is
that it permits defining interfaces (contracts) which expect a notnull
parameter or guarantees a notnull return value.


Ian talked about automatically inferring the notnullness of pointers but
it will not work because of these things:

1. In the case of dynamically loaded modules it will not be able to
infer interfaces (both the interface construct in Go or imported
function parameters).

2. It will fail in exactly the same cases this proposal defines explicitly.


Now I have to admit that it is not such a big thing as sliced bread and
a lot of “proposals” (or just call them as talks) were about certain
parts of this one so it is not that unique. The most important reason it
took so long is that I do not had too much free time and I can write in
English only painfully slowly. So if it seems a little bit sparse is
because it is a little bit boring for me now...



Note:

1. This language feature will not raise the IQ of idiots not even by 2
points. So I am not interested in fictional stories about what could
idiots do in the workplace because of this language feature.

2. Every programmer can just use nullable pointers and code exactly the
way they are doing now (except they use ? characters instead of *). The
only difference will be if a function expects a notnull pointer and it
cannot be derived from the caller that this constraint is met (and so
the programmer must put some “if p != nil” there which was probably
required anyway). More about this at the end of this proposal.

3. This notnull feature will not help too much in the cases when data
structures are always initialized to notnull pointers (you can see it in
ring.go where it makes things more ugly). They help mostly when pointers
can contain nil (see it in list.go).

4. This proposal assumes that all notnull pointers are aligned,
otherwise it would violate the memory barrier thing.

5. No, I will not create a formal proposal...



Rules for notnull pointers:


p *T // notnull pointer to T


q ?T // nullable pointer to T


T? is a shadow type of T, if T has notnull pointers then all those are
converted to nullable pointers in T?, if T has embedded struct E then
its type is converted to E?, otherwise T? is equivalent with T


“p = q” only allowed if it can be inferred from an “if q != nil” or from
a “q = qq” where qq *T that q is not null. In this case the compiler
must use the cached q value to avoid inserting reads to the possibly
volatile location of q.


new(T) has a type of *T?


&T(...) has a type of *T if and only if all the notnull pointers are
initialized by notnull pointers, otherwise it has a type of *T?


If the compiler notices an “if p != nil” then it should only emit a
warning (and treat the p = nil case as unreachable code) because we want
to help the programmer when he converts his code to notnull pointers and
do not want to drive him crazy.


In case of r *T? we have the following rules:


“p = r” only allowed if it can be inferred from an if “r.a != nil && r.b
!= nil...” or from a “r = rr” where rr *T or from “r.a = a; r.b = b...”
where a *A, b *B... Returning r as *T or passing r to a function as a *T
parameter means an implicit conversion.


Before a *T? into *T conversion happens the compiler must insert a
memory barrier with release semantics (so that no writes can pass below
the barrier). The compiler must delay this memory barrier to the
absolute last point before the conversion (or the last write to the
struct, but it would be hard if r is passed into a function which
accepts *T?).

r := new(T);

r.e = 3;

r.a = &A(...);

r.d = 2;

r.b = &B(...);

// if we would insert the memory barrier here then the programmer would
have noticed a strange compiler behavior that a shared r behaves as a
properly initialized struct except its c member...

r.c = 1;

// memory barrier here

return r;


Note that the programmer can always use “if r.a == nil || r.b == nil {
halt(); }” when the compiler cannot infer from simple data flow analysis
the notnullness of pointers (the compiler can always assume nullable
pointers if in doubt). In other words, inferring the notnullness of
pointers does not have to be foolproof, the programmer should follow Go
idioms for which this mechanism hopefully works.


Also note that in case of very complex circular references the
programmer can use nullable references everywhere and in that case the
code will look like just as it looks now without notnull pointers.


Finally note that this proposal assumes that the new() operator never
returns null. In my original proposal it would have meant exceptions in
out of memory conditions. Also the explicit *T? into *T conversion via
cast could throw an exception because without exceptions the programmer
now has to use halt(). This explicit cast could be a no-op (except the
memory barrier) if the compiler already inferred the notnullness.


type S struct { a ?A }

s1:= new(S); // s1 *S?

s1.a = new(A); // s1.a is *A now

s2 := s1; // s2 *S

type R struct { s S }

r1 := new(R); // r1 *R?

r1.s1.a = new(A); // r1.s1.a is *A now

r2 := r1; // r2 *R


Arrays:


With arrays we have the same problems that with structures but it
requires even more compiler smartness to figure out whether all the
elements are initialized (no matter whether they are notnull pointers or
structures with notnull pointers). Because it would require implementing
first-order logic in the go compiler's typer I would rather restrict
this to recognize some loop idioms (“for i=0; i<len(arr); i++ {}” and
“for i := range arr {}”) and only prove that one element is initialized
in every loop iteration.


For example the following does not have to be recognized:

var arr [10]*int;

for i := 0; i < len(arr); i = i+2 { [i] = new(int); }

for i := 1; i < len(arr); i = i+2 { [i] = new(int); }


The programmer who wants to use notnull pointers with fancy
initialization semantics then has to write this as a last resort:

for _, item := range arr { if item == nil { halt(); } }



Examples:


list.go demonstrates the usefulness of notnull pointers. There is a
questionable design decision in the original source that in
PushFront/PushBack it initializes the passed in list pointer (l) if it
is not initialized. I cannot think of ANY possible use of this behavior
because the id field is never reset after initialization so it can
happen only after new(Element) and is not our goal to eliminate calling
library functions without initialization? Nevertheless I have left the
code as it was since I simply did not understood that part.


ring.go demonstrates the ugliness of notnull pointers. The main problem
is that the notnullness of “next” and “prev” pointers follows from the
“ringness” of the Ring structure. In the original source, after the
New() function initializes the ring (hopefully without semantic errors),
the code assumes the “ringness” or the ring and can call ring functions
without checking whether the ring is empty (which means == nil). This
problem can be sidestepped in the case of the New function as the
modified source shows but can be problematic in the case the caller code
only checks if n == 0. Example:

n := calculate some size for n...

r := New(n);

curr := r;

for i := 0; i < n; i++ {

curr.Value = calculate next element...

curr = curr.Next();

}

Here the original code would execute the loop 0 time so it would not
dereference a nil. On the other hand the notnull pointer code will not
compile without FunnyNew(n) because it cannot derive the implicit
invariant that (Len(r) > 0) => (r != nil).

If it is absolutely necessary that only the program logic should infer
the notnullness then a similar accessor function would solve this problem:

func (this *Structure) GetNotnullA() (*AType) {

if this.A == nil { halt(); }

return A;

}

An example for such code is if a special allocator retains some number
of allocated structures and use the explicitly freed structures to
satisfy further allocation attempts. Now this code would make GC less
frequent but would nullify the advantages of garbage collection (you
know, if it would turn out that the freed structure was referenced in
another thread, etc...) and of course it would make creating such
accessor functions mandatory.


Note that I do not propose that Go should implement some pre and post
conditions and invariants like Spec# because I think that Go is simply
not THAT language. For the relevant video watch this if you have some
free time: ...and after 30 minutes of searching I have concluded that
the Lang.NET symposium's site is down and I cannot link the Spec# video
anymore where one of the authors fights for ~20 minutes with the
compiler which simply rejects his program because of missing parameter
checks and such things (I remind you that this fight was most likely
planned) and when he finally succeeds turns to the audience and asks the
$1000000 question: “Now WHO wants this?” :)


If you do not implement a Spec# category reasoning engine (and I propose
not to do) which would make the language very complex then it will not
be able to infer so complex invariants and it will be painful in certain
cases (because it will be dumb). What is my opinion however is that even
if it makes coding sometimes a little bit more difficult it is
absolutely worthy!!! Unfortunately we will not know for sure about this
tradeoff until you implement this feature... Probably the best way would
be to delay the implementation until you create a Go compiler written in
Go, where it will be much easier to code this code flow analysis (the
bootstrap compiler would just ignore notnullness or use asserts).



Exception handling:


And finally here comes the bane of the notnull pointers stuff, which is
called error handling. In the original proposal this would have been my
proposal for exception handling which is unfortunately seems necessary
for notnull pointers.


The problem is that if a function returns a notnull pointer and an error
value then in case of an error the pointer will be null. It could be
“fixed” with such code which seems ugly (the “|| p == nil” part):

func Function() (p ?T, ok bool) { … }

p, ok := Function();

if !ok || p == nil { return false; }

a := p.a;


A bigger problem with the code above is this case:

func Function() (p ?T, status int) { … }

p, status := Function();

if status < 0 || p == nil { return status; }

a := p.a;

Here it is a possible error if Function() returns nil and a positive
status in which case the caller will think that the operation succeeded
when it did not.


So the way it could be solved starts with a snippet from my original
proposal: “What I noticed in the do-we-need-exceptions debates that both
sides assume some inherent it-is-really-an-exceptional-condition filter
and this makes the debate totally pointless since those conditions
depend on the problem space the developer is most likely wants to use
the Go language for. In the example given by somebody (and while I
parked the proposal on my disc I have lost track of this message...)
opening a file can fail and in that example (low level library) it is
totally normal operation so inlined error handling code is clearly looks
better (that is why I am not sure abut exceptions anymore). However as
another example failing to open the config file for an application
server is a total failure (like not knowing the access rights or not
knowing the database connection string) in which condition terminating
the program is clearly the best choice. Because in the latter problem
space usually the only error handling that can be done is to show the
error dialog to the user or log it and try again later. My point is that
the definition of the exceptional condition is totally depends on the
caller. Not recognizing it just leads to duplicating every API like Open
and TryOpen, the first trowing an exception while the latter simply
giving back some error code, we could just as easily call it
OpenForSystemProgrammers... So my proposition is to handle exceptional
cases in the caller because exception support (like division by zero)
seems to be unavoidable to me but correct me if I am wrong.”


The “exception” mechanism is simple: every function must define how it
returns errors and the calling code can choose from 3 options:

1. expect and handle the error:

result, err := Func();

if err < 0 { handle error... return err; }

2. expect but explicitly ignore the error:

result, _ := Func();

3. do not expect the error and throw an exception if it fails

result := Func();

// here the compiler should translate it to “result, __err := Func(); if
__err < 0 { throw __err; }” automatically


Now in the 3rd case the compiler would know that it has to insert an “if
__err < 0 {...}” because every function should be annotated with
something like this:

func Func() (result T, err int) fail err < 0 { … }

It would not imply writing more characters in the source because already
all the functions are annotated with comments like “if the function
fails it returns a negative value” (or at least I hope so...) and so it
would only mean that the compiler can enforce such post conditions.


It can be connected to notnull pointers in the following ways:

1. using the following fail clause

func Func() (p ?T, err int) fail err < 0 || p == nil { … }

Personally I do not like this variant...

2. having some more pointer type, the notnullonsuccess pointer, annonated as

func Func() (p !:(T, err int) fail err < 0 { … }

Personally I do not like it either...

3. having some primitive post condition (which would require
defining a class of “primitive expressions” which would not exceed
the capabilities of the future Go compiler)

func Func() (p ?T, err int) success err >= 0 => p != nil { … }

4. fixing just the ugliness with using 1. and overloading the meaning of
fail

func Func() (p ?T, err int) fail err < 0 || p == nil { … }

result, err := Func();

if fail { handle error... return err; } // here the compiler would
replace “fail” with “err < 0 || p == nil”

a := p.a;


Of course I personally prefer 3. but I am interested in other options
and potential problems as well. Instead of 4. I would like an IDE to
automatically fill in the if condition... In the case of 3. it seems
that the compiler could automatically infer from the body of Func() that
=> p != nil but as I have stated at the start, the point of notnull
pointers is to be able to write such contracts into an interface.


BTW I like very much the already proposed exception handling where it is
only possible to catch by starting a goroutine. It will not be as
elegant as the Erlang solution (note that I have never programmed in
Erlang just have read about it) because there is mutable state in Go but
this seems to be the best possible option to me right now. How it can be
added to the Go language is a matter of further debate.



Missing stuff:


Most likely the null value return as EOF in channels would require some
considerations.


Comments about more problems are welcome!

[NoiseEHC]

of course the attachment was missed...

[NoiseEHC]

> Why should anyone do that? If pointer was expected to be set to non-
> null, and was not set, then it's a bug. Let program crash with null-
> pointer exception, nothing can be done anyway.
If we follow your logic neither garbage collection is required (since
not freeing memory or freeing twice is just a bug so the program should
just crash), nor type safety is required (because mistyping a name or
not declaring a member is a bug so just crash).
So you are just using your taste as an absolute rule... Please do not! I
am really not interested in such endless theoretical debates about
personal tastes.

[emghazal]

On Dec 3, 3:24 am, NoiseEHC <Noise...@freemail.hu> wrote:
>
> “if r.a == nil || r.b == nil { halt(); }”

halt()?
Don't you mean panic()? Or am I missing something?

I just wanted to point that out. The topic of non-null types is out of
my scope, and I cannot add anything to the discussion.

[NoiseEHC]

You are right, I just used halt() instead of some throw clause (if it will be ever implemented).
Currently it would be panic() but it is just an implementation detail.

Probably all those "if p != nil { halt(); }" should be inlined by the compiler to "test [eax],eax" and the null pointer exception handler (I think it is called signal handler on UNIX) should treat it as a taken if and just halt the current goroutine.

emghazal wrote:

[NoiseEHC]

Hi!

It was such a shockingly brilliant proposal that you are still
speechless aren't you? :)

Some more missing parts:
How it modifies interfaces pointing to null/notnull structures?
How it relates to the type of slices of arrays of null/notnull structures?

[Jonathan Amsterdam]

> It was such a shockingly brilliant proposal that you are still
> speechless aren't you? :)

I just felt this topic is sort of a dead horse. But to continue
beating it for a moment longer:

I think the whole shadow type thing -- an untried, novel (though
clever!) idea that infects the entire language, all to assist for that
tiny bit of execution time between creation and initialization of non-
null pointers -- is another demonstration of why non-null pointers
don't work for Go. Any solution is considerably more cumbersome than
the problem is worth (in the eyes of the Go designers, who clearly
don't view it as a billion-dollar mistake, and in my eyes as well,
having spent days debugging (or failing to debug) race conditions, but
rarely more than an hour per null-pointer bug).

[NoiseEHC]

> I think the whole shadow type thing -- an untried, novel (though
> clever!) idea that infects the entire language, all to assist for that
> tiny bit of execution time between creation and initialization of non-
> null pointers -- is another demonstration of why non-null pointers
> don't work for Go. Any solution is considerably more cumbersome than
> the problem is worth (in the eyes of the Go designers, who clearly
> don't view it as a billion-dollar mistake, and in my eyes as well,
> having spent days debugging (or failing to debug) race conditions, but
> rarely more than an hour per null-pointer bug).
>

Actually the implementation is just an "initialized bool" per pointer
and as many bools per struct as it has pointers. When all are true ->
not a shadow type. The complexity only stems from the fact that all the
assignment rules need considerations about its resulting type (all the
other data flow checking code is already required in an optimizing
compiler). I could understand if Go designers decided that it did not
worth the effort but I personally expected at least some response like:
"We have evaluated your proposal and decided that it does not worth the
effort...". Never mind, today learned another lesson.

BTW without notnull pointers Go will be just as shitty as java. If you
have ever used eclipse probably you have noticed that there is an
enormous number of null pointer reference errors in the log. Clearly
seems the most frequent bug (because the even more frequent - memory
management - is fixed in java) to me. In the case of eclipse I just hope
that those errors (which are violations of hidden contracts) will not
corrupt my project. Personally I think that making those hidden
contracts explicit would be worthwhile but it is just a windmill I will
not fight that is sure.

ps:
About assisting between creation and initialization:
Notnull pointers do not assist this. They assist the case where nil is a
totally normal value and it makes the propagation of the result of !=
nil checks automatic. After all sooner or later the pointer has to be
checked whether it is nil or not. Currently no routine checks passed in
pointers and a runtime just crashes so this at least forces programmers
to be more careful (but it does not help them at all).

[Qtvali]

> ps:
> About assisting between creation and initialization:
> Notnull pointers do not assist this. They assist the case where nil is a
> totally normal value and it makes the propagation of the result of !=
> nil checks automatic. After all sooner or later the pointer has to be
> checked whether it is nil or not. Currently no routine checks passed in
> pointers and a runtime just crashes so this at least forces programmers
> to be more careful (but it does not help them at all).

Java does a lot of static analysis about code. For example, you can
not put some test code with its own return statement into the
beginning of function, you also must put return into the end of
function which is proven to never reach that point. And there are many
other such features.

So, doing the same thing with notnull plus static analysis would make
it sure that not null check is called and that this variable is never
null. This would mean that if there is a function, which does not
return null with input you give it and it's not set to notnull return
value, you must make an explicit check afterwards and many other
things, including making those nullpointer exceptions somewhat rare.

[NoiseEHC]

> Java does a lot of static analysis about code. For example, you can
> not put some test code with its own return statement into the
> beginning of function, you also must put return into the end of
> function which is proven to never reach that point. And there are many
> other such features.
>
> So, doing the same thing with notnull plus static analysis would make
> it sure that not null check is called and that this variable is never
> null. This would mean that if there is a function, which does not
> return null with input you give it and it's not set to notnull return
> value, you must make an explicit check afterwards and many other
> things, including making those nullpointer exceptions somewhat rare.
>

The problem is that all that static analysis fill fail spectacularly on
package boundaries (and also on the special cases mentioned in the
proposal but they are not that interesting). In case everything is
statically compiled, static analysis will only fail those "not that
interesting special cases" (see the Singularity project how they fixed
it in Sing# which is a Spec# variant for defining interfaces) but who
knows, it may be useful in the normal cases anyway. In Singularity OS
they prohibited loading code dynamically but they can do this because
code generation happens at installation time (.NET IL is bytecode). The
same thing would mean no libraries for Go, or it would make impossible
to update system libraries because of bugfixes or security holes
(because if you can upgrade libraries separately then you cannot compile
the library with the program which uses it). In the case of Singularity
OS, upgrading libraries results in recompilation.

The following cannot be expressed in Go (or Java or C#):
1. I define here an interface and every implementor of this member must
guarantee that he will return an object on success. After this, the
package which defines the interface does not have to check notnullness
because it is guaranteed.
2. I define here a function and every caller must guarantee that he will
not pass in a nil pointer for this parameter. After this, the function
does not have to check notnullness because it is guaranteed.
All the "complexity" is just the minimal functionality required to make
1. and 2. checkable by a simple compiler.

Now it could be (and should have been) debated whether I made a mistake
or whether it is too complex for the goal of eliminating null pointer
references but as we know it will not happen so I just end this message now.

[Tambet]

For arrays and such there could also be then, imho, a way to declare
them non-null types - for example, hashmap is a hashmap and I dont see
any reason to explicitly call a constructor, because I will put there
items anyway. If an object has no special behavior when it's being
constructed, it should act like primitive - just be there when I need
it, set to zero or empty or smth. There would, then, be another
special value in addition to nil to clear it. Constructor would have a
tag for turning this off. Most nullpointers of mine are empty
hashmaps. Some naming convention would differ data and active
variables, which mostly should be initialized. But this is really
complex topic when talking about the results.

Also, for functions it's convinient shortcut to return nil if key
parameter is nil - can also be inlined. In objective-c you can call
methods of nil pointer, they return nil - anyway, "equals" should be
defined for nils, other key error is to compare, log/stringify or copy
them. Also adding singletons to represent nils of specific types and
interfaces would remove much need to check - certain functions
obviously should return false, empty string or 0 when called on them,
like length of nil-string is zero; nil hashmap has zero elements and
all keys return nil, enumeration does nothing (another common error).
This could also propagate errors far away from their sources - for
example, having autoinitialization provides hard bug if several values
are set to point the same object, but actually each contains
different; autoinitialization might also happen on first touch of
value. In many cases, it would also make code to do exactly what you
would expect it to do with nil - for example, adding all elements of
nil set to another set does not change it (this is known from math)
and setting window name to nil would make it empty, doing boolean
operation would consider it false and adding an element sends it
nowhere, it's common in shell scripts with files to use null that way
- but you can also leave the last one unimplemented. Sizeof(nil) would
be 0 and it could be easily reflected (don't know the current
behavior) ...or why having nils at all? Should make them usable. When
adding nil integer to number I would consider it being 0 but throw
exception/halt when multiplying. Converting nil to integer should be
clearly zero ...ok, I'm going too far, but why not give nil's a
meaning beyond stating the meaning explicitly each time? For example,
if price of product is nil, it's bad but authors of book is less and
optional field of form could be. Such interfaces would provide some
guarantee for many kinds of objects. So-called defaults, then.

--
Tambet

[Jessta]

2009/12/7 NoiseEHC <Nois...@freemail.hu>:

> Now it could be (and should have been) debated whether I made a mistake or
> whether it is too complex for the goal of eliminating null pointer
> references but as we know it will not happen so I just end this message now.
>

eg.
ret,err := somefunction();

so ret can be null and err can be null. but both won't be null at the
same time(although this is not currently guaranteed)
In this case both ret and err would have to be declared nullable, thus
would both have to be checked for nil before being used.

mostly I use this pattern
ret,err := somefunction();
if err != nil { //handle the error}
someotherfunction(ret);

In the case of nullable, I'd have to do,
ret,err := somefunction();
if err != nil { print(err.String())}
if ret != nil {
someotherfunction(ret);
} else {
//useless nil check.
}

null pointers are a big problem for java because java has a nasty mix
of null returns and exceptions. Go doesn't have this problem.
It's very likely that in Go code a function that will return nil will
be a function with multiple return.
The added complexity and programmer annoyance isn't worth it. There
will always be hidden contracts between code, solving this one
specific case doesn't solve the problem as a whole.

- Jessta

--
=====================
http://jessta.id.au

[NoiseEHC]

This case is covered at the end of my proposal (exceptions).
In programming languages which handle errors through exceptions (for
example Java and C#) this is solved by the fact that when an exception
is thrown then the return value does not get assigned. And while those
languages does not have notnull pointers, they do catch using
uninitialized local variables so this works.
In your example either err == nil in which case it does not matter
whether ret is null or not, or err != nil which implies ret != nil. The
solution is that the function definition must explicitly state the error
condition (BTW the success condition results in better looking code). It
would also be a solution to have exceptions in a Go idiomatic way...

[Ian Lance Taylor]

NoiseEHC <Nois...@freemail.hu> writes:

>> I think the whole shadow type thing -- an untried, novel (though
>> clever!) idea that infects the entire language, all to assist for that
>> tiny bit of execution time between creation and initialization of non-
>> null pointers -- is another demonstration of why non-null pointers
>> don't work for Go. Any solution is considerably more cumbersome than
>> the problem is worth (in the eyes of the Go designers, who clearly
>> don't view it as a billion-dollar mistake, and in my eyes as well,
>> having spent days debugging (or failing to debug) race conditions, but
>> rarely more than an hour per null-pointer bug).
>>
> Actually the implementation is just an "initialized bool" per pointer
> and as many bools per struct as it has pointers. When all are true ->
> not a shadow type. The complexity only stems from the fact that all
> the assignment rules need considerations about its resulting type (all
> the other data flow checking code is already required in an optimizing
> compiler). I could understand if Go designers decided that it did not
> worth the effort but I personally expected at least some response
> like: "We have evaluated your proposal and decided that it does not
> worth the effort...". Never mind, today learned another lesson.

I looked over your proposal but it was, as you said, half-finished.
You skimmed over the details of when and how the language should
handle "if (p != nil)"; since you are proposing to change the
language, that is no longer a compiler detail, it is a language
construct.

You did write an implementation of ring, nicely done, but the result
seemed to me to be overly complex with some duplicated code. Also, a
ring of zero elements is perfectly natural and should not be
disallowed.

The introduction of shadow types for every struct type complicates the
language significantly. I think it is very unlikely that we would
adopt any suggestion along these lines into the language. It might
work better to look into explicitly nullable types--but I'm certainly
not saying that any such feature would be adopted either.

Mixing this idea with exceptions is putting the cart before the horse:
do the exceptions first!

Ian

[NoiseEHC]

> I looked over your proposal but it was, as you said, half-finished.
> You skimmed over the details of when and how the language should
> handle "if (p != nil)"; since you are proposing to change the
> language, that is no longer a compiler detail, it is a language
> construct.
>
>

It is half finished because it reached such a size that I was thinking
that if there was some hidden conceptual errors then failing early with
the proposition would be much-much better to me than writing it for
another two weeks.

BTW It should handle if(p != nil) when there is an if(p != nil) in the
code...

> You did write an implementation of ring, nicely done, but the result
> seemed to me to be overly complex with some duplicated code. Also, a
> ring of zero elements is perfectly natural and should not be
> disallowed.
>
>

You have misunderstood that part.

First, it handles the zero element ring as nil.

Second, I did not write ring.go, I just converted the existing one into
the notnull pointer style. Unfortunately there are two main problems
with ring.go:

1. It is an example where notnullness does not make things easier
because you know the pointers in rings are never null so by definition
notnullness can never help in this case. It just makes things uglier
(but not too much).

2. What makes ring.go really ugly is that the original one has a broken
API. The ring allocating function has to work for 0 and negative values
as well which is just plain wrong!!! It is wrong conceptually that
New(-1) makes the same result as New(0). New(-1) should simply be an
error in my opinion. What a bigger problem is that either the caller has
to handle the empty ring differently or the ring should check in every
function whether the ring is empty. The original ring.go simply crashed
when called with empty rings (except Len() and Iter()) and it even had
comments about not to call with empty rings but we both know that nobody
reads comments... Now if all ring functions would check if the ring is
empty then we would not need notnull pointers here at all (but would
have some runtime errors...). If we leave the checking to the caller
then he has to check whether the ring is empty (which is an if "ring !=
nil" check) before he calls any of those functions unless he wants to
crash the program and those are exactly the same checks needed for
converting the nullable ring pointer into a notnull ring pointer. And
this is not a lucky coincidence but my belief is that almost all the
code which is not braindamaged lies in this category. In the allocation
case, instead of calling FunnyNew (which is unnecessary IMHO) the caller
should simply assign nil (BTW the caller would not even reach the
allocation because it would notice before that "Hey, we have 0 elements,
do not even enter those routines which make things with elements because
we do not have any of them...").

What that braindamaged example showed in the proposition is the real
ugliness of notnull pointers where the notnullness would follow from
program state other than whether the pointer != nil. In that example n >
0 would imply r != nil so the code would not check for nil but would
have to because of the notnull requirement the program would not compile
otherwise. Of course it is a stupid example because nobody would write
such code but it shows a type of algorithms which would be problematic
to write with notnull pointers. Another such example would be checking
the emptiness of ring by "if r.Len() != 0 { .... }". I hope that this is
clear now.

Braindamaged code IMHO:
list PushFront/PushBack
ring Next/Prev/Move with uninitialized rings

The ring which I would have been written is attached...

> The introduction of shadow types for every struct type complicates the
> language significantly. I think it is very unlikely that we would
> adopt any suggestion along these lines into the language. It might
> work better to look into explicitly nullable types--but I'm certainly
> not saying that any such feature would be adopted either.
>

It would add different * and ? notation and the shadow type which will
not be used directly by 99% of programmers. It would also add type
conversion rules which are totally natural because this is the minimum
required support. I can understand if it is too much for eliminating
null pointer crashes but calling it *significant* complication is a
little bit unrealistic to me. However it is your call so that was my
last windmill for a while (I find it very unlikely that you will change
your mind when you will finally understand what I wanted to show with
ring.go).

> Mixing this idea with exceptions is putting the cart before the horse:
> do the exceptions first!
>
>

Originally I started with exceptions just reversed the order since
nutnull pointers come up on the mailing list and I have already
translated list.go...

[Ian Lance Taylor]

NoiseEHC <Nois...@freemail.hu> writes:

>> I looked over your proposal but it was, as you said, half-finished.
>> You skimmed over the details of when and how the language should
>> handle "if (p != nil)"; since you are proposing to change the
>> language, that is no longer a compiler detail, it is a language
>> construct.
>>
>>
> It is half finished because it reached such a size that I was thinking
> that if there was some hidden conceptual errors then failing early
> with the proposition would be much-much better to me than writing it
> for another two weeks.
>
> BTW It should handle if(p != nil) when there is an if(p != nil) in the
> code...

My point is that your proposal appeared to rely on the language
handling "if p != nil" in a certain way. Unless I misunderstand, that
is how you implicitly convert a ?T into a *T. That needs to be
spelled out in a way that always works.

>> You did write an implementation of ring, nicely done, but the result
>> seemed to me to be overly complex with some duplicated code. Also, a
>> ring of zero elements is perfectly natural and should not be
>> disallowed.
>>
>>
> You have misunderstood that part.

My apologies.

> First, it handles the zero element ring as nil.

But ring.New(0) fails.

> 1. It is an example where notnullness does not make things easier
> because you know the pointers in rings are never null so by definition
> notnullness can never help in this case. It just makes things uglier
> (but not too much).

It's exactly because the pointers are never null, but that allocation
nevertheless returns null values, that ring.go is interesting.

> 2. What makes ring.go really ugly is that the original one has a
> broken API. The ring allocating function has to work for 0 and
> negative values as well which is just plain wrong!!! It is wrong
> conceptually that New(-1) makes the same result as New(0). New(-1)
> should simply be an error in my opinion.

New(0) should work. New(-1) is not very interesting. I think it
would be fine to simply have it fail.

> What a bigger problem is that
> either the caller has to handle the empty ring differently or the ring
> should check in every function whether the ring is empty. The original
> ring.go simply crashed when called with empty rings (except Len() and
> Iter()) and it even had comments about not to call with empty rings
> but we both know that nobody reads comments...

Calling Next or Prev on an empty ring is in effect an out-of-bounds
array access. A Go program will crashe on an out-of-bounds array
access. It seems perfectly reasonable to me that it should crash on a
Next or Prev call on an empty ring. What else should happen?

> It would add different * and ? notation and the shadow type which will
> not be used directly by 99% of programmers. It would also add type
> conversion rules which are totally natural because this is the minimum
> required support. I can understand if it is too much for eliminating
> null pointer crashes but calling it *significant* complication is a
> little bit unrealistic to me.

For example, you haven't considered the effect on type reflection and
type assertions. It's going to get increasingly complicated.

Thanks for the updated Ring. It seems to me that any Ring which can
be empty will have to be written as ?Ring. Then, if I understand
correctly, code like
if ring.Len() != 0 {
next = ring.Next();
}
won't compile. You will have to write
if ring != nil {
next = ring.Next();
}
which I guess is OK though it may be confusing.

Ian

[NoiseEHC]

My point is that your proposal appeared to rely on the language
handling "if p != nil" in a certain way.  Unless I misunderstand, that
is how you implicitly convert a ?T into a *T.  That needs to be
spelled out in a way that always works.
  

It does not convert types. It simply allows assignments between types and passing types to functions and checks returned types.
A p of type ?T in an "if p != nil { ... }" block will not have a type of *T but will be able to be assigned to a *T or passed into a function which requires *T (and can be dereferenced). It can be also be returned as *T. (And meaning that after an "if p == nil { halt(); }" the remaining part of the function is in an "if p != nil { ... }" block but it seems trivial to me.)
I think these rules are spelled out in the proposal (somehow I always mix the words proposal and proposition...).

But ring.New(0) fails.

  

Because the caller should not allocate a zero length ring. All pointers to rings are zero length rings by default.

1. It is an example where notnullness does not make things easier
because you know the pointers in rings are never null so by definition
notnullness can never help in this case. It just makes things uglier
(but not too much).
    

It's exactly because the pointers are never null, but that allocation
nevertheless returns null values, that ring.go is interesting.

  

I do not think so. The notnull pointer proposal correctly handles this part. Where it requires ugliness is the allocation part where a *theoretical* AI compiler should invent a "ringness" invariant for the ring and an "almostringness" invariant for the loop, and then prove that at the last step the ring allocation code satisfies "ringness". The interesting thing is that the proposal makes this circular self-reference initialization ugly (but not too much). My instinct tells me that almost all real word things will be just a little bit more ugly or none at all (and it will make the calling code much prettier in exchange). Now this last sentence *should have been attacked* in the conversation following my proposal (by using real word examples).

2. What makes ring.go really ugly is that the original one has a
broken API. The ring allocating function has to work for 0 and
negative values as well which is just plain wrong!!! It is wrong
conceptually that New(-1) makes the same result as New(0). New(-1)
should simply be an error in my opinion.
    

New(0) should work.  New(-1) is not very interesting.  I think it
would be fine to simply have it fail.

  

New(0) should be never called. See above.

What a bigger problem is that
either the caller has to handle the empty ring differently or the ring
should check in every function whether the ring is empty. The original
ring.go simply crashed when called with empty rings (except Len() and
Iter()) and it even had comments about not to call with empty rings
but we both know that nobody reads comments... 
    

Calling Next or Prev on an empty ring is in effect an out-of-bounds
array access.  A Go program will crashe on an out-of-bounds array
access.  It seems perfectly reasonable to me that it should crash on a
Next or Prev call on an empty ring.  What else should happen?

  

The compiler should not compile a program which has the possibility to crash that way. The real question is whether this proposal will allow detecting this for enough algorithms and whether those cases justify the added complexity. Of course it would require implementing the proposal (LOT of work!!!) so first this proposal should have been invalidated by real world examples if possible (see above).

It would add different * and ? notation and the shadow type which will
not be used directly by 99% of programmers. It would also add type
conversion rules which are totally natural because this is the minimum
required support. I can understand if it is too much for eliminating
null pointer crashes but calling it *significant* complication is a
little bit unrealistic to me.
    

For example, you haven't considered the effect on type reflection and
type assertions.  It's going to get increasingly complicated.
  

That is true. Also needs considering pointers to interfaces and pointers returned from closed channels and slices of array of notnull pointers. These considerations mean a lot of work and since you have already decided I do not know whether it meaningful talking about them. Normally it would have been looked like this:
1. Attack proposal that it has some big flaws.
2. If survives then try to finish it by using the collective knowledge of this list.
3. Evaluate positive and negative consequences.
4. If positive wins over negative then implement 0.1.
5. If it turns out to be more pain than gain then drop the proposal.
Unfortunately it did not work this way on this mailing list.

Thanks for the updated Ring.  It seems to me that any Ring which can
be empty will have to be written as ?Ring.

Correct. Ring is special because unlike other objects it has a special meaning for nil value. However the calling program has to treat that special value specially.

  Then, if I understand
correctly, code like
	if ring.Len() != 0 {
		next = ring.Next();
	}
won't compile.  You will have to write
	if ring != nil {
		next = ring.Next();
	}
which I guess is OK though it may be confusing.

  

Theoretically yes. Practically you will not write such code ever (otherwise notnull pointers would not have been working in Spec# - or Eiffel as I heard about that).

[Ian Lance Taylor]

NoiseEHC <Nois...@freemail.hu> writes:

>> My point is that your proposal appeared to rely on the language
>> handling "if p != nil" in a certain way. Unless I misunderstand, that
>> is how you implicitly convert a ?T into a *T. That needs to be
>> spelled out in a way that always works.
>>
> It does not convert types. It simply allows assignments between types
> and passing types to functions and checks returned types.
> A p of type ?T in an "if p != nil { ... }" block will not have a type
> of *T but will be able to be assigned to a *T or passed into a
> function which requires *T (and can be dereferenced). It can be also
> be returned as *T. (And meaning that after an "if p == nil { halt();
> }" the remaining part of the function is in an "if p != nil { ... }"
> block but it seems trivial to me.)
> I think these rules are spelled out in the proposal (somehow I always
> mix the words proposal and proposition...).

Let me try again. You have to spell out in precise detail when the
language permits an assignment from ?T to *T. For example, is this
OK?
if p.next != nil {
var x *T = p.next;

}


>> Calling Next or Prev on an empty ring is in effect an out-of-bounds
>> array access. A Go program will crashe on an out-of-bounds array
>> access. It seems perfectly reasonable to me that it should crash on a
>> Next or Prev call on an empty ring. What else should happen?
>>
>>
> The compiler should not compile a program which has the possibility to
> crash that way. The real question is whether this proposal will allow
> detecting this for enough algorithms and whether those cases justify
> the added complexity. Of course it would require implementing the
> proposal (LOT of work!!!) so first this proposal should have been
> invalidated by real world examples if possible (see above).

I understand that you are not proposing anything about out-of-bounds
array access, but I am curious as to your position on how those should
be handled. Am I wrong in seeing the cases as related?

Ian

[Ken Thompson]

i agree with ian, there is an isomorphism between
dynamic (vs static) catching of an array/slice index
out of bounds and a reference thru a nil pointer.
this discussion concentrates on pointers and leaves
arrays alone. they are exactly the same.

maybe this can be handled outside of the compiler.
how about someone converting ESC (extended static check)
as a (non-compiler) tool for go. this sounds like value-added
for both camps. my fear then is that code will become mucked up
with lots of stupid annotations. my case in point is
(void)f();
as a totally stupid side-effect of running lint.

[NoiseEHC]

> Let me try again. You have to spell out in precise detail when the
> language permits an assignment from ?T to *T. For example, is this
> OK?
> if p.next != nil {
> var x *T = p.next;
> }
>

From the proposal:

“p = q” only allowed if it can be inferred from an “if q != nil” or from
a “q = qq” where qq *T that q is not null.

So yes, it is OK.

> I understand that you are not proposing anything about out-of-bounds
> array access, but I am curious as to your position on how those should
> be handled. Am I wrong in seeing the cases as related?
>
>

They are absolutely similar. And as I see you have already invented
slices!!! You just do not like notnull pointers...

The main difference is that statical array bound checking requires first
order logic while pointers do not. In the general sense neither problem
can be solved generally (== halting problem) but my instinct tells me
(and we know from history) that at least notnull pointers can be done
practically. (This is equivalent with the attackable sentence in the
prior message.) I do not think that I could propose a similar proposal
for array indices so I just do not do it. Slices should make array
indices much more safe and that is all we can do about that (that is my
belief).

[NoiseEHC]

> i agree with ian, there is an isomorphism between
> dynamic (vs static) catching of an array/slice index
> out of bounds and a reference thru a nil pointer.
> this discussion concentrates on pointers and leaves
> arrays alone. they are exactly the same.
>

See the answer to Ian.

> maybe this can be handled outside of the compiler.
> how about someone converting ESC (extended static check)
> as a (non-compiler) tool for go. this sounds like value-added
> for both camps. my fear then is that code will become mucked up
> with lots of stupid annotations. my case in point is
> (void)f();
> as a totally stupid side-effect of running lint.
>

An outside static analyzer (which is equivalent with Ian's proposal of
inferring notnullness automatically) will not work for the reasons I
have already mentioned. It is possible that explicitly annotating the
program with ? and * will not work either but replacing it with a
solution for which we know it does not work seems lame.
The bigger problem is that if you do not do notnull pointers now
(assuming that it does work) then it is guaranteed that you will not be
able to fix the platform later. For more information ask the Spec#
developers about the pain they suffered...

[Ben Tilly]

On Mon, Dec 7, 2009 at 3:43 PM, NoiseEHC <Nois...@freemail.hu> wrote:
>
>> Let me try again.  You have to spell out in precise detail when the
>> language permits an assignment from ?T to *T.  For example, is this
>> OK?
>>        if p.next != nil {
>>                var x *T = p.next;
>>        }
>>
>
> From the proposal:
> “p = q” only allowed if it can be inferred from an “if q != nil” or from a
> “q = qq” where qq *T that q is not null.
>
> So yes, it is OK.

[...]

You've missed Ian's point. That snippet isn't threadsafe. If another
thread is manipulating p then the value of p.next could change from
testing the conditional to doing the assignment.

Of course under the hood the compiler could convert that into:

next := p.next;
if nil != next {
var x *T = next;
}

and now it is threadsafe. But now you've imposed strong and
non-obvious restrictions on what the compiler can do without creating
potential threading issues. Over multiple compilers for Go, it is
inevitable that compiler bugs will accidentally cause threadsafe code
to turn into non-threadsafe executables.

Cheers,
Ben

[Ian Lance Taylor]

inspector_jouve <kaush...@gmail.com> writes:

> Both errors (null-pointer and array-index-out-of-bounds) are just
> variants of more general
> kind of error: variable-out-of-range. What about integer that happens
> to fall out of range?

To address this specific point, in Go, division by zero is a crash.
For other cases, the language will always return an in-range value,
although if there is an overflow it may not be what you want.

http://golang.org/doc/go_spec.html#Integer_overflow

Ian

[Ian Lance Taylor]

NoiseEHC <Nois...@freemail.hu> writes:

>> Let me try again. You have to spell out in precise detail when the
>> language permits an assignment from ?T to *T. For example, is this
>> OK?
>> if p.next != nil {
>> var x *T = p.next;
>> }
>>
> From the proposal:
> “p = q” only allowed if it can be inferred from an “if q != nil” or
> from a “q = qq” where qq *T that q is not null.
>
> So yes, it is OK.

Aside from the thread safety issue, here we are talking about the
language, not any particular implementation. We have to be able to
write a specification such that all implementations of the language do
the same thing. "It can be inferred" is not a specification. How is
this inference done? When does it work and when does it fail?

>> I understand that you are not proposing anything about out-of-bounds
>> array access, but I am curious as to your position on how those should
>> be handled. Am I wrong in seeing the cases as related?
>>
>>
> They are absolutely similar. And as I see you have already invented
> slices!!! You just do not like notnull pointers...

Slices are essentially a safe way to do pointer arithmetic. I don't
really see them as related to out-of-bounds array access. You can
have out-of-bounds access to a slice, too.

> The main difference is that statical array bound checking requires
> first order logic while pointers do not. In the general sense neither
> problem can be solved generally (== halting problem) but my instinct
> tells me (and we know from history) that at least notnull pointers can
> be done practically. (This is equivalent with the attackable sentence
> in the prior message.) I do not think that I could propose a similar
> proposal for array indices so I just do not do it. Slices should make
> array indices much more safe and that is all we can do about that
> (that is my belief).

We know that there are other languages which supports non-null
pointers. What do those languages do about out-of-bounds array
accesses?

Ian

[NoiseEHC]

> You've missed Ian's point.

What I missed is why he asked questions which are already answered in
the proposal?

> That snippet isn't threadsafe. If another
> thread is manipulating p then the value of p.next could change from
> testing the conditional to doing the assignment.
>
> Of course under the hood the compiler could convert that into:
>
> next := p.next;
> if nil != next {
> var x *T = next;
> }
>
> and now it is threadsafe. But now you've imposed strong and
> non-obvious restrictions on what the compiler can do without creating
> potential threading issues. Over multiple compilers for Go, it is
> inevitable that compiler bugs will accidentally cause threadsafe code
> to turn into non-threadsafe executables.
>
>

From the proposal:
“p = q” only allowed if it can be inferred from an “if q != nil” or from

a “q = qq” where qq *T that q is not null. *In this case the compiler

must use the cached q value to avoid inserting reads to the possibly

volatile location of q*.

It means that the compiler *must* convert it into exactly that code (and
optimize away later if possible).

Now the last part seems to me a little bit red herring because
inevitable compiler bugs will accidentally cause working code to turn
crashing or data destroying so I do not see why thread safety issues
should be treated so specially.

There is a bigger problem with Go which is that its defined memory model
is not defined. I mean that you cannot write any clever tricks with it
or create locking primitives or similar. I can understand that Go
designers want to force people to use channels (or just hate
programmers) but it will bite back badly later I promise you unless you
cover every one of the concurrency primitives in packages (like double
checked locking). But the point is that because of this memory model
there is an absolutely not defined unsynchronized behavior so this
possible-changing-compiler-bug problem is the least you should care about.

(This hate programmers was intended as a joke...)

ps:
What I do not see is where the published memory model defines that zero
initialization of an allocated memory block fences itself from the
storage of the pointer to the memory block? I mean if you allocate a
struct and store the pointer into a global, then another thread could
observe a garbage non zeroed pointer member and dereference it. Good bye
type safety.

[Nigel Tao]

2009/12/8 NoiseEHC <Nois...@freemail.hu>:

>> You've missed Ian's point.
>
> What I missed is why he asked questions which are already answered in the
> proposal?

His question, which I don't think is answered by the proposal, is what
does "can be inferred from" mean exactly, when you say "“p = q” only

allowed if it can be inferred from an “if q != nil” or from a “q = qq”
where qq *T that q is not null".

Which of these cases are OK (the p variables are of type *T and q
variables are of type ?T)?

if q == nil {
// do something
} else {
p = q;
}

What about?

if q != nil {
f(r, s, t); // Does it matter if f can modify q?
p = q;
}

What about?

if q != nil {
p0 = q;
p1 = q;
}

What about?

if s > t && q != nil {
p = q;
}

What about?

a1 := a0;
if a0.q != nil {
p = a1.q;
}


If some of those are OK and others aren't, what's the specification
for telling which is which?

[NoiseEHC]

Now I see. Thanks!

You showed two problems with notnull pointers.

1. While it works with local values, it can fail with aliased struct
members (or arrays).

> What about?
>
> if q != nil {
> f(r, s, t); // Does it matter if f can modify q?
> p = q;
> }
>

As in the proposal it would use the cached q but it could be very
confusing if f() modifies q.

2. It can result in a compatibility nightmare when the later, more
advanced compiler can infer more than the prior one and so the code
which is legal in the later Go cannot be compiled with the prior one.

> What about?
>
> if s > t && q != nil {
> p = q;
> }
>
>

Problematic if the more advanced compiler can infer s > t to be always
true and the older one cannot.

> What about?
>
> a1 := a0;
> if a0.q != nil {
> p = a1.q;
> }
>
>

Problematic if the more advanced compiler can detect aliasing and the
older one cannot.

> If some of those are OK and others aren't, what's the specification
> for telling which is which?
>
>

Actually all your concerns were "defined" (mentioned) in the proposal
but now I see that this does not give us a notconfusing language.

"Note that the programmer can always use “if r.a == nil || r.b == nil {
halt(); }” when the compiler cannot infer from simple data flow analysis
the notnullness of pointers (the compiler can always assume nullable
pointers if in doubt). In other words, inferring the notnullness of
pointers does not have to be foolproof, the programmer should follow Go
idioms for which this mechanism hopefully works. "

So I will make a more specified proposal (will define "can be inferred"
more strictly) and will think about whether the defined behavior in 1.
is what we really want.

Thanks for the good examples.

ps:
Are there more problematic examples?

[Jonathan Amsterdam]

> So I will make a more specified proposal (will define "can be inferred"
> more strictly) and will think about whether the defined behavior in 1.
> is what we really want.

You may want to take a look at how Eiffel handled this. They added a
special language construct that introduces a fresh variable.
http://docs.eiffel.com/sites/default/files/void-safe-eiffel.pdf

[Ian Lance Taylor]

NoiseEHC <Nois...@freemail.hu> writes:

> What I do not see is where the published memory model defines that
> zero initialization of an allocated memory block fences itself from
> the storage of the pointer to the memory block? I mean if you allocate
> a struct and store the pointer into a global, then another thread
> could observe a garbage non zeroed pointer member and dereference
> it. Good bye type safety.

That is intentional. In general, if one goroutine assigns a value to
a global variable, another goroutine may not look at that global
variable until it has synchronized via a channel communication or a
mutex.

This part of the Go memory model is essentially the same as C++0x (C++
before C++0x has no multi-thread memory model). In C++0x, if one
thread sets a global variable, another thread may only look at it if
some release/acquire operation has taken place (in C++0x, typically
via a variable explicitly declared to be atomic).

In other words, you have correctly observed that you can not use
changes to global variables as synchronization points.

Ian

[NoiseEHC]

In the attached example can consume crash on "*p = 64738;" ?

While in produce "s.global = new(Object);" guarantees that reading "s.global.unsafe" always returns nil, in consume "var p *int = s.global.unsafe;" can read the garbage value from the heap which was there before new() zeroed out the memory block.

Am I right or wrong? Where is it in the memory model?

ps:
I will answer your last unanswered message just it will take some time.

[Ian Lance Taylor]

NoiseEHC <Nois...@freemail.hu> writes:

> In the attached example can consume crash on "*p = 64738;" ?

Yes.

> While in produce "s.global = new(Object);" guarantees that reading
> "s.global.unsafe" always returns nil, in consume "var p *int =
> s.global.unsafe;" can read the garbage value from the heap which was
> there before new() zeroed out the memory block.
>
> Am I right or wrong? Where is it in the memory model?

You are right: the memory model permits this program to crash.

That is what I tried to say in the message to which you replied:

That is intentional. In general, if one goroutine assigns a value
to a global variable, another goroutine may not look at that
global variable until it has synchronized via a channel
communication or a mutex.

Ian

[NoiseEHC]

Okay, that was the question whether this was a deliberate decision.

Now this decision unfortunately will make creating a sandboxed Go environment impossible. Please reconsider. (This little bit is what makes Go's memory model different from the .NET v1 memory model if I am not mistaken.)

The other problem is that there is no volatile or memory barrier in the language. It means that there will be no way to reliably implement double locking (or similar) because the compiler can freely reorder reads and writes in a function as long as the current thread cannot notice it. You know, exactly this situation happened in Java and a lot of Java code was written which took advantage of the compiler's implementation behavior (and broke later). And so the Java spec had to be changed... Do not make the same mistake please.ďż˝ (Here I am not talking about processor synchronization primitives but about some construct which could tell the compiler not to reorder reads and writes over this fence. Of course the programmer should use processor synchronization primitives explicitly in addition to this compiler fence.)

(I have to admit that I could live with this second problem but the first one is very-very important to fix IMHO.)

[Ian Lance Taylor]

NoiseEHC <Nois...@freemail.hu> writes:

> Now this decision unfortunately will make creating a sandboxed Go
> environment impossible. Please reconsider. (This little bit is what
> makes Go's memory model different from the .NET v1 memory model if I
> am not mistaken.)

Can you explain this further?

Note that any other decision has some very serious performance costs
when using global variables. .NET is quite different since (as I
understand it) .NET code is run by a JIT.

> The other problem is that there is no volatile or memory barrier in
> the language. It means that there will be no way to reliably implement
> double locking (or similar) because the compiler can freely reorder
> reads and writes in a function as long as the current thread cannot
> notice it. You know, exactly this situation happened in Java and a lot
> of Java code was written which took advantage of the compiler's
> implementation behavior (and broke later). And so the Java spec had to

> be changed... Do not make the same mistake please. (Here I am not

> talking about processor synchronization primitives but about some
> construct which could tell the compiler not to reorder reads and
> writes over this fence. Of course the programmer should use processor
> synchronization primitives explicitly in addition to this compiler
> fence.)

In my experience, the number of people who can write correct
multi-processor code using memory barriers is very small. One of the
things that initially attracted me personally to Go was the chance to
use a language where it was easy and natural to write multi-processor
code which works correctly. (Even using the word "volatile" in this
context leads to misunderstanding; I wrote this blog entry before I
heard about Go: http://www.airs.com/blog/archives/154 ).

The Go memory model explicitly defines sync.Mutex, sync.RWMutex, and
once.Do. Use those any time you might be tempted to use memory
barriers.

Ian

[hong]

> I understand that you are not proposing anything about out-of-bounds
> array access, but I am curious as to your position on how those should
> be handled.  Am I wrong in seeing the cases as related?

How about this idea? Today, Go process crashes on nil pointer access.
If we assume *T is pointer, and ?T is nullable pointer. Accessing *T
will
never get null pointer crash, and access ?T may crash. We can define
the behavior that assign from ?T to *T will crash immediate if the
pointer
happens to be nil. This semantics is in line with current Go style and
will need very minimum change to the language and runtime.

BTW, I am not sure where we are on this issue. If we agree a decent
not-null is a good thing for Go, then finding the right syntax and
semantics
and style is just matter of time. I am not sure if we agreed upon the
first part yet.

Hong

[Ian Lance Taylor]

I think the Go developers are generally skeptical. I think Jonathan
Amsterdam summed up the issues very well a couple of weeks ago. I'm
personally not sure that having *T = ?T crash at run time brings a
benefit that is greater than the cost.

Ian

[Russ Cox]

> What I do not see is where the published memory model defines that zero
> initialization of an allocated memory block fences itself from the storage
> of the pointer to the memory block? I mean if you allocate a struct and
> store the pointer into a global, then another thread could observe a garbage
> non zeroed pointer member and dereference it. Good bye type safety.

The memory model doesn't need to say that,
because from a language definition point of view,
memory is never reused. It's just the implementation
that has to concern itself with such details.
The memory model is about what programs should
do, not what the underlying implementation has to do.

Russ

[Hong Zhang]

> I think the Go developers are generally skeptical.  I think Jonathan
> Amsterdam summed up the issues very well a couple of weeks ago.  I'm
> personally not sure that having *T = ?T crash at run time brings a
> benefit that is greater than the cost.

This is a judgment call and different people sure have different opinions
on it. Having *T as not-null make a lot of code much more reasonable
and reduce many unnecessary null check, which can reduce codebase
by couple of percent. The runtime cost will be a dereference at *T = ?T,
which is a single instruction generated by compiler. It will add some
complexity to the language, which will be the greatest cost.

BTW, if we ever want to do it, please consider using &T and *T. &T
is like T& in C++, which is reference to something. *T will continue
to be nullable so it compatible with C/C++ and current Go codebase.

If Go ever has exception, then *T = ?T will be nil exception. It is nicer
to have it happen explicit location instead of arbitrary places.

Hong

[Ian Lance Taylor]

Hong Zhang <ho...@google.com> writes:

>> I think the Go developers are generally skeptical.  I think Jonathan
>> Amsterdam summed up the issues very well a couple of weeks ago.  I'm
>> personally not sure that having *T = ?T crash at run time brings a
>> benefit that is greater than the cost.
>
> This is a judgment call and different people sure have different opinions
> on it.

Well, yes.

> It will add some
> complexity to the language, which will be the greatest cost.

I agree.

> BTW, if we ever want to do it, please consider using &T and *T. &T
> is like T& in C++, which is reference to something. *T will continue
> to be nullable so it compatible with C/C++ and current Go codebase.

Unfortunately I suspect that using &T would complicate the parser,
since we already permit code like &T{0}.

Ian

[Russ Cox]

> BTW, if we ever want to do it, please consider using &T and *T. &T
> is like T& in C++, which is reference to something. *T will continue
> to be nullable so it compatible with C/C++ and current Go codebase.

I think it's unlikely that something like this will happen
(all the discussions seem to end up concluding that it doesn't
fit well into a language like Go), but you're right that it's a mistake
to redefine the meaning of the * in *T. Instead of &T for non-null,
which has the problem Ian pointed out, you could use +T,
since + already means "non-empty version of *" in other contexts.

Russ

[NoiseEHC]

Now this decision unfortunately will make creating a sandboxed Go
environment impossible. Please reconsider. (This little bit is what
makes Go's memory model different from the .NET v1 memory model if I
am not mistaken.)
    

Can you explain this further?

Note that any other decision has some very serious performance costs
when using global variables.  .NET is quite different since (as I
understand it) .NET code is run by a JIT.

  

Now here is the specific part from the ECMA .NET spec:
http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-335.pdf

"12.6.8 Other memory model issues
All memory allocated for static variables (other than those assigned RVAs within a PE file, see Partition II) and
objects shall be zeroed before they are made visible to any user code.
A conforming implementation of the CLI shall ensure that, even in a multi-threaded environment and without
proper user synchronization, objects are allocated in a manner that prevents unauthorized memory access and
prevents invalid operations from occurring. In particular, on multiprocessor memory systems where explicit
synchronization is required to ensure that all relevant data structures are visible (for example, vtable pointers)
the Execution Engine shall be responsible for either enforcing this synchronization automatically or for
converting errors due to lack of synchronization into non-fatal, non-corrupting, user-visible exceptions.
It is explicitly not a requirement that a conforming implementation of the CLI guarantee that all state updates
performed within a constructor be uniformly visible before the constructor completes. CIL generators can
ensure this requirement themselves by inserting appropriate calls to the memory barrier or volatile write
instructions."

While it does allow buggy code (when a freshly constructed object's pointer is observed by another thread before the stores in the constructor are observed by that thread), it does not allow that the other thread observe a not-yet-zeroed-out memory block.

The problem is that this allowed behavior in Go allows breaking type safety without using the unsafe package or something like that. This type safety breaking can lead to process memory corruption or breaking out of sandboxing. By sandboxing I mean implementing later some mechanism which would restrict access to certain "safe" packages to loadable modules, allowing only transparent modules to load. A transparent module could theoretically break out of this sandboxing and call into an "unsafe" API. (Please, I know that this does not replace OS security but it has its uses when later will be hopefully implemented.)

The two things which can break in the current Go memory model:
1. Zeroing memory at object allocation can be observed by another thread later than that thread observes the pointer pointing to the still uninitialized memory.
2. Interfaces have two processor word sized entries and so will be observed by another thread independently (can observe a vtable for type1 while a pointer to a struct of type2).
3. Can similar problems happen with channels and slices?
The first problem can be fixed by a release fence at the end of the zeroing out (no-op on x86) or by zeroing out the memory on garbage collection (this latter is very bad for caches IMHO).
The second can be fixed by writing strongly (simple write on x86) a zero to the object pointer before updating an interface which is not on the stack.

That way those programs would be just as buggy as you like it but at least could not break type safety.


Now the availability of the necessarily weak hardware is another question but at least fixing the memory model document would be fine.
You should better read this:
http://blogs.msdn.com/cbrumme/archive/2003/05/17/51445.aspx

[NoiseEHC]

The memory model should also define what the underlying implementation guarantees to the programs. The current memory model document does not guarantee type safety if I am right. Note that probably it does not matter because the implementation will fix those issues (hopefully) but writing 5 lines to the definition would not hurt in my opinion.

[Russ Cox]

> The two things which can break in the current Go memory model:
> 1. Zeroing memory at object allocation can be observed by another thread
> later than that thread observes the pointer pointing to the still
> uninitialized memory.

People keep saying this, but it's not true.
Variables start out zeroed. This is part of the
language spec, not an operational definition.
The implementation must make sure that
zeroed means zeroed. But the language definition
says nothing about reusing memory, so the memory
model doesn't need to define how memory gets reused.

> 2. Interfaces have two processor word sized entries and so will be observed
> by another thread independently (can observe a vtable for type1 while a
> pointer to a struct of type2).

We know how to fix this: you have to make an interface value
a pointer to those two words, never overwrite them, and let
the garbage collector clean up the mess.

> 3. Can similar problems happen with channels and slices?

Slices yes, channels no. The fix for slices is similar but can
still avoid reallocation on each slice operation.

> The second can be fixed by writing strongly (simple write on x86) a zero to
> the object pointer before updating an interface which is not on the stack.

No it can't. You have to change the memory layout.

Russ

[NoiseEHC]

2. Interfaces have two processor word sized entries and so will be observed
by another thread independently (can observe a vtable for type1 while a
pointer to a struct of type2).
    

We know how to fix this: you have to make an interface value
a pointer to those two words, never overwrite them, and let
the garbage collector clean up the mess.

  

So you want to increase the size of every struct by a pointer?

3. Can similar problems happen with channels and slices?
    

Slices yes, channels no.  The fix for slices is similar but can
still avoid reallocation on each slice operation.

  

The second can be fixed by writing strongly (simple write on x86) a zero to
the object pointer before updating an interface which is not on the stack.
    

No it can't.  You have to change the memory layout.

  

Why does not it work?

[Ian Lance Taylor]

NoiseEHC <Nois...@freemail.hu> writes:

> The memory model should also define what the underlying implementation
> guarantees to the programs. The current memory model document does not
> guarantee type safety if I am right. Note that probably it does not
> matter because the implementation will fix those issues (hopefully)
> but writing 5 lines to the definition would not hurt in my opinion.

I understand your point about type safety now, thanks for explaining
it.

I don't think the memory model should be changed for this, though.
All we really want to guarantee for programs is that something
reasonable will happen--specifically that they won't get invalid
addresses, at least not in some operating modes. I don't see any need
to be specific about exactly what will happen. In particular I don't
think we want to guarantee that you can set a global variable in one
goroutine and read it in a different goroutine without a
synchronization event of some sort. A program which does that should
be prepared to crash.

Ian

[hong]

> I think it's unlikely that something like this will happen
> (all the discussions seem to end up concluding that it doesn't
> fit well into a language like Go), but you're right that it's a mistake
> to redefine the meaning of the * in *T.  Instead of &T for non-null,
> which has the problem Ian pointed out, you could use +T,
> since + already means "non-empty version of *" in other contexts.

Actually, many fancy features would work very well with Go if they
are designed with the rest of the language, and it will cost way
less to implement than many other languages. For example,
since types are independent, it is much easier to implement
generics and function overloading, then we can move language
concepts like []byte to library Array[byte], and magic operators
will become regular functions, i.e. a + b is the same as a.+(b).
Go would be more complicated that way. But the result will be
much better than piling features after-thought, like current C++,
Java, C#.

Today, it will be much harder to introduce notnull. For example,
new()/make()/init() will not work with notnull types. We end up
have to use nullable pointers inside structs. Or we need much
further changes to language and library. Probably not worth it
for the current target.

Hong

[NoiseEHC]

It took some time until I finished this message but here it comes:

Let me try again.  You have to spell out in precise detail when the
language permits an assignment from ?T to *T.  For example, is this
OK?
	if p.next != nil {
		var x *T = p.next;
	}
  
      

From the proposal:
“p = q” only allowed if it can be inferred from an “if q != nil” or
from a “q = qq” where qq *T that q is not null.

So yes, it is OK.
    

Aside from the thread safety issue, here we are talking about the
language, not any particular implementation.  We have to be able to
write a specification such that all implementations of the language do
the same thing.  "It can be inferred" is not a specification.  How is
this inference done?  When does it work and when does it fail?

  

Now as I have read the Eiffel notnullness specification, I can see that my proposal is 90% of the Eiffel specification if we fix the two problems Nigel Tao has shown me. Is there a remote chance that you will ever include notnullness to the language or you just let people wasting their time writing endless proposals because you do not ever consider them? I ask this because for me it takes ~10x as much to write something in English than it takes for you. (BTW there was no thread safety issue in my proposal.)

I understand that you are not proposing anything about out-of-bounds
array access, but I am curious as to your position on how those should
be handled.  Am I wrong in seeing the cases as related?

  
      

They are absolutely similar. And as I see you have already invented
slices!!! You just do not like notnull pointers...
    

Slices are essentially a safe way to do pointer arithmetic.  I don't
really see them as related to out-of-bounds array access.  You can
have out-of-bounds access to a slice, too.

  

I just showed that you have invented some language construct (slice) which makes the language safer. You are right that there is a difference between them that notnull pointers completely solve a problem (null dereferencing) while slices can only solve it partially (out of bound) but at least you tried to make it safer. Why not do this with notnull pointers too?

The main difference is that statical array bound checking requires
first order logic while pointers do not. In the general sense neither
problem can be solved generally (== halting problem) but my instinct
tells me (and we know from history) that at least notnull pointers can
be done practically. (This is equivalent with the attackable sentence
in the prior message.) I do not think that I could propose a similar
proposal for array indices so I just do not do it. Slices should make
array indices much more safe and that is all we can do about that
(that is my belief).
    

We know that there are other languages which supports non-null
pointers.  What do those languages do about out-of-bounds array
accesses?

  

Now here is the part why I am sending this message.

In Spec# array access requires that the compiler can deduce from some invariant or precondition that the index is right. Normally the compiler is smart enough that it can infer from simple (or moderately complex) loops the loop invariant which is usually enough. In the cases when it cannot infer the index's range (or it is simply impossible to do so) then the programmer has to specify a precondition for the function and delegate the index checking to the callers.

So it is trivial to see that in this case:
void SetArray(int[] Array, int Index, int Value) {
��� Array[Index] = Value;
}
there is no way to infer whether Index is right so you can compile it in Spec# only if you specify a precondition of: Index >= 0 && Index < Array.Length.

Now my point is that those things only work if you allow specifying pre and post conditions in the language (first order logic required) and Go is clearly not developed into this direction so there will be no feature which would fix out-of-bound conditions that is sure. However notnullness can be done without pre and post conditions so these are clearly different problem spaces to me.

[Ian Lance Taylor]

NoiseEHC <Nois...@freemail.hu> writes:

>> Aside from the thread safety issue, here we are talking about the
>> language, not any particular implementation. We have to be able to
>> write a specification such that all implementations of the language do
>> the same thing. "It can be inferred" is not a specification. How is
>> this inference done? When does it work and when does it fail?
>>
>>
>
> Now as I have read the Eiffel notnullness specification, I can see
> that my proposal is 90% of the Eiffel specification if we fix the two
> problems Nigel Tao has shown me. Is there a remote chance that you
> will ever include notnullness to the language or you just let people
> wasting their time writing endless proposals because you do not ever
> consider them? I ask this because for me it takes ~10x as much to
> write something in English than it takes for you.

I think it is very unlikely that we will add notnullness to the
language. But if somebody produced a clear proposal which addressed
all issues and did not make the language more complex then we would
consider it seriously. That is, we are not opposed to the idea in
theory; we are opposed to the practical consequences of the idea.

I apologize if I seem to be leading you on. I was responding to your
proposal by asking questions to make it clear that (I think) there are
critical areas which your proposal did not address.

> (BTW there was no
> thread safety issue in my proposal.)

Yes, there is a thread safety issue. You wrote that in code like
this:

if p.next != nil {
var x *T = p.next
}

that x could be presumed to be non-nil. But that is only true if
p.next did not change between the test and the assignment. Since the
goal of your proposal, as I understand it, is to ensure that nil
pointers can not occur when a value is presumed to be non-nil, you
have to consider that one way or another. You can explicitly say that
you don't care about this kind of race condition--that is, you can say
that it is OK if a program with race conditions crashes due to a nil
pointer dereference--but I don't think you can simply ignore it.

Thanks for the information about Spec#. I think the distinction
between *T and ?T could be seen as another way of writing a
precondition.

Ian

[NoiseEHC]

> I think it is very unlikely that we will add notnullness to the
> language. But if somebody produced a clear proposal which addressed
> all issues and did not make the language more complex then we would
> consider it seriously. That is, we are not opposed to the idea in
> theory; we are opposed to the practical consequences of the idea.
>
>

This "90% of Eiffel" has meant that there is no way (I could think of)
that notnullness could be implemented with less new language features.
Eiffel authors did not do that and I do not even consider Spec# which is
more complex than Eiffel (in my opinion). However it does not mean too
much new language elements:
1. *p and ?p (and one of them already exists)
2. shadow type (almost the same as checking uninitialized variables)
3. conversion rules
4. "exceptions" as primitive post conditions
Now if "not more complex" means only accepting the first one then I
agree, notnullness will not happen in Go anytime.

> I apologize if I seem to be leading you on. I was responding to your
> proposal by asking questions to make it clear that (I think) there are
> critical areas which your proposal did not address.
>
>

Ehem, I have no idea what this "I seem to be leading you on" expression
means in English but apology accepted blindly... :)
Seriously: I really have no idea what part from my message does it
answer. What is sure that writing something complex in English takes 10x
as much time for me than for you so I would rather save my time if you
could just tell me not to bother to do something. The problem is when
you (here I mean you == Go authors) completely avoid answering some idea
on this list so at least a one liner such as: "over my dead body" (or:
"it is really a bad idea") would allow me not wasting my time (and it
seems to me that it is not only my opinion).

> Yes, there is a thread safety issue. You wrote that in code like
> this:
>
> if p.next != nil {
> var x *T = p.next
> }
>

No, it should use the second time the cached p.next which != nil if the
compiler would follow my proposal. It is not a race condition just
confusing code. But not a race condition.... :) So I do not say that
this should not be fixed in the proposal just not a race condition.

[Ian Lance Taylor]

NoiseEHC <Nois...@freemail.hu> writes:

>> I think it is very unlikely that we will add notnullness to the
>> language. But if somebody produced a clear proposal which addressed
>> all issues and did not make the language more complex then we would
>> consider it seriously. That is, we are not opposed to the idea in
>> theory; we are opposed to the practical consequences of the idea.
>>
>>
>
> This "90% of Eiffel" has meant that there is no way (I could think of)
> that notnullness could be implemented with less new language
> features. Eiffel authors did not do that and I do not even consider
> Spec# which is more complex than Eiffel (in my opinion). However it
> does not mean too much new language elements:
> 1. *p and ?p (and one of them already exists)
> 2. shadow type (almost the same as checking uninitialized variables)
> 3. conversion rules
> 4. "exceptions" as primitive post conditions
> Now if "not more complex" means only accepting the first one then I
> agree, notnullness will not happen in Go anytime.

That approach definitely makes the language more complex. I do not
think that we would implement any proposal along those lines.

>> Yes, there is a thread safety issue. You wrote that in code like
>> this:
>>
>> if p.next != nil {
>> var x *T = p.next
>> }
>>
> No, it should use the second time the cached p.next which != nil if
> the compiler would follow my proposal. It is not a race condition just
> confusing code. But not a race condition.... :) So I do not say that
> this should not be fixed in the proposal just not a race condition.

This is a good example of additional complexity in the language. It
means, for example, that the language can no longer be implemented by
a naive interpreter, but that the interpreter must be aware that
p.next must be cached here. It can be done--nobody disputes that--but
it increases complexity.

Ian

[Joseph]

Hi Ken et al,

On Dec 7 2009, 11:02 pm, Ken Thompson <k...@google.com> wrote:
> i agree with ian, there is an isomorphism between
> dynamic (vsstatic) catching of an array/slice index
> out of bounds and a reference thru a nil pointer.
> this discussion concentrates on pointers and leaves
> arrays alone. they are exactly the same.
>
> maybe this can be handled outside of the compiler.
> how about someone converting ESC (extendedstaticcheck)
> as a (non-compiler) tool for go. this sounds like value-added
> for both camps. my fear then is that code will become mucked up
> with lots of stupid annotations. my case in point is
>       (void)f();
> as a totally stupid side-effect of running lint.

Would the Go creators be interested in an ESC/Go?

Best,
Joe Kiniry (one of "the ESC guys" who has been keeping an eye on Go)
http://kindsoftware.com/

[Ian Lance Taylor]

Joseph <kin...@gmail.com> writes:

> Hi Ken et al,
>
> On Dec 7 2009, 11:02 pm, Ken Thompson <k...@google.com> wrote:
>> i agree with ian, there is an isomorphism between
>> dynamic (vsstatic) catching of an array/slice index
>> out of bounds and a reference thru a nil pointer.
>> this discussion concentrates on pointers and leaves
>> arrays alone. they are exactly the same.
>>
>> maybe this can be handled outside of the compiler.
>> how about someone converting ESC (extendedstaticcheck)
>> as a (non-compiler) tool for go. this sounds like value-added
>> for both camps. my fear then is that code will become mucked up
>> with lots of stupid annotations. my case in point is
>>       (void)f();
>> as a totally stupid side-effect of running lint.
>
> Would the Go creators be interested in an ESC/Go?

I certainly expect that there would be some interesting cases where
static checking could be applied to Go outside of the compiler.

Ian

[Ealdwulf Wuffinga]

Ian Lance Taylor wrote:

>
> Yes, there is a thread safety issue. You wrote that in code like
> this:
>
> if p.next != nil {
> var x *T = p.next
> }
>
> that x could be presumed to be non-nil. But that is only true if
> p.next did not change between the test and the assignment. Since the
> goal of your proposal, as I understand it, is to ensure that nil
> pointers can not occur when a value is presumed to be non-nil, you
> have to consider that one way or another. You can explicitly say that
> you don't care about this kind of race condition--that is, you can say
> that it is OK if a program with race conditions crashes due to a nil
> pointer dereference--but I don't think you can simply ignore it.
>

Is the type switch statement in go thread safe? Eg, if we have:
switch i := p.next.(type) {
case nil:
...
case *int:
...
}

Then if the value of p.next is changed by another thread, is the type of
i in the branches of the switch guaranteed to be correct?

It seems to me that the simplest way of introducing non-null pointers
into go would be if variant types were first introduced. Then non-null
pointers could work in exactly the way that they do in ML-style languages:

Suppose that we can declare variant types as follows:

type Foo union { X, Y, Z }

such that Foo is a type like interface{} except that it only accepts
values of the type X, Y or Z. (This is not very much like a C union, so
it might be desirable to pick another name).

Then, ?T would be an alias for union {nil, *T}. It would only be
possible to examine the value of an ?T by using a type switch (or a
function which itself used a type switch). Therefore, access to ?T
would be thread safe if general type switches are thread safe.
Uninitialized *T would be banned, unless it was part of a union also
containing nil.

Potential objections to this that I can see are:

- it might be considered a bit cumbersome to only be able to access ?T
via a switch. ML-like languages tend to add syntactic sugar for dealing
with variant types.
- It may be unobvious when a type contains an *T and therefore can't be
declared uninitialized. This can be mitigated by good compiler error
messages.

Ealdwulf

[Ian Lance Taylor]

Ealdwulf Wuffinga <eald...@googlemail.com> writes:

> Is the type switch statement in go thread safe? Eg, if we have:
> switch i := p.next.(type) {
> case nil:
> ...
> case *int:
> ...
> }
>
> Then if the value of p.next is changed by another thread, is the type
> of i in the branches of the switch guaranteed to be correct?

Yes. The expression is evaluated first and (in effect) stored in a
temporary variable. Then the type switch is done. In each case with
a type, in effect a new local variable is created with the same name,
and the temporary variable is assigned to the new local variable.

Ian

[NoiseEHC]

>
> It seems to me that the simplest way of introducing non-null pointers
> into go would be if variant types were first introduced. Then non-null
> pointers could work in exactly the way that they do in ML-style
> languages:
>

It is solved in Eiffel (and my never will be finished proposal) by
introducing a new name for the shared field/variable's (locals are
thread safe always) current notnull value. The following idiom could do
this without syntactic sugar.

if p := p.next; p != nil {
// here p != nil, strange isn't it... :)
}

However the trick is not how to define nullable and notnull pointer
types but how to make the system useful. Unfortunately Eiffel's (and so
mine) proposal is the minimal plumbing needed what seems too much for Go...

[Steven Blenkinsop]

Yeah, basically, what you just said why I never really got the deal
about notnull pointers. When it comes down to it, notnull is just a
domain issue, like dividing by zero. Currently, only runtime checking
exists for domains. Of course, people seem to find it tedious to have
to do that all the time (one of the main (bad) reasons many people
want exceptions). You could do this:

package ...

type notNil struct {
*int
}

func NotNil(ptr *int) (notNil) {
if ptr == nil { panic("pointer is nil, not nil pointer needed") }

return notNil{prt}
}

This way, as long as you always recieve notNil's and never assign to
notNil.int yourself, you'll know that neither you nor anyone using
your package is passing you a nil pointer. Though it seems a bit
overkill to me just to avoid some if statements. But thats just an
idle side thought...

Also a side note, I find the Some x | None approach to "pointers" in
ocaml neat in that nil (None) isn't a value, its an alternative to
having a value, so if you don't offer the alternative, you don't have
nil. I know they aren't really pointers, and they wouldn't fit into
Go, but they're neat nonetheless.

[Johann Höchtl]

On Dec 3 2009, 1:24 am, NoiseEHC <Noise...@freemail.hu> wrote:
> Hi Russ (and members of this mailing list of course)!
>
> Here is the missing ring.go file I have promised to you. :)
>
> In fact this is a rewrite of a formal proposal which I was starting to
> write in 2009.11.16... It was originally named as “proposal for
> exception handling and notnull pointers” but in the meantime it seemed
> better finishing the notnullness of pointers so here it is.
>

I would really love to see discussion on Exceptions going on, as I
personally consider it the most laking feature, preventing the
adoption of the go language in the large scale (IMHO!). Lua seems to
do well without them and I hacked Lua some time yet never liked it
and there are hacks using pcall / error to circumvent the restriction.

Having said that, has the core team already settled on a descission
wheather exceptions (or function-wise sort of, probly named
differently?) are planed for sometime to become part of the language
besides "exceptions remain an open issue. "?

[NoiseEHC]

> I would really love to see discussion on Exceptions going on, as I
> personally consider it the most laking feature, preventing the
> adoption of the go language in the large scale (IMHO!). Lua seems to
> do well without them and I hacked Lua some time yet never liked it
> and there are hacks using pcall / error to circumvent the restriction.
>
> Having said that, has the core team already settled on a descission
> wheather exceptions (or function-wise sort of, probly named
> differently?) are planed for sometime to become part of the language
> besides "exceptions remain an open issue. "?
>

Actually the last exception proposal was made by me as "[go-nuts] Yet
Another Proposal For Exceptions (and it is almost finished now...)".
Unfortunately it did not get any feedback from Go authors so I cannot
comment on whatever they are settled on. BTW now that we know that there
will be no notnull pointers in Go, my (already simple) proposal could be
simplified even more by permitting only one error condition on a way
like Esko Luontola proposed. (My feeling that it will not be accepted
even though it would just solve the exception problem perfectly.)

[Ian Lance Taylor]

Johann Höchtl <johann....@gmail.com> writes:

> Having said that, has the core team already settled on a descission
> wheather exceptions (or function-wise sort of, probly named
> differently?) are planed for sometime to become part of the language
> besides "exceptions remain an open issue. "?

No, we haven't. Exceptions, um, remain an open issue.

Ian

[Ryanne Dolan]

I wouldn't call pcall/error a hack... they were designed to be used that way.  It is rather easy to implement pcall/error in an interpreted language, so Lua's designers got a lot of functionality without adding much to the runtime.

The hack-ish-ness comes from the overuse of pcall/error; exceptions have a similar problem.  No one calls exceptions a hack tho...

I proposed earlier that Go should adopt pcall, but I'd recommend leaving out 'error' for the same reasons I'd say leave out exceptions.  I think it is perfectly sufficient to have protected calls without user-defined errors/exceptions.  So long as you can trap built-in runtime exceptions (array out of bounds, divide by zero, etc), I'd be happy.

Thanks.
Ryanne

--
www.ryannedolan.info