Repeating the billion dollar mistake?

[Sebastian Sylvan]

On slide 14 of the Go Talk the first design principle is "Keep
concepts orthogonal", and yet Go violates this principle by repeating
Hoare's "Billion Dollar Mistake" (
http://www.infoq.com/presentations/Null-References-The-Billion-Dollar-Mistake-Tony-Hoare
) by combining the two orthogonal concepts of "reference to" and
"nullable" into a single construct.

I know many language designers have said in retrospect that they
regret getting it wrong (e.g. several from the C# team), so if you're
going to make a new language why not do it right this time and let
these two features be two separate features like they're supposed to
be?

[Ian Lance Taylor]

Go doesn't have references.

Pointers are not references, as can be seen by the fact that special
syntax is required to access the value that a pointer points to.

Ian

[Sebastian Sylvan]

That's an entirely irrelevant distinction. The word "reference" is used in my post (and in Hoare's talk) to refer to all kinds of reference constructs, pointers included.

The point is that there's no reason to take two orthogonal concepts ("reference/pointer to" and "nullable") and slap together in one construct (producing one of the most common sources of bugs in the process - a billion dollar's worth if you believe Hoare) when we know perfectly well how to avoid it (see e.g. Eiffel's Void safety).

There should be two orthogonal concepts. One which is "reference to" (a pointer), and one of which is "may be null". You can have a reference that's not nullable (i.e. it's statically known to never ever be null, which means you don't have to check it), and you can have regular values that are nullable, and indeed you can have nullable references (which is the only thing Go has today).

--
Sebastian Sylvan

[Peter Froehlich]

Hi all,

This is terminology confusion at work (now a Go person commits the sin
of thinking in C++ or Java I guess?). Reference = Pointer in the Hoare
talk, there's no difference. I like that you call C's NULL "nil" in
Go, another piece of Oberon at work (it's NIL in Oberon). :-D

Cheers,
Peter
--
Peter H. Froehlich <http://www.cs.jhu.edu/~phf/>
Senior Lecturer | Director, Johns Hopkins Gaming Lab

[levi]

On Nov 13, 5:33 pm, Ian Lance Taylor <i...@google.com> wrote:

> Sebastian Sylvan <sebastian.syl...@gmail.com> writes:
> > On slide 14 of the Go Talk the first design principle is "Keep
> > concepts orthogonal", and yet Go violates this principle by repeating
> > Hoare's "Billion Dollar Mistake" (

> >http://www.infoq.com/presentations/Null-References-The-Billion-Dollar...

> > ) by combining the two orthogonal concepts of "reference to" and
> > "nullable" into a single construct.
>
> > I know many language designers have said in retrospect that they
> > regret getting it wrong (e.g. several from the C# team), so if you're
> > going to make a new language why not do it right this time and let
> > these two features be two separate features like they're supposed to
> > be?
>
> Go doesn't have references.
>
> Pointers are not references, as can be seen by the fact that special
> syntax is required to access the value that a pointer points to.

Well, its not only pointers. Function values can be nil and map values
can be nil as well. Or channel values. And so on. The OP is right,
Hoare's mistake is indeed repeated.

-Levi

[Ian Lance Taylor]

Sorry for my terminology confusion.

Go doesn't have nullable types in general. We haven't seen a real
desire for them.

I personally think that in a systems programming language, a pointer
is a reasonable concept: it intuitively represents a memory address.
I don't personally think that permitting pointers to be nil is a
billion dollar mistake. In my C/C++ programming I've never noticed
that NULL pointers are a noticeable source of bugs.

Ian

[Sebastian Sylvan]

A pointer is a reasonable concept, but there's no reason combine the concept of a pointer with the concept of "nullable". Representing a memory address is perfectly fine and useful, but from there it doesn't follow that we should also allow it to point to an invalid memory address that crashes if you try to use it. You've already decided to opt for safety with regards to array slices, why stray from both that, and the principle of orthogonality, in this case? It doesn't remove any flexibility, it just pulls out unrelated concepts into two orthogonal features rather than one mashed together feature.

I'd say that in my experience probably the majority, or at least a very large minority, of asserts/crashes when programming C/C++ are null pointer asserts.

--
Sebastian Sylvan

[Valentin Nechayev]

On 13 ноя, 21:29, levi <greenspan.l...@googlemail.com> wrote:

> > Pointers are not references, as can be seen by the fact that special
> > syntax is required to access the value that a pointer points to.
>
> Well, its not only pointers. Function values can be nil and map values
> can be nil as well. Or channel values. And so on. The OP is right,
> Hoare's mistake is indeed repeated.

The problem is that any real programming can't avoid this "mistake":
all alternatives are more opaque and horrible. If to insist on having
all pointers of real values, one can't correctly show some of them as
not having real value. If to use separate validness flag, one would
forget to check it. If to point to fictive data area, it would become
global area to exchange with garbage data. And so on. Do we all
respect Hoare, but this itself isn't bug, but only way to trigger
another bugs.

[Sebastian Sylvan]

I'm not sure what you mean. This is a solved problem in many existing "real" languages (e.g. Eiffel, Haskell, etc.). Have you looked at e.g. Eiffel's Void safety? In what way does that not make sense to you?

You simply have *two* "decorators" for types. One for "pointer to" which can never be null (except in well defined scenarios, such as constructors, which end with a dynamic test to ensure the "not null" assumption is valid in the rest of the program). The other decorator is for "nullable" which makes any value (including pointers) potentially null. 

There obviously needs to be a way to convert a nullable value to a non-nullable one. There are various forms of this that you can look up (e.g. Eiffel merely detects that you've checked that the pointer isn't null in an if statement, and inside the if clause the type of the pointer is changed to a non-nullable one).

You don't lose any expressiveness. You can still have a "nullable pointer" if that's what you need to indicate that you don't have a "real" value. So in C syntax it might look something like:

int* x = &p; // pointer - always valid to derefence after initialization is complete

int? y = NULL; // nullable int, either an int or NULL

int*? z = NULL; // nullable pointer, identical to * in standard C - either a pointer or NULL

The benefit is that since for probably 90% of pointers you do *not* want them to be null, most of the code would be using non-nullable pointers, which means that you can omit all pointer checks from your code (and any that the runtime performs). Most functions arguments have non-nullness as precondition, for example. This means that non-nullable pointers would spread and the nullness would be confined to very few areas, and these nullable pointers would be checked and converted to a non-null pointers (or handled if it's null) as soon as possible.

--
Sebastian Sylvan

[Peter Froehlich]

Hi all,

On Fri, Nov 13, 2009 at 2:48 PM, Ian Lance Taylor <ia...@google.com> wrote:
> Go doesn't have nullable types in general.  We haven't seen a real
> desire for them.

Huh? nil is a valid pointer value, right? So they are "nullable".

> I personally think that in a systems programming language, a pointer
> is a reasonable concept: it intuitively represents a memory address.

Personally I think that much of the trouble with pointers stems from
the idea that "pointers are memory addresses". Abstractly pointers are
values that denote variables, "addresses" are simply a way to
implement pointers; you could implement them as URLs as well.

Since you don't have pointer arithmetic in Go (right?) there's really
no reason to ever think of them as specifically memory addresses. In
any case, there's really very little you can do aside from adding a
"nonnull" qualifier to the language or adding different pointer types
with different constraints. Without more mechanisms, the problem is
not going away.

Note that personally, I don't think there's a major problem with
pointers the way they are. Sure, it would be nice to have better ways
to express their varied uses (indeed Wirth found a cute way to do
type-safe and memory-safe address arithmetic in Oberon-SA) but if in
doubt and you want to have a minimal language, there's a lot of other
stuff you could worry about.

> I don't personally think that permitting pointers to be nil is a
> billion dollar mistake.  In my C/C++ programming I've never noticed
> that NULL pointers are a noticeable source of bugs.

No, the biggest problem are memory management issues and array bounds.
Of course to some extent both of these are related to pointers, but
you can also solve them and still have pointers (without address
arithmetic anyway).

[Valentin Nechayev]

On Fri, Nov 13, 2009 at 10:33 PM, Sebastian Sylvan <sebastia...@gmail.com> wrote:

I'm not sure what you mean. This is a solved problem in many existing "real" languages (e.g. Eiffel, Haskell, etc.). Have you looked at e.g. Eiffel's Void safety? In what way does that not make sense to you?

You simply have *two* "decorators" for types. One for "pointer to" which can never be null (except in well defined scenarios, such as constructors, which end with a dynamic test to ensure the "not null" assumption is valid in the rest of the program). The other decorator is for "nullable" which makes any value (including pointers) potentially null. 

Well, you're talking about some different thing. The approach you showed is really language support of runtime checking of pointer to be non-null, and if you prohibit direct dereferencing of "nullable" type, this would be good protection (gathered in one conversion instead of each dereferencing). But I've told about general approach to show "placeholder" instead of real value: null pointer is the easiest method to show absense of value, all another methods are much more cumbersome and error-prone.

--
-netch-

[Peter Froehlich]

Tracking whether a pointer is or isn't NULL is a good idea, I agree.
But once you've reached the "not NULL" state, you may have to do
*other* checks at runtime. Take a look at this:

if (p != NULL) {
// p promoted to nonnull type qualifier
p = q;
}

Inside the then part you claim p will be treated as "nonnull" and
therefore no more null pointer checks are needed. Good. But you have
two options when it comes to the assignment: (a) allow it regardless
of whether q is nonnull, then you'll need a dynamic check or (b) only
allow it if q is nonnull already. In the first case you end up with
dynamic checks, in the second you end up with the "const pollution"
problem that Go otherwise tries to avoid (if I understand it
correctly).

[Sebastian Sylvan]

On Fri, Nov 13, 2009 at 8:43 PM, Valentin Nechayev <net...@gmail.com> wrote:

On Fri, Nov 13, 2009 at 10:33 PM, Sebastian Sylvan <sebastia...@gmail.com> wrote:

I'm not sure what you mean. This is a solved problem in many existing "real" languages (e.g. Eiffel, Haskell, etc.). Have you looked at e.g. Eiffel's Void safety? In what way does that not make sense to you?

You simply have *two* "decorators" for types. One for "pointer to" which can never be null (except in well defined scenarios, such as constructors, which end with a dynamic test to ensure the "not null" assumption is valid in the rest of the program). The other decorator is for "nullable" which makes any value (including pointers) potentially null. 

Well, you're talking about some different thing. The approach you showed is really language support of runtime checking of pointer to be non-null,

Well, I would phrase it as: language support for static guarantees that a pointer is always non-null at the dereference site, with language-enforced dynamic checks in the few places where you do have a nullable pointer (which can't be dereferenced) and you want to convert it to a regular pointer.

 

and if you prohibit direct dereferencing of "nullable" type, this would be good protection (gathered in one conversion instead of each dereferencing). But I've told about general approach to show "placeholder" instead of real value: null pointer is the easiest method to show absense of value, all another methods are much more cumbersome and error-prone.

Yes, you'd still have null, it just wouldn't be conflated with the concept of pointers - it would work for *any* type, not just pointers. 

These are really two totally different things ("nullable" vs "pointer"). It's an accident of history that they've been combined into one construct in so many languages, and there now seems to be a pretty broad consensus among language designers that it's a mistake that we unfortunately have to live with for many existing languages and VMs because it's so ingrained. Go is a new language, though, so it seems a shame for yet another group of language designers to make the same mistake again (especially when they have a stated goal to avoid just this kind of thing!), when so many others have already lived to regret it.

--
Sebastian Sylvan

[Sebastian Sylvan]

I definitely don't think you should allow any implicit conversions between nullable and non-nullable values! Your example would have to be something like:

if (p != NULL) {
 // p promoted to nonnull type qualifier

 if ( q != NULL ) { // if q is declared outside the outer if, you could just check it there

     p = q;

 }
}

You definitely *want* pointers to propgate non-nullness. You can always convert a regular pointer to a nullable one easily, and converting a nullable one to a regular pointer is just a simple check. What happens in practice though is that 90%+ of your pointers are non-nullable so you rarely have to check at all. You have a few places where you get a nullable one back to signify something special, and what you do is you almost immediately convert it to a non-nullable one (in order to actually do anything useful with it, like pass it to a function that usually takes non-nullable pointers) and from there on in you don't have to do any more checks on it.

--
Sebastian Sylvan

[Ian Lance Taylor]

Sebastian Sylvan <sebastia...@gmail.com> writes:

> You simply have *two* "decorators" for types. One for "pointer to" which can
> never be null (except in well defined scenarios, such as constructors, which
> end with a dynamic test to ensure the "not null" assumption is valid in the
> rest of the program). The other decorator is for "nullable" which makes any
> value (including pointers) potentially null.

This leads us in the direction of the const type qualifier. In my
personal opinion, this kind of thing should not be part of the type.
I think this amounts to a language design choice. I think that
calling it a billion dollar mistake amounts to hyperbole.

Ian

[Valentin Nechayev]

On Fri, Nov 13, 2009 at 10:53 PM, Sebastian Sylvan <sebastia...@gmail.com> wrote:

Yes, you'd still have null, it just wouldn't be conflated with the concept of pointers - it would work for *any* type, not just pointers. 

These are really two totally different things ("nullable" vs "pointer"). It's an accident of history that they've been combined into one construct in so many languages, and there now seems to be a pretty broad consensus among language designers that it's a mistake that we unfortunately have to live with for many existing languages and VMs because it's so ingrained.

Saying again, you insist that combining these two concepts was incorrect. I try to show it was correct because is the simplest and supported way to show pointer without value (comparing it to literal 0 is just psychologic artifact). This doesn't avoid us from using another kind of pointers - non-nullable - as you showed here.

But, as soon as it's incorrect to do _checking_ without assigning of null pointer as you show:

var pp *foo nullable;
var p *foo;
if pp != nil { p = pp; }

(it's quite easy to write code without such check)

you shall combine checking and assignment in one action:

try {
  p = pp;
  do_something(p);
}

This means _generating exception_ because you can't move further. And, Go doesn't have exceptions yet.
 

Go is a new language, though, so it seems a shame for yet another group of language designers to make the same mistake again (especially when they have a stated goal to avoid just this kind of thing!), when so many others have already lived to regret it.

Well, it's rather good idea to invent "non-nullable" pointers as language construct... but let's be correct and avoid to blame for very early release.

--
-netch-

[Sebastian Sylvan]

I'm sorry but this makes no sense to me. In what way does this have a relation to the const qualifier? You can convert between them at any time, you just have to make sure you handle the failure case in the unsafe direction.

It's not about adding information to types, it's about having a less error prone view of what a pointer *is*. Your argument seems to me to be just as applicable to the distinction between ints and bools as well - why don't we just stick those under the same type? Answer: Because they're fundamentally different! Just like the concept of pointers is different to the concept of nullable values.

It's his mistake, he can call it what he wants. I don't think he's wrong in that a "feature" that introduces potential runtime crashes all over the place has been an incredibly expensive mistake.

--
Sebastian Sylvan

[Sebastian Sylvan]

On Fri, Nov 13, 2009 at 9:12 PM, Valentin Nechayev <net...@gmail.com> wrote:

On Fri, Nov 13, 2009 at 10:53 PM, Sebastian Sylvan <sebastia...@gmail.com> wrote:

Yes, you'd still have null, it just wouldn't be conflated with the concept of pointers - it would work for *any* type, not just pointers. 

These are really two totally different things ("nullable" vs "pointer"). It's an accident of history that they've been combined into one construct in so many languages, and there now seems to be a pretty broad consensus among language designers that it's a mistake that we unfortunately have to live with for many existing languages and VMs because it's so ingrained.

Saying again, you insist that combining these two concepts was incorrect. I try to show it was correct because is the simplest and supported way to show pointer without value (comparing it to literal 0 is just psychologic artifact). This doesn't avoid us from using another kind of pointers - non-nullable - as you showed here.

But, as soon as it's incorrect to do _checking_ without assigning of null pointer as you show:

var pp *foo nullable;
var p *foo;
if pp != nil { p = pp; }

This would be invalid, p would have to be initialized at the declaration site. Something like:

if pp != nil { 

   var p *foo;

   p = pp; 

   // rest of code using p in this clause

}

 

(it's quite easy to write code without such check)

you shall combine checking and assignment in one action:

try {
  p = pp;
  do_something(p);
}

I'm sorry, I just don't understand what you mean here. Could you try to explain it differently?  

--
Sebastian Sylvan

[Sebastian Sylvan]

On Fri, Nov 13, 2009 at 9:27 PM, Sebastian Sylvan <sebastia...@gmail.com> wrote:

On Fri, Nov 13, 2009 at 9:12 PM, Valentin Nechayev <net...@gmail.com> wrote:

On Fri, Nov 13, 2009 at 10:53 PM, Sebastian Sylvan <sebastia...@gmail.com> wrote:

Yes, you'd still have null, it just wouldn't be conflated with the concept of pointers - it would work for *any* type, not just pointers. 

These are really two totally different things ("nullable" vs "pointer"). It's an accident of history that they've been combined into one construct in so many languages, and there now seems to be a pretty broad consensus among language designers that it's a mistake that we unfortunately have to live with for many existing languages and VMs because it's so ingrained.

Saying again, you insist that combining these two concepts was incorrect. I try to show it was correct because is the simplest and supported way to show pointer without value (comparing it to literal 0 is just psychologic artifact). This doesn't avoid us from using another kind of pointers - non-nullable - as you showed here.

But, as soon as it's incorrect to do _checking_ without assigning of null pointer as you show:

var pp *foo nullable;
var p *foo;
if pp != nil { p = pp; }

This would be invalid, p would have to be initialized at the declaration site. Something like:

if pp != nil { 

   var p *foo;

   p = pp; 

   // rest of code using p in this clause

}

Ugh, this, obviously:

if pp != nil { 

   var p *foo = pp; 

   // rest of code using p in this clause

}

 

--
Sebastian Sylvan

[Valentin Nechayev]

On Fri, Nov 13, 2009 at 11:27 PM, Sebastian Sylvan <sebastia...@gmail.com> wrote:

This would be invalid, p would have to be initialized at the declaration site. Something like:

if pp != nil { 

   var p *foo;

   p = pp; 

   // rest of code using p in this clause

}

Well, let's the programmer make mistake:

if pq != nil { // pq is some another variable

  var p *foo;
  p = pp;
  // rest of code using p in this clause
}

You simply missed the real check and allowed `p' to be null. Congrats.

I'm sorry, I just don't understand what you mean here. Could you try to explain it differently?  

You shall NOT allow to continue execution if  conversion fails.

--
-netch-

[Sebastian Sylvan]

On Fri, Nov 13, 2009 at 9:41 PM, Valentin Nechayev <net...@gmail.com> wrote:

On Fri, Nov 13, 2009 at 11:27 PM, Sebastian Sylvan <sebastia...@gmail.com> wrote:

This would be invalid, p would have to be initialized at the declaration site. Something like:

if pp != nil { 

   var p *foo;

   p = pp; 

   // rest of code using p in this clause

}

Well, let's the programmer make mistake:

if pq != nil { // pq is some another variable

  var p *foo;
  p = pp;
  // rest of code using p in this clause
}
 

You simply missed the real check and allowed `p' to be null. Congrats.

This would simply fail to compile with type errors on the assignment (p is non-nullable after all, and pp is nullable -- as different as strings and floats are).

 

I'm sorry, I just don't understand what you mean here. Could you try to explain it differently?  

You shall NOT allow to continue execution if  conversion fails. 

Execution should not even start if conversion could "fail" in an unsafe way! The compiler would stop you.

--
Sebastian Sylvan

[Valentin Nechayev]

On Fri, Nov 13, 2009 at 11:49 PM, Sebastian Sylvan <sebastia...@gmail.com> wrote:

if pp != nil { 

   var p *foo;

   p = pp; 

   // rest of code using p in this clause

}

Well, let's the programmer make mistake:

if pq != nil { // pq is some another variable

  var p *foo;
  p = pp;
  // rest of code using p in this clause
}
 

You simply missed the real check and allowed `p' to be null. Congrats.

This would simply fail to compile with type errors on the assignment (p is non-nullable after all, and pp is nullable

The same for your example, exactly. Compiler don't know whether pp isn't null. If you doubt, compare with the following:

var i int;
for i := 0; i < 10; i++ {
   if i == 5 {
      if pp == nil { return; }
  }
}
p = pp;

then replace 5 with 12.

And, for any your attempt to prove that compiler can detect fact of checking `pp' I will create counterexample :)

OTOH if you teach compiler only to compare pointers with conditions in if-block, this would be simply inapplicable in real life.
 

--
-netch-

[podperson]

> Personally I think that much of the trouble with pointers stems from
> the idea that "pointers are memory addresses". Abstractly pointers are
> values that denote variables, "addresses" are simply a way to
> implement pointers; you could implement them as URLs as well.

This is certainly my problem with pointers, and I wonder why you're so
allergic to the idea that -- by default -- when you allocate a pointer
you would also allocate a valid thing of appropriate size for it to
point to. If you explicitly want to deal with actual memory addresses
to blobs of stuff handed to you from a foreign API or whatever then
you can explicitly declare a potentially unsafe pointer, but the rest
of the time you're free from this rubbish. Just as GC frees you from
free, Sylvan is proposing to free you from checking for null pointers
_everywhere_. It's a big win and what's the price? The compiler shoves
some extra space into function headers and initializes pointer
variables to point to them -- chances are you were going to do the
same thing manually anyway.

If you're going to have pointers that could be urls -- as you suggest
-- then by all means do away with pointers entirely. But it seems to
me a URL is a very different beast from a "variable". I have natural
expectations that a url might not "be there", whereas I pretty much
expect my variables to be there.

[Sebastian Sylvan]

On Fri, Nov 13, 2009 at 9:57 PM, Valentin Nechayev <net...@gmail.com> wrote:

On Fri, Nov 13, 2009 at 11:49 PM, Sebastian Sylvan <sebastia...@gmail.com> wrote:

if pp != nil { 

   var p *foo;

   p = pp; 

   // rest of code using p in this clause

}

Well, let's the programmer make mistake:

if pq != nil { // pq is some another variable

  var p *foo;
  p = pp;
  // rest of code using p in this clause
}
 

You simply missed the real check and allowed `p' to be null. Congrats.

This would simply fail to compile with type errors on the assignment (p is non-nullable after all, and pp is nullable

The same for your example, exactly. Compiler don't know whether pp isn't null. If you doubt, compare with the following:

I made a mistake in my example that I subsequently corrected. The compiler would've failed on that one too. Sorry. 

You should always have to initialize any non-null pointers to a valid value. So the compiler would always know that something is potentially null, or definitely not null by virtue of regular old type checking (no magic needed!).

It's just how you can't assign an int to a string, you have to convert it to a string first. This is exactly the same thing. They're different types, so the compiler would stop you doing any of that.

--
Sebastian Sylvan

[Valentin Nechayev]

On Sat, Nov 14, 2009 at 12:06 AM, Sebastian Sylvan <sebastia...@gmail.com> wrote:

I made a mistake in my example that I subsequently corrected. The compiler would've failed on that one too. Sorry. 

You should always have to initialize any non-null pointers to a valid value.

Well, this is more closer. But what shall the code do if initializing value is null and so initialization shall fail? This code shall not execute further...
 

So the compiler would always know that something is potentially null, or definitely not null by virtue of regular old type checking (no magic needed!).

It's just how you can't assign an int to a string, you have to convert it to a string first. This is exactly the same thing. They're different types, so the compiler would stop you doing any of that.

It can't  stop me at compile time because doesn't know real source pointer value.

--
-netch-

[Sebastian Sylvan]

On Fri, Nov 13, 2009 at 10:10 PM, Valentin Nechayev <net...@gmail.com> wrote:

On Sat, Nov 14, 2009 at 12:06 AM, Sebastian Sylvan <sebastia...@gmail.com> wrote:

I made a mistake in my example that I subsequently corrected. The compiler would've failed on that one too. Sorry. 

You should always have to initialize any non-null pointers to a valid value.

Well, this is more closer. But what shall the code do if initializing value is null and so initialization shall fail? This code shall not execute further...

It wouldn't let you initialize it to a null value because a null value would have a different type and would therefore give a type error at compile time. Again, there's no magic here, these are simply two variables with different types. You don't need any runtime checks to ensure that you haven't accidentally initialized a string with an int because the compiler prevents you from doing that. Same thing here.

 

So the compiler would always know that something is potentially null, or definitely not null by virtue of regular old type checking (no magic needed!).

It's just how you can't assign an int to a string, you have to convert it to a string first. This is exactly the same thing. They're different types, so the compiler would stop you doing any of that.

It can't  stop me at compile time because doesn't know real source pointer value.

It knows what type it is. That's all it needs. I think you may have missed what I wrote above about how Eiffel actually does the "null cast". That would explain the confusion. Look at this:

 

var p *int nullable = foo; //p can potentially be null

if p != nil{

  // inside this clause, the compiler knows p is not nil, so strips away the "nullness" from the type, giving it the type *int

  var q *int = p; // no type error!

}

var w *int = p; // TYPE ERROR. Here p is nullable.

Personally, I'm not a big fan of piggy-backing on the if statement to do this type promotion. It feels a bit too magic to me. I'd prefer a separate statement, maybe something like:

null_cast p {

  // we only end up here if p was non-null, and its type in this clause is non-nullable

}

else{ // optional

  // do something else? Here p is still nullable

}

--
Sebastian Sylvan

[Sverre Rabbelier]

Heya,

On Fri, Nov 13, 2009 at 23:10, Valentin Nechayev <net...@gmail.com> wrote:
> Well, this is more closer. But what shall the code do if initializing value
> is null and so initialization shall fail? This code shall not execute
> further...

You mean like this?

var p *int nullable = nil;
if p != nil {
var pp *int = p;
} else {
// can't do anything with p since it's nil

}

> It can't  stop me at compile time because doesn't know real source pointer
> value.

But it does, it either knows that it's a non-null pointer (since
that's it's type), or that it's a possibly null pointer, in which case
you can't make the assignment.

--
Cheers,

Sverre Rabbelier

[Yegor]

Seems like Go's switch statement might be powerful enough to implement
Scala's Some/None optional value idiom (http://www.artima.com/forums/
flat.jsp?forum=276&thread=228428). As opposed to null/not-null
reference/pointer, an Option either holds a value or it doesn't. The
only way to get the value is to explicitly cast the option to Some or
None case classes, so there's no way you can forget to handle the no-
value situation. While Go doesn't have case-classes, its switch
statement can be used for dynamic type discovery. So in Go it might
look something like this:

o := getOptionalValue() // never returns null
switch o.(type) {
case Some:
return Some(o).value;
case None: // but may return no value
return "No value";
}

... with corresponding performance penalties, of course.

Yegor

On Nov 13, 2:32 pm, Sebastian Sylvan <sebastian.syl...@gmail.com>
wrote:
> On Fri, Nov 13, 2009 at 9:27 PM, Sebastian Sylvan <

[Jon Harrop]

On Friday 13 November 2009 22:20:07 Yegor wrote:
> Seems like Go's switch statement might be powerful enough to implement
> Scala's Some/None optional value idiom (http://www.artima.com/forums/
> flat.jsp?forum=276&thread=228428). As opposed to null/not-null
> reference/pointer, an Option either holds a value or it doesn't. The
> only way to get the value is to explicitly cast the option to Some or
> None case classes, so there's no way you can forget to handle the no-
> value situation. While Go doesn't have case-classes, its switch
> statement can be used for dynamic type discovery. So in Go it might
> look something like this:
>
> o := getOptionalValue() // never returns null
> switch o.(type) {
> case Some:
> return Some(o).value;
> case None: // but may return no value
> return "No value";
> }
>
> ... with corresponding performance penalties, of course.

You really want an option and non-null pointers in a high-level language with
compilation down to nullable pointers when the option type is parameterized
over a reference type. F# does this, for example.

--
Dr Jon Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/?e

[Valentin Nechayev]

On Sat, Nov 14, 2009 at 12:17 AM, Sebastian Sylvan <sebastia...@gmail.com> wrote:

if p != nil{

  // inside this clause, the compiler knows p is not nil, so strips away the "nullness" from the type, giving it the type *int

  var q *int = p; // no type error!

} 

var w *int = p; // TYPE ERROR. Here p is nullable.

All this means too strict code style limitation which isn't applicable for real tasks.
 

Personally, I'm not a big fan of piggy-backing on the if statement to do this type promotion. It feels a bit too magic to me. I'd prefer a separate statement, maybe something like:

null_cast p {

  // we only end up here if p was non-null, and its type in this clause is non-nullable

}

The same story.
 

--
-netch-

[Sverre Rabbelier]

Heya,

On Fri, Nov 13, 2009 at 23:49, Valentin Nechayev <net...@gmail.com> wrote:
> All this means too strict code style limitation which isn't applicable for
> real tasks.

No, what Sebastian is saying is that since you don't _want_ a nullable
pointer 9 out of 10, it actually makes your code safer.

--
Cheers,

Sverre Rabbelier

[Sebastian Sylvan]

On Fri, Nov 13, 2009 at 10:49 PM, Valentin Nechayev <net...@gmail.com> wrote:

On Sat, Nov 14, 2009 at 12:17 AM, Sebastian Sylvan <sebastia...@gmail.com> wrote:

if p != nil{

  // inside this clause, the compiler knows p is not nil, so strips away the "nullness" from the type, giving it the type *int

  var q *int = p; // no type error!

} 

var w *int = p; // TYPE ERROR. Here p is nullable.

All this means too strict code style limitation which isn't applicable for real tasks.

I strongly disagree. 

There are plenty of languages out there today that already do this and it just isn't an issue.

Furthermore, these checks would only be in the cases where you're already checking for null anyway (or at least should be!), such as functions that may indicate failure by returning null. Any pointer where you today wouldn't check for null before using it must be a pointer you consider "guaranteed to be non-null", and in this system that pointer wouldn't have a nullable type to begin with, which means you don't need to check it (it's guaranteed to be safe).

Also, check out any Haskell or F# programs and see how often you see Maybe/Option (which is what their "nullable" is called) in them. They're there, but they're extremely rare. The vast majority of variables are not nullable.

--
Sebastian Sylvan

[roger peppe]

the difficulty with doing something like this in go
as it stands is that go relies on zero values
all over the place. zero values are cheap to create
and cheap on memory, as well as being a
convenient default initial value.

if you had non-nullable pointers, then creating the "zero value"
for a struct containing such a thing would also
entail allocating space for all its references,
only for the code to write the correct values over them again
and leave the original to be deallocated.

not great for a language that's trying to be efficient.

moreover it's not even clear that in the presence of
discriminated unions the compiler could know which
arm of the union to choose for the zero value.

in haskell you can't create a value out of nothing - hence
the existence of concepts like MonadZero.
i don't know what go would like with that kind
of restriction, but i suspect it would be a very different
language.

[levi]

On Nov 13, 9:10 pm, Valentin Nechayev <net...@gmail.com> wrote:
> On 13 ноя, 21:29, levi <greenspan.l...@googlemail.com> wrote:
>
> > > Pointers are not references, as can be seen by the fact that special
> > > syntax is required to access the value that a pointer points to.
>
> > Well, its not only pointers. Function values can be nil and map values
> > can be nil as well. Or channel values. And so on. The OP is right,
> > Hoare's mistake is indeed repeated.
>
> The problem is that any real programming can't avoid this "mistake":
> all alternatives are more opaque and horrible.

It is horrible to have maps, functions, channels which are nil. And it
is horrible when language designers permit things for themselves that
language users are not allowed to, like in these ad-hoc cases having
reference semantics for *certain* built-in types [1].

The other important question is - why pointers at all?

-Levi

------------------
[1] Not to mention the fixed set of parameterized types in Go, but
that's another story.

[Jon Harrop]

What is wrong with .NET's solution to this problem?

[Ian Lance Taylor]

Sebastian Sylvan <sebastia...@gmail.com> writes:

> On Fri, Nov 13, 2009 at 9:06 PM, Ian Lance Taylor <ia...@google.com> wrote:
>
>> This leads us in the direction of the const type qualifier. In my
>> personal opinion, this kind of thing should not be part of the type.
>> I think this amounts to a language design choice. I think that
>> calling it a billion dollar mistake amounts to hyperbole.
>>
>
> I'm sorry but this makes no sense to me. In what way does this have a
> relation to the const qualifier? You can convert between them at any time,
> you just have to make sure you handle the failure case in the unsafe
> direction.

If you don't push nonnull through your program, then what benefit have
you really gained?

> It's not about adding information to types, it's about having a less error
> prone view of what a pointer *is*. Your argument seems to me to be just as
> applicable to the distinction between ints and bools as well - why don't we
> just stick those under the same type? Answer: Because they're fundamentally
> different! Just like the concept of pointers is different to the concept of
> nullable values.
>
> It's his mistake, he can call it what he wants. I don't think he's wrong in
> that a "feature" that introduces potential runtime crashes all over the
> place has been an incredibly expensive mistake.

Go currently crashes if you dereference a nil pointer. Presumably
with a nonnull qualifier, it would crash if you assign a nil pointer
to a pointer type with the nonnull qualifier. What is the fundamental
difference? Either way your program is taking an invalid action, and
it crashes.

Ian

[Sebastian Sylvan]

On Fri, Nov 13, 2009 at 11:17 PM, Ian Lance Taylor <ia...@google.com> wrote:

Sebastian Sylvan <sebastia...@gmail.com> writes:

> On Fri, Nov 13, 2009 at 9:06 PM, Ian Lance Taylor <ia...@google.com> wrote:
>

>> This leads us in the direction of the const type qualifier.  In my
>> personal opinion, this kind of thing should not be part of the type.
>> I think this amounts to a language design choice.  I think that
>> calling it a billion dollar mistake amounts to hyperbole.
>>
>
> I'm sorry but this makes no sense to me. In what way does this have a
> relation to the const qualifier? You can convert between them at any time,
> you just have to make sure you handle the failure case in the unsafe
> direction.

If you don't push nonnull through your program, then what benefit have
you really gained?

You don't "push" it through your program any more than you "push" booleans through your programs. Some things are pointers, some things are nullable, some things are nullable pointers. You could store all your booleans as ints and convert them to bools when you need to, but that would be silly, of course you store them as bools everywhere since that's what they actually are.

 

> It's not about adding information to types, it's about having a less error
> prone view of what a pointer *is*. Your argument seems to me to be just as
> applicable to the distinction between ints and bools as well - why don't we
> just stick those under the same type? Answer: Because they're fundamentally
> different! Just like the concept of pointers is different to the concept of
> nullable values.
>
> It's his mistake, he can call it what he wants. I don't think he's wrong in
> that a "feature" that introduces potential runtime crashes all over the
> place has been an incredibly expensive mistake.

Go currently crashes if you dereference a nil pointer.  Presumably
with a nonnull qualifier, it would crash if you assign a nil pointer
to a pointer type with the nonnull qualifier.  What is the fundamental
difference?  Either way your program is taking an invalid action, and
it crashes.

No, with my suggestion the compiler would catch it. No runtime crash. That's a huge win.

A pointer type wouldn't be nullable, so it would be a different type from a nullable type, which would give a type error if you try to assign one to the other (although an implicit conversion from non-nullable to nullable may be convenient).

--
Sebastian Sylvan

[Intrigued Goer]

On Nov 13, 1:52 pm, Sebastian Sylvan <sebastian.syl...@gmail.com>
wrote:
> On slide 14 of the Go Talk the first design principle is "Keep
> concepts orthogonal", and yet Go violates this principle by repeating
> Hoare's "Billion Dollar Mistake" (http://www.infoq.com/presentations/Null-References-The-Billion-Dollar...
> ) by combining the two orthogonal concepts of "reference to" and
> "nullable" into a single construct.
>
> I know many language designers have said in retrospect that they
> regret getting it wrong (e.g. several from the C# team), so if you're
> going to make a new language why not do it right this time and let
> these two features be two separate features like they're supposed to
> be?

Oi, I get all warm fuzzy from the thought of non-nullable pointers.
The amount of Java software I've seen printing uncaught null pointer
exceptions makes me shake my head. The amount of stupid, repetitive,
error-prone, potentially never-executed null checking code I've
written in C(++) makes me want to never code in C(++) again. And then
there's all the code which has just an assert (or just assumption that
it'll segfault if it's null...), where it's been assumed that any null
pointer bugs were caught in testing, and more importantly any future
code will also be so thoroughly tested (hahaa) that these functions
never get called with null parameters...

Go has no exceptions to handle null pointers nicely, and not even C++-
like references to indicate that something probably shouldn't be null
(though C++ doesn't enforce this, a pity), so handling null pointers
is as much work as in C. But go does have multiple return values
(alternative to returning null to signal error), while it does not
have pointer arithmetic to mess up pointers. And checking this would
just be a type check, so no compile time penalty (type checking is
done anyway), and no run time penalty (nothing done at run time).

To the language developers: Please please implement this. Go has the
other features to make this integrate well with the language, and null
pointer handling is such a huge PITA and such a huge source of errors.
Don't mess up this chance to do things right! And make sure non-
nullable is the default for pointers. You'll kill a million bugs
before they're even made.

For the "null_cast" thing I propose syntax:
ifsafe pointer1[, pointer2...] {
// pointers temporarily non-nullable in here
} else {
// some of the pointers known to be null
}

[Sebastian Sylvan]

On Fri, Nov 13, 2009 at 11:05 PM, roger peppe <rogp...@gmail.com> wrote:

the difficulty with doing something like this in go
as it stands is that go relies on zero values
all over the place. zero values are cheap to create
and cheap on memory, as well as being a
convenient default initial value.

if you had non-nullable pointers, then creating the "zero value"
for a struct containing such a thing would also
entail allocating space for all its references,
only for the code to write the correct values over them again
and leave the original to be deallocated.

I would prefer if you didn't allocate the initial space when the variable is guaranteed to be initialized before it's used. E.g. in Eiffel the references are nullable in the constructor (so you can initialize them at your leisure), and then there's a dynamic check after the constructor has finished to make sure everything has been assigned. 

For Go the idiomatic solution may be to just initialize anything that won't be explicitly initialized to a default instance... Not sure, don't have a good enough "feel" for Go yet to know what the correct thing to do about those pointers are.

Of course, you wouldn't be *removing* nullable pointers, so you're free to use them in cases where it would be too expensive to initialize them to a default target.

moreover it's not even clear that in the presence of
discriminated unions the compiler could know which
arm of the union to choose for the zero value.

I don't think you necessarily need full discriminated unions in the language. Just "nullable" as hard coded special case. "Nullable pointer" would probably compile down to a regular old pointer with a magic value for "null" so it's identical to the current system in that respect.

--
Sebastian Sylvan

[Jon Harrop]

On Friday 13 November 2009 23:17:44 Ian Lance Taylor wrote:
> Sebastian Sylvan <sebastia...@gmail.com> writes:
> > On Fri, Nov 13, 2009 at 9:06 PM, Ian Lance Taylor <ia...@google.com> wrote:
> >> This leads us in the direction of the const type qualifier. In my
> >> personal opinion, this kind of thing should not be part of the type.
> >> I think this amounts to a language design choice. I think that
> >> calling it a billion dollar mistake amounts to hyperbole.
> >
> > I'm sorry but this makes no sense to me. In what way does this have a
> > relation to the const qualifier? You can convert between them at any
> > time, you just have to make sure you handle the failure case in the
> > unsafe direction.
>
> If you don't push nonnull through your program, then what benefit have
> you really gained?

You can write code that cannot ever suffer from null pointer/reference issues,
eliminating an important class of bugs and making your code more reliable.

[Ian Lance Taylor]

So it is that conversion that will crash at runtime.

I'm sorry, I'm not seeing the huge win. If I wanted to tackle this
problem, I would have the compiler interally annotate every function
which dereferenced a pointer. Then I would have the compiler check
those annotations on every parameter pass. Then I would have the
compiler warn whenever a nil pointer gets passed to a function which
may eventually dereference it. I wouldn't touch the type system.

Ian

[Sebastian Sylvan]

No, nothing will crash at runtime. That's the whole point. You are statically guaranteed that every single pointer access is always 100% safe. There are no runtime crashes involved in any aspect of this.

If you have a nullable pointer and you want to actually do anything with it (e.g. dereference it) you have to cast it to a regular pointer - this is a safe cast though, again no crashes. You just have to specify what you want to do in the case it's actually not null, and what to do if it is null. E.g.

if p != nil {

 // here you can use p with 100% statically guaranteed safety, because you can only get here if p is not null at runtime

 // the compiler would change p's type to a pointer here

}

else

{

  // here you have to deal with the fact that p was null... 

  // p is still a nullable here...
}

You can of course choose to invoke a crash in the else clause, but then at least you're being naughty explicitly. The whole rest of your program can completely ignore the possibility of nulls because they're just not possible.

--
Sebastian Sylvan

[Sverre Rabbelier]

Heya,

On Sat, Nov 14, 2009 at 00:54, Sebastian Sylvan
<sebastia...@gmail.com> wrote:
> You can of course choose to invoke a crash in the else clause, but then at
> least you're being naughty explicitly. The whole rest of your program can
> completely ignore the possibility of nulls because they're just not
> possible.

How would you do that if it is not allowed to dereference a nullable
pointer (which would be the sensible thing to do IMO)?

--
Cheers,

Sverre Rabbelier

[Sebastian Sylvan]

I meant choose to invoke a crash in the general sense, not by dereferencing the nullable pointer (which is indeed not possible). E.g. just call  exit(1) or some such.

The point is that at no point do you need to worry about a pointer being null. It may still be the case that a nullable value would indicate fatal errors so that you must crash, but at least it's not being passed around the program for god knows how long until it finally crashes miles from the original problem.

 

--
Sebastian Sylvan

[Jon Harrop]

On Friday 13 November 2009 23:46:35 Ian Lance Taylor wrote:
> So it is that conversion that will crash at runtime.
>

> I'm sorry, I'm not seeing the huge win...

The huge win occurs when there is no conversion.

[Sverre Rabbelier]

Heya,

On Sat, Nov 14, 2009 at 01:15, Sebastian Sylvan
<sebastia...@gmail.com> wrote:
> I meant choose to invoke a crash in the general sense, not by dereferencing
> the nullable pointer (which is indeed not possible). E.g. just call  exit(1)
> or some such.

Ah, I misunderstood your "being naughty" for "trying to dereference
the pointer anyway". But yes, it does make sense that a nullable
pointer turning out to actually be null could signal an error
condition.

--
Cheers,

Sverre Rabbelier

[jqb]

On Nov 13, 3:17 pm, Ian Lance Taylor <i...@google.com> wrote:
> Sebastian Sylvan <sebastian.syl...@gmail.com> writes:

> > On Fri, Nov 13, 2009 at 9:06 PM, Ian Lance Taylor <i...@google.com> wrote:
>
> >> This leads us in the direction of the const type qualifier.  In my
> >> personal opinion, this kind of thing should not be part of the type.
> >> I think this amounts to a language design choice.  I think that
> >> calling it a billion dollar mistake amounts to hyperbole.
>
> > I'm sorry but this makes no sense to me. In what way does this have a
> > relation to the const qualifier? You can convert between them at any time,
> > you just have to make sure you handle the failure case in the unsafe
> > direction.
>
> If you don't push nonnull through your program, then what benefit have
> you really gained?

Your thinking is mired in the non-analogous "const" case. const is
about how a variable will be used, not what values it can take on.

A pointer is nonnull (it's a reference to an object) and most pointer
variables are not intended to take nil values, thus nonnull should be
the default, and nothing has to be "pushed through". Pointers that are
intended to be able to have a nil value should be declared nullable
(or nilable). Since the issue is what the set of allowed values is,
*of course* it should be part of the type -- your statement that it
shouldn't suggests some very confused thinking.

> > It's not about adding information to types, it's about having a less error
> > prone view of what a pointer *is*. Your argument seems to me to be just as
> > applicable to the distinction between ints and bools as well - why don't we
> > just stick those under the same type? Answer: Because they're fundamentally
> > different! Just like the concept of pointers is different to the concept of
> > nullable values.
>
> > It's his mistake, he can call it what he wants. I don't think he's wrong in
> > that a "feature" that introduces potential runtime crashes all over the
> > place has been an incredibly expensive mistake.
>
> Go currently crashes if you dereference a nil pointer.  Presumably
> with a nonnull qualifier, it would crash if you assign a nil pointer
> to a pointer type with the nonnull qualifier.

We're talking about static typing here. In a language that implements
the nonnull/nullable distinction, you can't assign nil to a non-
nullable pointer variable -- that isn't possible, any more than it's
possible to assign the address of an int to a pointer to float
variable; they aren't type-compatible. And it's not a nonnull
qualifier -- it's a *nullable* qualifier; nonnull is the default.

>  What is the fundamental
> difference?  Either way your program is taking an invalid action, and
> it crashes.

The fundamental difference is that static typing turns runtime
failures into compile-time failures. By distinguishing a type that
includes only pointers (references to objects) from a type that
includes both pointers and nil, one can prevent ever treating nil as
if it were a pointer (i.e., dereferencing it) at runtime.

[k...@golang.org]

it is clear that you all have heard of the billion dollar blunder,
but few are aware of the lesser known, but more important, $1.98
mis-step. suppose you want to divide by the variable n, wouldnt
it be nice to guarantee that n is not zero. thus the qualifier
type non-zero, (abbreviated nz with apology to new zealand). so
we add a type adjective to the types int, uint, byte, etc. now

var n nz uint = 0;

would be flagged by the compiler while

var n nz uint;

would assign 1 to n. that way, you can divide by n without worries.

but there is another "bad" value that you might want to avoid, 1<<31
for signed int32. so i guess we come up with some sort of parametric
adjective to accommodate this.

var n butnot(256) byte;

but this will get an error since 256 is not a byte, so we change
to

var n butnot(-256) byte;

several things now work out, we can repeat the parametric type to
exclude many values. thus we can avoid all of the poles/zeroes in
an equation. we can also drop the special declaration of non-null
with this more general form.

lets now fall back to simple nz since we are still debating generic
types. in practice, we will want to sometimes pass an int to a
function that requires an nz-int. the compiler could do the check
for you by compiling inline a compare and panic. i dont think that
is very good, but it could be an option flag to the compiler or
maybe an ifdef or environment variable.

a better solution would be to flag it as an error so that the
programmer could convert the value from one type to another before
calling. thus the programmer would rewrite

f(v);

into

n := (nz int)v; f(n)

where the conversion would check for zero and panic. we have a
solution for that too.

n,ok := (nz int)v; if ok { f(n) }

where n gets the value v if v is not zero and gets the value 1 if
v is zero. since n is now a valid nz-int, we could just write

n,_ := (nz int)v;
f(n);


thus all the pieces are in place for f() to divide
without testing.

[Sebastian Sylvan]

I understand what you're trying to do, but you should try to find an example that has a little more relevance.

For one thing the whole purpose of null is to signify "invalid" and it's always incorrect to dereference it, while zero is a perfectly valid integer in most cases. Second, once a pointer is non-null it doesn't generally become potentially null on most of its operations (+,-,%,* and others could return zero). Third, nullness is an orthogonal concept to "pointerness", while zero is a subset of integers.

In short: Your facetious example would lead to tons of annoying checks whenever you try to do anything - which I assume is your "point",  but we know from real world experience that separating pointers from null-ness does not carry that same penalty at all. This isn't an academic argument - plenty of languages do this. You can look at them and evaluate how it works (hint: extremely well).

This isn't about splitting apart the domain of a type to avoid bad things, it's about not munging two different concepts together into the same type for no good reason other than an historical accident. It's analogous to how C treats just about anything as booleans, while Go (and most other languages) has a proper boolean type. There's no reason why an int should be treated like a boolean, and there's no reason why a pointer should be treated as a nullable.

If you have any actual arguments about why these two orthogonal concepts should be conflated, I think everyone would be happy to hear them, but I don't see how this kind of reply is productive.

--
Sebastian Sylvan

[Ian Lance Taylor]

jqb <jqba...@gmail.com> writes:

> A pointer is nonnull (it's a reference to an object) and most pointer
> variables are not intended to take nil values, thus nonnull should be
> the default, and nothing has to be "pushed through". Pointers that are
> intended to be able to have a nil value should be declared nullable
> (or nilable). Since the issue is what the set of allowed values is,
> *of course* it should be part of the type -- your statement that it
> shouldn't suggests some very confused thinking.

Sorry for my confusion.

What is the value of an uninitialized variable of pointer type?

Ian

[Sebastian Sylvan]

Someone suggested that in keeping with Go's policy on uninitialized values that it would simply point to a "zero value" of whatever type it's pointing to. I like this idea more and more. Of course, you'd make sure to try to initialize it at construction to avoid creating that zero value (just like for any other field at the moment).

Most other languages that do this tend to ensure that it's impossible to construct an object with a pointer that hasn't been initialized by the way they set up object construction. This would be more challenging and require more thought for Go (especially since there's no built in concept of constructors), I think.

Yet others, primarily Eiffel I think, do it with a dynamic check at the end of construction IIRC (which is less than ideal, but at least the crash happens where the problem is, not miles away from it hours later). Even if you go the Eiffel route, at least you narrow down the potential problem area to the initialization - once you're through that you don't have to worry about it any more.

--
Sebastian Sylvan

[Ola Fosheim Grøstad]

On 13 Nov, 20:48, Ian Lance Taylor <i...@google.com> wrote:
> billion dollar mistake.  In my C/C++ programming I've never noticed
> that NULL pointers are a noticeable source of bugs.

The cost of a single NULL-dereference after deployment of a server can
be quite big.

Not all programmers are equally careful. You are probably a better
programmer than most. I agree with Hoare: language designers should
take the responsibility for errors made by their users. Rule #1 for
usability: you cannot use yourself as a model for the users of your
own product.

Anyway, C++ has references (&) which kinda are (not quite) like non-
null reference that you cannot assign to... So they have something
better than nullable pointers, at least. If I had a vote I'd vote for
non-null references, but then again I'd also vote for contracts
(invariant checking) with the ability to require some of them to be
established at compile time. I think it would be nice for debugging to
be able to attach contracts to types and have them checked everywhere.
Much better than asserts. NULL-dereferencing is just a special case of
what should be adressed IMO.

Ola.

[Hans]

On Nov 13, 3:52 am, Sebastian Sylvan <sebastian.syl...@gmail.com>
wrote:
> ... yet Go violates this principle by repeating

> Hoare's "Billion Dollar Mistake" (http://www.infoq.com/presentations/Null-References-The-Billion-Dollar...
> ) by combining the two orthogonal concepts of "reference to" and
> "nullable" into a single construct.

I watched the video, or at least as much as I could before getting
really sick of hearing Hoare pontificate.

It's only a billion dollar mistake if you assume all programmers never
advance beyond the novice level. After listening to Hoare, I can
believe that he would make that assumption.

[Peter Froehlich]

This discussion must have deteriorated pretty far while I wasn't
looking: We've reached the "Hoare is stupid!" level of the argument.
Amazing. :-D

[Ola Fosheim Grøstad]

> What is the value of an uninitialized variable of pointer type?

You may require the compiler to ensure that it is provably not used
until it has obtained a valid reference.

[Dwight Schauer]

This might catch some obvious cases.

But for pointer elements of a data structure that is passed some where
else, that might be difficult to track at compile time. How does the
compiler know if you are passing it elsewhere for them to be
initialized or used?

[Ola Fosheim Grøstad]

On 14 Nov, 00:54, Sebastian Sylvan <sebastian.syl...@gmail.com> wrote:
if p != nil {
>  // here you can use p with 100% statically guaranteed safety, because you
> can only get here if p is not null at runtime
>  // the compiler would change p's type to a pointer here}
>
> else
> {
>   // here you have to deal with the fact that p was null...
>   // p is still a nullable here...
>
> }
>
> You can of course choose to invoke a crash in the else clause, but then at
> least you're being naughty explicitly. The whole rest of your program can
> completely ignore the possibility of nulls because they're just not
> possible.

If the compiler actually KNOWS that p is null then it should of course
issue an error. Why should the compiler allow programmers to be
suicidal? Even with separate compiled units you should be able to
propagate that kind of information in situations where functions
always assume that a parameter is non-null, even without a non-null
pointer type.

I never quite understood why compilers should just meet the specs of
the language and not do more if they are capable of infering more...?
I think Go will have to either settle for being non-suicidal or
suicidal. If they go for the latter, then give me inline asm plz...

Ola.

[Hans]

> We've reached the "Hoare is stupid!" level of the argument.
> Amazing. :-D

I didn't say he was stupid. I do, however, feel confident in saying
that to claim his mistake cost a billion dollars, he must have a very
low opinion of all other software developers.

[Dwight Schauer]

On Sat, Nov 14, 2009 at 3:31 PM, Ola Fosheim Grøstad
<ola.foshe...@gmail.com> wrote:
> If the compiler actually KNOWS that p is null then it should of course
> issue an error. Why should the compiler allow programmers to be
> suicidal? Even with separate compiled units you should be able to
> propagate that kind of information in situations where functions
> always assume that a parameter is non-null, even without a non-null
> pointer type.
>
> I never quite understood why compilers should just meet the specs of
> the language and not do more if they are capable of infering more...?
> I think Go will have to either settle for being non-suicidal or
> suicidal. If they go for the latter, then give me inline asm plz...

The more I think about this issue, the more I agree that it is
something the compiler should handle, and not be suicidal.

But this issue extends to index range checking at compile time as
well, which is a different subject, which would force the use of
ranged integer types.

[Ola Fosheim Grøstad]

On 14 Nov, 22:24, Dwight Schauer <dscha...@gmail.com> wrote:
> This might catch some obvious cases.
>
> But for pointer elements of a data structure that is passed some where
> else, that might be difficult to track at compile time. How does the
> compiler know if you are passing it elsewhere for them to be
> initialized or used?

invariant 1: you are never allowed to assign null to a non-null
pointer

invariant 2: you are never allowed to hand-off uninitialized non-null
pointers to units that cannot be proved to either not initialize them
or provably make them "not live" (ready for "garbage collection").

I believe you can easily prove invariant 2 in terms of reachability.
If you can reach either an unknown location or a location where it is
being used without it being established as initialized then the
compiler should issue an error. You simply make that a requirement for
non-null pointers. (formally in terms of more precise mathematical
language, e.g. an algorithm.)

Ola.

[Sebastian Sylvan]

On Sat, Nov 14, 2009 at 9:31 PM, Ola Fosheim Grøstad <ola.foshe...@gmail.com> wrote:

On 14 Nov, 00:54, Sebastian Sylvan <sebastian.syl...@gmail.com> wrote:
if p != nil {
>  // here you can use p with 100% statically guaranteed safety, because you
> can only get here if p is not null at runtime
>  // the compiler would change p's type to a pointer here}
>
> else
> {
>   // here you have to deal with the fact that p was null...
>   // p is still a nullable here...
>
> }
>
> You can of course choose to invoke a crash in the else clause, but then at
> least you're being naughty explicitly. The whole rest of your program can
> completely ignore the possibility of nulls because they're just not
> possible.

If the compiler actually KNOWS that p is null then it should of course
issue an error.

I don't think so. It should allow you to do something else instead. If you open a file and then do a "null cast" kind of thing on it, and end up in the "this pointer was null" clause because the file wasn't found, you should be allowed to print a warning and return, or something else. That "something else" may be "exit(1)", but it may be possible to recover in a lot of these cases. The "null cast" is really just about specifying two paths of execution - one for when it's null, and one for when it isn't.

 

Why should the compiler allow programmers to be
suicidal?

I didn't mean that they would be allowed to dereference the nullable pointer in the else clause. They still can't do that. They just have the opportunity to either crash (by calling some error routine) or handle it in some other way when possible.

--
Sebastian Sylvan

[Ola Fosheim Grøstad]

On 14 Nov, 22:50, Sebastian Sylvan <sebastian.syl...@gmail.com> wrote:

> I didn't mean that they would be allowed to dereference the nullable pointer
> in the else clause. They still can't do that. They just have the opportunity
> to either crash (by calling some error routine) or handle it in some other
> way when possible.

Ah, ok. I misunderstood you and you misunderstood me... ;-)
(I usually don't think of calling an error routine as a crash.)

Ola.

[zimbatm]

Don't forget that go can return multiple values.

func SomeFunc() *SomeType

given only the preceding function signature, you cannot have the
guarantee that calling that function won't give you a nil. It thus
implies that each time you get the value of SomeFunc(), you need a
branch statement to check the presence of nil.

func SomeFunc2() (ret *SomeType, err os.Error)

In practice, functions returning a "maybe-nil-pointer" will also
return a second argument that gives the reason of failure. I believe
that
makes all the difference. While not a syntactic or compiler check, it
is already a good defence against segmentation faults.

[Ian Lance Taylor]

Sebastian Sylvan <sebastia...@gmail.com> writes:

> On Sat, Nov 14, 2009 at 7:59 PM, Ian Lance Taylor <ia...@google.com> wrote:
>
>> What is the value of an uninitialized variable of pointer type?
>
>
> Someone suggested that in keeping with Go's policy on uninitialized values
> that it would simply point to a "zero value" of whatever type it's pointing
> to. I like this idea more and more. Of course, you'd make sure to try to
> initialize it at construction to avoid creating that zero value (just like
> for any other field at the moment).

A zero value does not seem like the ideal choice to me. Except for
the case where pointing to a zero value is correct for the program, it
changes a program error from a crash to the use of an incorrect value.
The latter seems more dangerous to me on average, since it doesn't
crash.

> Most other languages that do this tend to ensure that it's impossible to
> construct an object with a pointer that hasn't been initialized by the way
> they set up object construction. This would be more challenging and require
> more thought for Go (especially since there's no built in concept of
> constructors), I think.

Right.

Ian

[Brian Slesinsky]

Null pointer exceptions are common in Java and tend to arise due to
misunderstandings between programmers. Interfaces allow separately
written callers and callees to be plugged into each other, and without
a clear specification at the interface level, it's unclear who's at
fault when a null pointer exception happens. Go is more flexible so it
seems like the potential for confusion is all the greater. But it's a
bit too soon to tell whether this will actually be a problem. We
should see whether it comes up as people outside Google start using
Go.

If it does, it seems like misunderstandings could best be avoided by
annotating each public function's parameters and return values as
either "nullable" or "never null" in a way the compiler can
understand. The compiler can figure out the rest. Perhaps the compiler
could check this without any annotations, but then errors will be non-
local and harder to understand, particularly when the caller and
callee are far removed and maintained by different organizations. It's
not good to have to read a third-party library's implementation to
determine that you violated its interface.

On Nov 15, 7:56 pm, Ian Lance Taylor <i...@google.com> wrote:
> Sebastian Sylvan <sebastian.syl...@gmail.com> writes:

[Sebastian Sylvan]

On Mon, Nov 16, 2009 at 3:56 AM, Ian Lance Taylor <ia...@google.com> wrote:

Sebastian Sylvan <sebastia...@gmail.com> writes:

> On Sat, Nov 14, 2009 at 7:59 PM, Ian Lance Taylor <ia...@google.com> wrote:
>

>> What is the value of an uninitialized variable of pointer type?
>
>
> Someone suggested that in keeping with Go's policy on uninitialized values
> that it would simply point to a "zero value" of whatever type it's pointing
> to. I like this idea more and more. Of course, you'd make sure to try to
> initialize it at construction to avoid creating that zero value (just like
> for any other field at the moment).

A zero value does not seem like the ideal choice to me.  Except for
the case where pointing to a zero value is correct for the program, it
changes a program error from a crash to the use of an incorrect value.
The latter seems more dangerous to me on average, since it doesn't
crash.

But this is true for all kinds of zero-initialization. Should we add a "null" value to ints and strings too for the same reason then?

--
Sebastian Sylvan

[Sebastian Sylvan]

Let me amend this by saying that I actually agree with you: Zero values are not an ideal solution to the problem of uninitialized values, because it only works when the zero value is correct for the program. I'd prefer if Go didn't go that route, but unfortunately that's what it does.

Ideally Go would instead just prevent you from creating uninitialized values at compile time (perhaps with a shorthand to generate defaults at the declaration site). 

Either way, I definitely do *not* think the right solution to the problem is to assign the uninitialized fields a magic "invalid value" that crashes on first use. That goes for types like int, float and string, and it goes for pointers as well.

So. I'm not saying that initializing to a zero value is necessarily the globally best way to do it, I'm just saying it's consistent with the way Go handles initialization for everything else. 

I don't think pointers should be a special case, but if you inist that it should be a special case it should at least be special in the sense of a compile time error rather than a runtime crash (i.e. rather than assigning null to it, simply disallow pointers from being uninitialized).

--
Sebastian Sylvan

[Ian Lance Taylor]

Sebastian Sylvan <sebastia...@gmail.com> writes:

>> On Mon, Nov 16, 2009 at 3:56 AM, Ian Lance Taylor <ia...@google.com> wrote:
>>>
>>> A zero value does not seem like the ideal choice to me. Except for
>>> the case where pointing to a zero value is correct for the program, it
>>> changes a program error from a crash to the use of an incorrect value.
>>> The latter seems more dangerous to me on average, since it doesn't
>>> crash.
>>>
>>
>> But this is true for all kinds of zero-initialization. Should we add a
>> "null" value to ints and strings too for the same reason then?
>>
>
> Let me amend this by saying that I actually agree with you: Zero values are
> not an ideal solution to the problem of uninitialized values, because it
> only works when the zero value is correct for the program. I'd prefer if Go
> didn't go that route, but unfortunately that's what it does.

Sure, I can see an argument for a language which does not permit
uninitialized variables. But Go is not that language. For most
types, given that there can be uninitialized variables, I think that
zero is a reasonable choice for the initialization value. I think you
are proposing that for a variable of type *T, we set the uninitialized
value to be not nil but new(T). Then we presumably prohibit assigning
nil to a pointer variable; perhaps we can eliminate nil entirely. The
benefit is that programs can not crash due to dereferencing a nil
pointer. The cost is more work allocating values and more work in the
GC--sometimes much more work if we have something like [1000]*T.

Is that a fair statement?

Ian

[Chas. Owens]

I would say that it is an error to not initialize a non-nilable type.

--
Chas. Owens
wonkden.net
The most important skill a programmer can have is the ability to read.

[Jon Harrop]

On Monday 16 November 2009 15:07:44 Ian Lance Taylor wrote:
> Sure, I can see an argument for a language which does not permit
> uninitialized variables. But Go is not that language.

When are uninitialized variables useful?

> For most
> types, given that there can be uninitialized variables, I think that
> zero is a reasonable choice for the initialization value. I think you
> are proposing that for a variable of type *T, we set the uninitialized
> value to be not nil but new(T). Then we presumably prohibit assigning
> nil to a pointer variable; perhaps we can eliminate nil entirely. The
> benefit is that programs can not crash due to dereferencing a nil
> pointer. The cost is more work allocating values and more work in the
> GC--sometimes much more work if we have something like [1000]*T.
>
> Is that a fair statement?

There is no need to degrade performance. You can just represent the empty
value of the optional pointer type by null at run time. F# does this, for
example. Indeed, in F# you can annotate the definition of a union type in
order to get the first argumentless constructor represented by null at run
time in order to improve performance (at the cost of run-time type
information on that value, e.g. None.GetType() fails with
NullReferenceException).

[Sebastian Sylvan]

If you go the route of zero initializing pointers then yes, you will create more work by having to create a bunch of zero objects. You already treat pointers specially (by including a "magic invalid value" in its domain), so it's not inconceivable that you would still treat it as a special case requiring that a pointer is always initialized at creation time. 

However, if the zero-value isn't what you want you would presumably set the pointer to its proper value fairly shortly after creation (probably even in an initialization expression) so it's not unlikely that most of this extra work could be optimized away. You wouldn't remove nil though, there are places where nil pointers are crucial (e.g. a linked list), it just wouldn't be the default.

Also, you gain performance by not having to do any nil checks on a non-nullable pointer (either the programmer, or the runtime, for whatever reason).

--
Sebastian Sylvan

[Ian Lance Taylor]

"Chas. Owens" <chas....@gmail.com> writes:

> On Sat, Nov 14, 2009 at 14:59, Ian Lance Taylor <ia...@google.com> wrote:
>> jqb <jqba...@gmail.com> writes:
>>
>> What is the value of an uninitialized variable of pointer type?
>

> I would say that it is an error to not initialize a non-nilable type.

But Go does have uninitialized values. I don't see how to ignore that
fact. That's what you get from "var v T" or "new(T)".

Ian

[Ian Lance Taylor]

Jon Harrop <j...@ffconsultancy.com> writes:

> On Monday 16 November 2009 15:07:44 Ian Lance Taylor wrote:
>> Sure, I can see an argument for a language which does not permit
>> uninitialized variables. But Go is not that language.
>
> When are uninitialized variables useful?

I mean uninitialized in the sense that no explicit initial value is
provided. Go specifies that value that is taken when there is no
explicit initial value, so a value which is not explicitly initialized
is often useful. Not always, of course.

>> For most
>> types, given that there can be uninitialized variables, I think that
>> zero is a reasonable choice for the initialization value. I think you
>> are proposing that for a variable of type *T, we set the uninitialized
>> value to be not nil but new(T). Then we presumably prohibit assigning
>> nil to a pointer variable; perhaps we can eliminate nil entirely. The
>> benefit is that programs can not crash due to dereferencing a nil
>> pointer. The cost is more work allocating values and more work in the
>> GC--sometimes much more work if we have something like [1000]*T.
>>
>> Is that a fair statement?
>
> There is no need to degrade performance. You can just represent the empty
> value of the optional pointer type by null at run time. F# does this, for
> example. Indeed, in F# you can annotate the definition of a union type in
> order to get the first argumentless constructor represented by null at run
> time in order to improve performance (at the cost of run-time type
> information on that value, e.g. None.GetType() fails with
> NullReferenceException).

But what is the initial value of the non-nullable pointer type?

Ian

[Szabolcs Nagy]

On 11/16/09, Jon Harrop <j...@ffconsultancy.com> wrote:
> On Monday 16 November 2009 15:07:44 Ian Lance Taylor wrote:
>> Sure, I can see an argument for a language which does not permit
>> uninitialized variables. But Go is not that language.
>
> When are uninitialized variables useful?

right before initialization

otherwise it would be hard to handle complex cross referencing structures

let's say there is a graph where each vertex has a couple of pointers
to other vertices and each of those pointer should be valid (non nil)
during the lifetime of the graph

how would you initialize such a structure if uninitialized or nil
pointers are not allowed?

[jqb]

This is another reason that out-of-bounds exceptions are preferable --
they don't require mangling types to include invalid values in case of
error. However, if pointers are by default nonnullable, SomeFunc2's
ret value will be initialized to a dummy value, so there would be no
change in the code.

[jqb]

On Nov 14, 11:15 am, k...@golang.org wrote:
> it is clear that you all have heard of the billion dollar blunder,
> but few are aware of the lesser known, but more important, $1.98
> mis-step.

The invalidity of your argument is highlighted by the fact that a) the
$1.98 misstep *isn't* more important b) As has been noted REPEATEDLY,
nonnull would the default, not a type adjective and c) most important
of all, 0 IS an integer but nil is NOT a pointer. But if, nonetheless,
you want to advocate for a "positive integer" or "natural number"
type, that isn't as ridiculous as you make out.

And there's another argument against your position: Tony Hoare isn't
an idiot.

[Chas. Owens]

The construct

var non_nilable_pointer <whateverthenotationis>int;

would not be legal. You would be required to say

var non_nilable_pointer <whateverthenotationis>int = <some valid address>

or

non_nilable_pointer := whateverthenotationis(<some valid address>)

Trying to use a a non-nilable pointer with something that expects a
nilable pointer and vice versa would be a type error. Non-nilable
pointers could be cast into nilable pointers and vice versa. I don't
see a need for special logic around this, if someone wants to screw up
their program by manually casting a value in a way that breaks things,
that is their problem, there are no special warning to tell you that
this is a stupid idea:

var a int32 = 300;
var b int8 = int8(a);

[Ian Lance Taylor]

Sebastian Sylvan <sebastia...@gmail.com> writes:

> On Mon, Nov 16, 2009 at 3:07 PM, Ian Lance Taylor <ia...@google.com> wrote:
>
>> Sure, I can see an argument for a language which does not permit
>> uninitialized variables. But Go is not that language. For most
>> types, given that there can be uninitialized variables, I think that
>> zero is a reasonable choice for the initialization value. I think you
>> are proposing that for a variable of type *T, we set the uninitialized
>> value to be not nil but new(T). Then we presumably prohibit assigning
>> nil to a pointer variable; perhaps we can eliminate nil entirely. The
>> benefit is that programs can not crash due to dereferencing a nil
>> pointer. The cost is more work allocating values and more work in the
>> GC--sometimes much more work if we have something like [1000]*T.
>>
>> Is that a fair statement?
>
>
> If you go the route of zero initializing pointers then yes, you will create
> more work by having to create a bunch of zero objects. You already treat
> pointers specially (by including a "magic invalid value" in its domain), so
> it's not inconceivable that you would still treat it as a special case
> requiring that a pointer is always initialized at creation time.

I don't see how that would fit into Go, which uses uninitialized
values in a couple of different places.

> However, if the zero-value isn't what you want you would presumably set the
> pointer to its proper value fairly shortly after creation (probably even in
> an initialization expression) so it's not unlikely that most of this extra
> work could be optimized away.

I don't really see it being easily optimized away for [1000]*T.

> You wouldn't remove nil though, there are
> places where nil pointers are crucial (e.g. a linked list), it just wouldn't
> be the default.

In a linked list you would use a nullable pointer? But the initial
value of a nullable pointer would not be nil either?

> Also, you gain performance by not having to do any nil checks on a
> non-nullable pointer (either the programmer, or the runtime, for whatever
> reason).

The runtime only does nil checks in very special cases, when a pointer
is used in a way that, if the pointer is nil, would cause a read of
invalid memory which is not on the first page of memory.

Ian

[Jon Harrop]

On Monday 16 November 2009 20:17:37 Szabolcs Nagy wrote:
> On 11/16/09, Jon Harrop <j...@ffconsultancy.com> wrote:
> > When are uninitialized variables useful?
>
> right before initialization
>
> otherwise it would be hard to handle complex cross referencing structures
>
> let's say there is a graph where each vertex has a couple of pointers
> to other vertices and each of those pointer should be valid (non nil)
> during the lifetime of the graph

So for the cycle a<->b<->c<->d<->a, you would do something like this:

Vertex a = [null; null]
Vertex b = [a; null]
Vertex c = [b; null]
Vertex d = [c; null]
a[0] := d
a[1] := b
b[1] := c
c[1] := d
d[1] := a

> how would you initialize such a structure if uninitialized or nil
> pointers are not allowed?

In an imperative language, I would use a dummy vertex or edge.

If possible, I would do it declaratively like this:

let rec a = [b; d] and b = [a; c] and c = [b; d] and d = [a; c]

[Erik Wikström]

On Nov 13, 9:00 pm, Sebastian Sylvan <sebastian.syl...@gmail.com>
wrote:
> On Fri, Nov 13, 2009 at 7:48 PM, Ian Lance Taylor <i...@google.com> wrote:

> > I personally think that in a systems programming language, a pointer
> > is a reasonable concept: it intuitively represents a memory address.
> > I don't personally think that permitting pointers to be nil is a
> > billion dollar mistake. In my C/C++ programming I've never noticed
> > that NULL pointers are a noticeable source of bugs.
>
> A pointer is a reasonable concept, but there's no reason combine the concept
> of a pointer with the concept of "nullable". Representing a memory address
> is perfectly fine and useful, but from there it doesn't follow that we
> should also allow it to point to an invalid memory address that crashes if
> you try to use it.

Do I understand you correctly that you would want to forbid NULL (or
whatever
you want to call it) as a value for a pointer/reference? In that case
how would
one model something like the following:

A user accessing a system must authenticate themselves, however some
users use
the built-in authentication-system while others use an external
authentication
server. Accounting might also be active for the user.

The normal way of implementing this might be something like this:

struct Server;

struct User
{
struct Server* Accounting;
struct Server* Authorization;
};

By setting the relevant pointers to NULL we can indicate that the user
does not
need to authenticate to an externa server and/or that accounting is
not active.
But if we must always have valid values for all pointers/references
this is no
longer possible, so how would you propose that we do instead?

I suppose that we might create dummy objects that can be refered to
when no
"real" object exists. However I don't see this solving any problems
since we
would then just exchange all the NULL-checks against dummy-checks
(which might
be harder to do since all values of all members might be valid,
resulting in an
isDummy member or similar). And when the programmers fails to properly
check for
a dummy we get other interesting errors.

I'm aware of various version of NULL-able types but I'm not
particulary fond of
them in a systems programming language. Unless you use clever tricks
the NULLed
versions of the objects eats just as much memory as the non-NULLed (or
more),
and if they don't there might be extra overhead. Besides I'm not very
fond of
the idea of of making a memory access just to discover that the object
at the
address is NULLed, in best case for me that means a five cycles stall,
and in
worst case we are talking about houndreds of cycles. While I'm aware
that most
people/programmers don't have to worry much about neither memory or
CPU there
are a lot of us that do, and for us a programming language must be
efficient.

I should admit that I have not keept up to date with the latest in
language
development so there might be some efficient solution to the problem,
in which
case I'd love to head about it.

--
Erik Wikström

[Ian Lance Taylor]

"Chas. Owens" <chas....@gmail.com> writes:

> The construct
>
> var non_nilable_pointer <whateverthenotationis>int;
>
> would not be legal. You would be required to say
>
> var non_nilable_pointer <whateverthenotationis>int = <some valid address>
>
> or
>
> non_nilable_pointer := whateverthenotationis(<some valid address>)

How do I say "p := new(*T)"?

How does it work if the pointer is in a struct?

How does it work if the pointer is in a struct which is embedded in
another struct?

How does it work if you have two structs which need to point to each
other?

Ian

[Ian Lance Taylor]

jqb <jqba...@gmail.com> writes:

> The invalidity of your argument is highlighted by the fact that a) the
> $1.98 misstep *isn't* more important b) As has been noted REPEATEDLY,
> nonnull would the default, not a type adjective and c) most important
> of all, 0 IS an integer but nil is NOT a pointer. But if, nonetheless,
> you want to advocate for a "positive integer" or "natural number"
> type, that isn't as ridiculous as you make out.

I haven't yet seen the argument where we avoid a type adjective. Can
you expand on that one?

Ian

[Sebastian Sylvan]

Uh, didn't you just argue for introducing data flow analysis a minute ago? If you create a value and then immediately initialize it, you could easily optimize the temporary zero-value away. Either way, it's still legal if you don't, and if it's a problem just use a nullable pointer.

 

> You wouldn't remove nil though, there are
> places where nil pointers are crucial (e.g. a linked list), it just wouldn't
> be the default.

In a linked list you would use a nullable pointer?  But the initial
value of a nullable pointer would not be nil either?

For a nullable value nil seems like a sensible default since it makes sense semantically. 

 

> Also, you gain performance by not having to do any nil checks on a
> non-nullable pointer (either the programmer, or the runtime, for whatever
> reason).

The runtime only does nil checks in very special cases, when a pointer
is used in a way that, if the pointer is nil, would cause a read of
invalid memory which is not on the first page of memory.

Presumably the GC checks for nil all the time when traversing the heap? 

--
Sebastian Sylvan

[Sebastian Sylvan]

Google "Erlang void safety". In short, you'd still have NULL for nullable values (which may be pointers), it's just that you wouldn't combine the concept of "point to" with "may be NULL" into a single type. You'd keep them as two orthogonal concepts that can be used together when needed.

--
Sebastian Sylvan

[Sebastian Sylvan]

EIFFEL! Not Erlang... Doh.

--
Sebastian Sylvan

[Szabolcs Nagy]

On 11/16/09, jqb <jqba...@gmail.com> wrote:
> On Nov 14, 11:15 am, k...@golang.org wrote:
>> it is clear that you all have heard of the billion dollar blunder,
>> but few are aware of the lesser known, but more important, $1.98
>> mis-step.
>
> The invalidity of your argument is highlighted by the fact that a) the

i found his argument compelling
(he might have some systems programming experience ..who knows)

> $1.98 misstep *isn't* more important b) As has been noted REPEATEDLY,
> nonnull would the default, not a type adjective and c) most important
> of all, 0 IS an integer but nil is NOT a pointer. But if, nonetheless,
> you want to advocate for a "positive integer" or "natural number"
> type, that isn't as ridiculous as you make out.
>
> And there's another argument against your position: Tony Hoare isn't
> an idiot.

i don't think anyone implied that tony hoare was an idiot

null pointers are useful because they are simple and efficient
solution to certain problems

i guess they can be hidden from the programmer behind 'maybe pointer'
types or using 'nullable'/'non-nullable' annotations, but i doubt that
would completely solve the problem: one can invent his personal dummy
pointer with the same problems anyway

pointers are like array indices, they might be out of bound or one
might access uninitialized values through them

one can make them safer by introducing a more complex mental model
(and some performance loss usually), but keep in mind that complex
mental models have a price too (about $1.98 billion)

[Jon Harrop]

On Monday 16 November 2009 19:47:09 Ian Lance Taylor wrote:
> But what is the initial value of the non-nullable pointer type?

I see. The default value of non-null reference types in F# is actually null
(e.g. of lists). However, you have to go to some lengths to get one.

[Jon Harrop]

On Monday 16 November 2009 21:14:19 Sebastian Sylvan wrote:
> Presumably the GC checks for nil all the time when traversing the heap?

This begs another question: is the GC partially specialized wrt the types in
the program? I assume Go doesn't use a uniform representation (like OCaml)
and it cannot JIT run-time generated code (like .NET) so does it generate GC
code to handle each type when the type is defined and compile it statically
along with everything else?

[Asger Ottar Alstrup]

Non-null is extremely important. It is not just Hoare that says this, but many others, including Anders Hejlsberg, the man behind C#. If he was to redo C# today, he would do something about null:

http://stackoverflow.com/questions/178026/why-is-null-present-in-c-and-java

See the link to Spec#, which has a proposal for a solution for this:

http://research.microsoft.com/en-us/projects/specsharp/krml136.pdf

See page 2:

"Many errors in modern programs manifest themselves as null-dereference errors, suggesting the importance of a programming language providing the ability to discriminate between expressions that may evaluate to null and those that are sure not to (for some experimental evidence, see [24,22]). In fact, we would like to eradicate all null-dereference errors."

Their references are:

24. Cormac Flanagan, K. Rustan M. Leino, Mark Lillibridge, Greg Nelson, James B. Saxe, and Raymie Stata. Extended static checking for Java. In Proceedings of the 2002 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), volume37, number 5 in SIGPLAN Notices, pages 234–245. ACM, May 2002.

22. Manuel Fähndrich and K. Rustan M. Leino. Declaring and checking non-null types in an object-oriented language. In Proceedings of the 2003 ACM Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2003, volume 38, number 11 in SIGPLAN Notices, pages 302–312. ACM, November 2003.

It seems those papers are found with these links:

http://www.eecs.umich.edu/~bchandra/courses/papers/Flanagan_ESC.pdf

http://research.microsoft.com/en-us/um/people/leino/papers/krml109.pdf

Regards,

Asger Ottar Alstrup

[Jon Harrop]

On Monday 16 November 2009 21:43:38 Asger Ottar Alstrup wrote:
> Non-null is extremely important. It is not just Hoare that says this, but
> many others, including Anders Hejlsberg, the man behind C#. If he was to

> redo C# today, he would do something about null...

Indeed. That is precisely why Microsoft went to such great lengths to address
this issue in F#. The solution they adopted has already been widely tested,
works well and offers optimal efficiency. Indeed, a partially specialized
run-time could even remove null checks when it knows (from the static type)
that a reference cannot be null, further improving performance.

However, I think the utility of this approach swings on lots of language
features that Go doesn't have and some that it may never get. Without that
foundation, there seems to be little motivation for doing anything at all
right now except to note that this is another compelling example of what
generics are good for.

[Ian Lance Taylor]

Sebastian Sylvan <sebastia...@gmail.com> writes:

>> The runtime only does nil checks in very special cases, when a pointer
>> is used in a way that, if the pointer is nil, would cause a read of
>> invalid memory which is not on the first page of memory.
>
>
> Presumably the GC checks for nil all the time when traversing the heap?

No, it looks up each pointer in its reverse data structures to find
the object to which it points. Pointers which do not point to any
known heap object are ignored; this obviously includes nil pointers.
This kind of approach is required because a pointer could point at a
global variable, or point into the C heap, and therefore not be under
the control of the garbage collector.

Ian

[Ian Lance Taylor]

Jon Harrop <j...@ffconsultancy.com> writes:

> On Monday 16 November 2009 21:14:19 Sebastian Sylvan wrote:
>> Presumably the GC checks for nil all the time when traversing the heap?
>
> This begs another question: is the GC partially specialized wrt the types in
> the program? I assume Go doesn't use a uniform representation (like OCaml)
> and it cannot JIT run-time generated code (like .NET) so does it generate GC
> code to handle each type when the type is defined and compile it statically
> along with everything else?

The current GC is known to be suboptimal, and is not specialized for
types at all. It is a simple mark and sweep, and will get confused by
e.g., int variables which hold values that look like pointers (get
confused in the sense that it will cause values to not get collected,
not get confused in the sense of causing the program to fail).

In the future I expect that the GC will use the type descriptors which
already exist for type reflection, or more likely some more efficient
data structure derived from them. There is no run-time compiled code
in Go so that issue does not arise.

Ian

[Sebastian Sylvan]

Ah okay. I guess the main performance benefit in that case would be from specializing functions that do take a nullable (i.e. when nil has semantic meaning) so that you can skip the manual nullness checks when you know the pointer is non-null. There probably aren't very many of these, though (in fact, the rarity of legitimate nil pointers is a major motivating factor for not putting nil in the main pointer type).

--
Sebastian Sylvan

[Jessta]

On 17/11/2009, Ian Lance Taylor <ia...@google.com> wrote:
> How do I say "p := new(*T)"?
>
> How does it work if the pointer is in a struct?
>
> How does it work if the pointer is in a struct which is embedded in
> another struct?
>
> How does it work if you have two structs which need to point to each
> other?
>
> Ian
>

There seems to be a bit of confusion here, the proposal is to have
non-nullable as the default with pointers being able to be declared
nullable if needed.

In Go there are a few ways a pointer can become nil,
1. a pointer is created and set to nil
2. a pointer is within a structure and hasn't yet been initialised
3. a pointer is set to the return value of a function that can return nil

The first case is unlikely to be a problem, the developer set the
pointer to nil for a good reason

The second case is more likely but isn't really helped by non-nullable
because most of the time a structure that contains a pointer is going
to need that pointer to be nullable(otherwise they would have just
embeded the thing it's pointing to in to the struct)

The third case is prevented by the (nil,error) multiple value return
convention for functions that will return a nil. A developer that
ignores this is obviously a bad developer and nobody should use their
code anyway. This developer is just as likely to call Just() on a
maybe monad in haskell without calling isJust() or isNothing() first.

In Java, C#, C, C++ the null pointer problem is usually due to
functions returning NULL and the developer not knowing that the
function could return NULL or just forgetting that it does.
With Go's (nil,error) convention this isn't likely to happen so I see
this as a non-problem.

- Jessta
--
=====================
http://jessta.id.au

[jqb]

On Nov 16, 1:17 pm, Szabolcs Nagy <nszabo...@gmail.com> wrote:

> On 11/16/09, jqb <jqbal...@gmail.com> wrote:
>
> > On Nov 14, 11:15 am, k...@golang.org wrote:
> >> it is clear that you all have heard of the billion dollar blunder,
> >> but few are aware of the lesser known, but more important, $1.98
> >> mis-step.
>
> > The invalidity of your argument is highlighted by the fact that a) the
>
> i found his argument compelling

Bully for you. I explained why his argument is invalid; do you have
any substantive response to my points?

> (he might have some systems programming experience ..who knows)

Strawman. I also have some -- 45 years worth.

> > $1.98 misstep *isn't* more important b) As has been noted REPEATEDLY,
> > nonnull would the default, not a type adjective and c) most important
> > of all, 0 IS an integer but nil is NOT a pointer. But if, nonetheless,
> > you want to advocate for a "positive integer" or "natural number"
> > type, that isn't as ridiculous as you make out.
>
> > And there's another argument against your position: Tony Hoare isn't
> > an idiot.
>
> i don't think anyone implied that tony hoare was an idiot
>
> null pointers are useful because they are simple and efficient
> solution to certain problems

As Fermi said, this so off base it isn't even wrong. No one claims
that nil, or nullable pointer variables, aren't useful. All of those
solutions are implementable in languages that make the pointer/
nullable distinction, in the same way -- those languages simply
provide static type checking that prevents dereferencing nil.

> i guess they can be hidden from the programmer behind 'maybe pointer'
> types or using 'nullable'/'non-nullable' annotations,

You guess? That's what we've been talking about.

> but i doubt that
> would completely solve the problem: one can invent his personal dummy
> pointer with the same problems anyway

Sebastian Sylvan addressed this. It's not the same problem; while
dereferencing a pointer to an invalid object is a bug, it isn't a
program crash. It's remarkable that Go is touted as being designed for
servers, but it neither provides protection against dereferencing nil
NOR exceptions -- which means that all dereferences of nil result in
global crashes. Exceptions allow keeping the consequences of bugs
local to a routine, or package, or thread -- such localization of
consequences may not matter to someone designing for the pdp-11.

> pointers are like array indices, they might be out of bound or one
> might access uninitialized values through them

If it were possible to avoid out-of-bounds array accesses via compile-
time static checks, that would be a good thing, yes?

> one can make them safer by introducing a more complex mental model
> (and some performance loss usually), but keep in mind that complex
> mental models have a price too (about $1.98 billion)

What performance loss? And there's nothing complex about the
distinction between pointers and nil that programmers don't already
have to be aware of.

[Sebastian Sylvan]

On Mon, Nov 16, 2009 at 10:26 PM, Jessta <jes...@gmail.com> wrote:

On 17/11/2009, Ian Lance Taylor <ia...@google.com> wrote:
> How do I say "p := new(*T)"?
>
> How does it work if the pointer is in a struct?
>
> How does it work if the pointer is in a struct which is embedded in
> another struct?
>
> How does it work if you have two structs which need to point to each
> other?
>
> Ian
>

There seems to be a bit of confusion here, the proposal is to have
non-nullable as the default with pointers being able to be declared
nullable if needed.

In Go there are a few ways a pointer can become nil,
1. a pointer is created and set to nil
2. a pointer is within a structure and hasn't yet been initialised
3. a pointer is set to the return value of a function that can return nil

The first case is unlikely to be a problem, the developer set the
pointer to nil for a good reason

The second case is more likely but isn't really helped by non-nullable
because most of the time a structure that contains a pointer is going
to need that pointer to be nullable(otherwise they would have just
embeded the thing it's pointing to in to the struct)

I don't buy this one. There are plenty of cases where you want multiple things pointing to the same value (or always pointing to *a* value, even if the specific value changes over time). In fact, without checking I'd estimate that most pointers in my classes would be non-null. If you're using C++ you can count the number of C++ style references in class headers and estimate that it's at least that many non-null pointers. In fact, most of the time when auto_ptr or shared_ptr are used as class members it's for managing life times of dynamic objects using the destructor (so it's set in the constructor, and usually not nulled out), so you can probably count a high proportion of those too. It's pretty rare IME that you have a class member where NULL is a legitimate and meaningful value.

 

The third case is prevented by the (nil,error) multiple value return
convention for functions that will return a nil. A developer that
ignores this is obviously a bad developer and nobody should use their
code anyway. 

NULL pointer access violations are hardly rare in Win32 code (which returns HRESULT and sets output using out-parameters). This argument is wishful thinking, IMO. The "that's a bad developer" argument isn't very good when you're talking about language design. If you allowed that argument you might as well stick with C and blame all its shortcomings on bad developers. Like Hoare says, it's appropriate for language designers to take responsibility for the mistakes of their users.

--
Sebastian Sylvan

[jqb]

On Nov 16, 1:10 pm, Ian Lance Taylor <i...@google.com> wrote:

Where did I say there would be no type adjective? Again, "nonnull
would be the default". The type adjective would be "nullable" or, as
was suggested at the very beginning of one of the threads on this
subject, '^', with nullable pointers being declared as, e.g., var
pointer_or_null *^ T

[jqb]

On Nov 16, 1:09 pm, Erik Wikström <Erik-wikst...@telia.com> wrote:
> On Nov 13, 9:00 pm, Sebastian Sylvan <sebastian.syl...@gmail.com>
> wrote:
>
> > On Fri, Nov 13, 2009 at 7:48 PM, Ian Lance Taylor <i...@google.com> wrote:
> > > I personally think that in a systems programming language, a pointer
> > > is a reasonable concept: it intuitively represents a memory address.
> > > I don't personally think that permitting pointers to be nil is a
> > > billion dollar mistake.  In my C/C++ programming I've never noticed
> > > that NULL pointers are a noticeable source of bugs.
>
> > A pointer is a reasonable concept, but there's no reason combine the concept
> > of a pointer with the concept of "nullable". Representing a memory address
> > is perfectly fine and useful, but from there it doesn't follow that we
> > should also allow it to point to an invalid memory address that crashes if
> > you try to use it.
>
> Do I understand you correctly that you would want to forbid NULL (or
> whatever
> you want to call it) as a value for a pointer/reference?

Please read the thread, as this has been discussed at length. It helps
if one understands the distinction between values and variables.
Pointer values are references to objects. nil is a value which is
distinct from any pointer value. A nullable pointer variable -- the
sort you are familiar with from C etc., can hold either a pointer
value or a nil value. The suggestion is to also have a nonnullable
pointer variables that can only hold pointers. Again, read the thread
for the details.

[jqb]

On Nov 16, 8:54 am, Jon Harrop <j...@ffconsultancy.com> wrote:

> On Monday 16 November 2009 15:07:44 Ian Lance Taylor wrote:
>
> > Sure, I can see an argument for a language which does not permit
> > uninitialized variables.  But Go is not that language.
>

> When are uninitialized variables useful?

When their values require complex calculations not readily expressed
in initializers. Initialization of arrays of pointers or recursive
structures come immediately to mind.

> > For most
> > types, given that there can be uninitialized variables, I think that

> > zero is a reasonable choice for the initialization value.  I think you
> > are proposing that for a variable of type *T, we set the uninitialized
> > value to be not nil but new(T).  Then we presumably prohibit assigning
> > nil to a pointer variable; perhaps we can eliminate nil entirely.  The
> > benefit is that programs can not crash due to dereferencing a nil
> > pointer.  The cost is more work allocating values and more work in the
> > GC--sometimes much more work if we have something like [1000]*T.
>
> > Is that a fair statement?
>

> There is no need to degrade performance. You can just represent the empty
> value of the optional pointer type by null at run time.

The issue is the empty value of mandatory pointer types, e.g., var
nonnull_pointer [1000]*T;
I believe in most cases, memory allocation can be avoided via simple
flow analysis. However, if not, the initializer could be a single
dummy "zero" value for the type, with minimal memory allocation and
minimal garbage to collect.

[jqb]

On Nov 14, 1:31 pm, Hans <h...@tradeflag.com> wrote:
> > We've reached the "Hoare is stupid!" level of the argument.
> > Amazing. :-D
>
> I didn't say he was stupid.

But your argument IS on that level.

> I do, however, feel confident in saying
> that to claim his mistake cost a billion dollars, he must have a very
> low opinion of all other software developers.

Perhaps he simply has a lot of experience and evidence, and isn't
guilty of the sort of fallacious thinking that substitutes "all" for
"enough to get that result".