2/15 Non-nullable prototype update

[Bob Nystrom]

Greetings! It's about time for an update on the non-nullable prototype.

Since we last talked, I implemented a bunch more of the static checking rules needed to handle nullable types:

• Report errors for non-nullable optional parameters without default values.
• Get nullable types working with least upper bounds.
• Define what methods are available on nullable types.
• Allow methods on null-aware operators on nullable types.
• Add basic type promotion for "== null" and "!= null".
• Handle nullable types on the LHS of "??".
  

I also hacked in support for "?" in the parsers for the VM and dart2js so they don't choke on those as I start annotating the core libraries. Afterwards, I realized I don't need that because I'm going to use DDC's fork of the SDK instead.

I still have some work to do around testing that generic classes work, but I think this means the basic static checking for nullable types is done. I'm sure there are lots of bugs.

I didn't implement any of the more sophisticated promotion rules since that's a bit beyond me. Instead, my plan is to just leave notes in the code where "better promotion would have helped here". The important part is to get data for how useful better promotion would be, and I think that's probably a more effective way to track that.

With that in place, I ran the checker over all of the SDK libraries. It generated a little more than 2109 errors. (I don't recall the exact number before I started fixing them.) Then I went through and fixed all of the errors in dart:core and some of the knock-on errors caused by API changes in core to other libraries.

That got it down to 1911 errors, and I learned a ton along the way.

Some informal observations:

• Like I noted years ago, most variables are not nullable. I needed to add few "?" to annotations for the things that do permit null. (I don't have exact numbers, but I can scrape that after the fact later.)
  
  
• The basic type promotion that we already have handles a lot of cases. In many places, I could see that a nullable variable was being used as a non-nullable one because it was in code that checked for null and the simple "!= null" promotion caught it.
  
  In other words, much existing code is already null safe, even when it works with nullable types.
  
  
• "??" is really handy for escaping a nullable variable:
  
      String? s1 = ...;
      var s2 = s1 ?? "default"; // Infers s2 is non-null.
  
  
• I found five breaking API changes in dart:core that were or would be needed to get it null-safe. That's not a big number (yay!) but since core is, well, core, that may still break a lot of users (boo). Aside from the subscript operator on Map, I don't think it will be too bad.
  
  
• Better inference/type promotion will go a long way towards improving the usability. With a handful of promotion features, almost all of dart:core would have been null-safe without any changes. I found eight places where I don't think inference could figure out something is non-null and a user would have to forcibly assert it to the type system.
  
  
• I found three places where it looks like the existing code might not be correctly handling null. It seems like you could get to a state where you call a method on a potentially null variable. In most cases, I think outside code ensures you can't get into that state, which is why we haven't seen it. Three isn't much, but consider that this is literally the world's most well-tested Dart code.
  
  
• Having to explicitly annotate optional parameters without default values as nullable feels tedious:
  
      method([int o])  // <-- Error.
      method([int? o]) // <-- OK.
  
  We may want to implicitly consider the parameter type nullable in that case, but that feels a little weird too. I don't like the type being something different from what you wrote. I think we'll have to do some exploration here to get to a usable syntax.

The big caveat to all of this is that it's only based on "dart:core". It may be that libraries farther up the stack are qualitatively difference in ways that affect the usability around null. I'll learn more as I work my way up.

But, overall, it's been a really promising couple of weeks. Getting "dart:core" null clean was about two days of work, while also battling a cold. That's not a lot of effort.

Cheers!

– bob

[Günter Zöchbauer]

Sounds promising \o/
Defining optional parameters without default as nullable wouldn't bother me - just my 2c.

[Filipe Morgado]

On Thursday, 16 February 2017 08:53:33 UTC, Günter Zöchbauer wrote:

Sounds promising \o/
Defining optional parameters without default as nullable wouldn't bother me - just my 2c.

+1

I would favor consistency over 1 less character.

And it would be clearer if optional parameters have a non-nullable default value.

    method([int o = 0])  // <-- OK.

    method([int? o = 0]) // <-- OK.

Just my 2c as well.

[Filipe Morgado]

Correction:

... And it would be clearer if non-nullable optional parameters have a default value ...

[Patrice Chalin]

Hi Bob. Great progress, esp. for someone battling on other fronts. Sorry that you aren't feeling well, and I hope you get rid of that cold soon!

• Having to explicitly annotate optional parameters without default values as nullable feels tedious:
  
      method([int o])  // <-- Error.
      method([int? o]) // <-- OK.
  
  We may want to implicitly consider the parameter type nullable in that case, but that feels a little weird too. I don't like the type being something different from what you wrote. I think we'll have to do some exploration here to get to a usable syntax.

I came to the conclusion that no matter which syntax and semantics one adopts for optional parameters, there is always an initial feeling of weirdness :)

That being said, consider the following declarations:

f(int i, [int j]) => ...
g(int? m, [int? n]) => ...

Under NNBD, the declaration of i allows the analyzer to report issues at points of call:

f(null) // warning: 1st argument should be non-null
g(null) // ok

I believe that the declaration of j should have the same interpretation:

f(0, null) // warning: 2nd argument should be non-null
g(0, null) // ok

Of course, from within the body of f, the type of j is nullable int. What is nice about this interpretation is that when j is null inside f, we now know that it is because no argument was provided (as opposed to the caller having supplied a value of null). Without such a nullity semantics, we cannot make that distinction.

What I've outlined above is the gist of the dual-view semantics proposed in Section E.1.1 of the DEP. More cases are covered there.

Cheers,

Patrice

[Bob Nystrom]

On Thu, Feb 16, 2017 at 9:00 AM, Patrice Chalin <pch...@gmail.com> wrote:

Hi Bob. Great progress, esp. for someone battling on other fronts. Sorry that you aren't feeling well, and I hope you get rid of that cold soon!

• Having to explicitly annotate optional parameters without default values as nullable feels tedious:
  
      method([int o])  // <-- Error.
      method([int? o]) // <-- OK.
  
  We may want to implicitly consider the parameter type nullable in that case, but that feels a little weird too. I don't like the type being something different from what you wrote. I think we'll have to do some exploration here to get to a usable syntax.

I came to the conclusion that no matter which syntax and semantics one adopts for optional parameters, there is always an initial feeling of weirdness :)

Yes, that may be true.

 

That being said, consider the following declarations:

f(int i, [int j]) => ...
g(int? m, [int? n]) => ...

Under NNBD, the declaration of i allows the analyzer to report issues at points of call:

f(null) // warning: 1st argument should be non-null
g(null) // ok

I believe that the declaration of j should have the same interpretation:

f(0, null) // warning: 2nd argument should be non-null
g(0, null) // ok

Yes, definitely. If you provide an argument for j, it should be an int.

 

Of course, from within the body of f, the type of j is nullable int. 

I think that's an anti-goal for me. If you declared it int, it's nice to be able to treat it as int internally and not have to worry about null. That does, of course, mean you need a default value to ensure it won't be null.

The way my prototype handles this is that this declaration is an error:

f([int i]) {}

If the optional parameter's type is non-nullable and you don't have a default value, it's an error just like an uninitialized variable of non-nullable type is.

What is nice about this interpretation is that when j is null inside f, we now know that it is because no argument was provided (as opposed to the caller having supplied a value of null). Without such a nullity semantics, we cannot make that distinction.

That's true, but I think I'm OK with not making being able to make that distinction. I think it causes more trouble than it's worth.

 

What I've outlined above is the gist of the dual-view semantics proposed in Section E.1.1 of the DEP. More cases are covered there.

Ah, right! I remembered that, but then didn't mention it in my update.

There is a related change that we've talked a little bit about on the language team. We have considered—I don't know how seriously yet—changing the semantics around passing null for an optional parameter. Right now, the explicit null overrides the default value. Personally, I think that's a mistake. It means you can't forward one optional parameter to another function with an optional parameter if the latter has a default value:

f([int i]) {

  g(i);

}

g([int i = 0]) {

  print(i);

}

g(); // 0

f(); // null

We've considered changing that so that an explicit null also means "use the default value, if any". That means a default value is effectively sugar for:

f([int i = 0]) {

  print(i);

}

// same as:

f([int i]) {

  i ??= 0;

}

Whereas today a default value has magic semantics that can't be expressed otherwise. Personally, in my own code, I almost never use default values and instead manually do the pattern here specifically because I don't want to distinguish between "no parameter" and "null parameter", since I find forwarding to be pretty common.

This change gets null closer to working like "undefined", which I think is probably an overall good thing.

Cheers!

– bob

[Patrice Chalin]

We've considered changing that so that an explicit null also means "use the default value, if any".

I wasn't aware that that language feature was up for review. That would be a wonderful change! (I recall folks asking for this a while back.)

With that in place (or at least the hope of that being adopted :), I would completely agree with your proposed semantics for optional parameters.

Cheers,

Patrice

[Bob Nystrom]

On Fri, Feb 17, 2017 at 6:31 AM, Patrice Chalin <pch...@gmail.com> wrote:

I wasn't aware that that language feature was up for review.

We've talked about it a bit informally, but I don't know how much the rest of the language team would be in favor of it.

Unlike when you wrote your original proposal, we are putting breaking changes on the table, so that gives us some more leeway here. Which specific breaking changes is always the hard question to answer. :)

That would be a wonderful change! (I recall folks asking for this a while back.)

With that in place (or at least the hope of that being adopted :), I would completely agree with your proposed semantics for optional parameters.

\o/

– bob

 

[Alex Tatumizer]

It seems "lateinit" is a logical necessity (however ugly):

https://kotlinlang.org/docs/reference/properties.html, lookup "lateinit"

If "lateinit" is supported, then optional parameters can be treated as "leteinit" by default (unless explicit default is provided)

[Bob Nystrom]

On Wed, Feb 22, 2017 at 9:39 AM, Alex Tatumizer <tatu...@gmail.com> wrote:

It seems "lateinit" is a logical necessity (however ugly):

https://kotlinlang.org/docs/reference/properties.html, lookup "lateinit"

I spent a little time trying to get dart_style null-clean and ran into a lot of fields that are not initialized to a value in constructors but are always non-null before they are used.

Swift's "lazy" modifier wouldn't help there because often the initialization is explicit and non-trivial. Kotlin's approach had slipped my mind. Thanks for bringing it up!

It would definitely help in the cases I saw.

 

If "lateinit" is supported, then optional parameters can be treated as "leteinit" by default (unless explicit default is provided)

You could treat all non-nullable variables as lateinit by default. That's basically what not having non-nullable types at all means. :)

I think it's better to be explicit and ask users to opt in so that they know that a runtime check and failure may occur.

If a user were to, say, remove the default value from an optional parameter, I think it would be better to give them a static error "Hey, this can be null now." instead of silently allowing that but turning later uses of that parameter into calls that could fail.

Cheers!

– bob

[Alex Tatumizer]

Consider the analogy with "final".

You can declare an instance variable like "final int x=0", and later remove "=0" (relying on constructor to initialize it) - without triggering any warnings, or need in extra markup like "lateinit"

It's the same situation.

Coming from different angle: suppose you provide a library for others to use, and people are looking at your method declarations. They don't want to see "lateinit in

foo([lateinit int x]) - because "lateinit" doesn't add any information to THEM. From library writer's viewpoint, annotation this might serve some (minimal) purpose, but not for the reader.

The story might be different when you declare a local variable though.

I think the least disruptive approach would be really treat them all as lateinit by default, but whenever compiler cannot prove that initialization really happens before use, require exclamation mark whenever the variable is accessed (exclamation mark triggers explicit runtime check for null). E.g.

int x;

... x may be initialized after declaration, but compiler cannot prove it

int y=x! // need to explicitly invoke runtime check

If you don't put exclamation mark there, compiler will complain - at every point of access.

[Brian Slesinsky]

Seems like you can either use another variable or extract a helper method?

For lazy initialization, you might have something like this:

Foo ensureFoo() {

  _foo ??= new Foo();

 return _foo;

}

_foo is nullable. (Otherwise the null-aware operator would not make sense.) But ensureFoo's return value can be inferred as not null, and you can save it in a local not-null variable, etc.

I'd be interested in hearing what initialization patterns Bob ran into in dart_style.

[Kasper Peulen]

Looking at this lateinit, I wonder if something similar could work with final. Where this would mean:

`lateinit final int i;`

That "i" can only be initiated once, to a non null value. I often have to rely on factory contructors to just get my fields final.

Using lateinit seems cleaner, and that could even work outside of the constructor right?

Met vriendelijke groeten, Kasper Peulen

--
You received this message because you are subscribed to the Google Groups "Dart Core Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to core-dev+unsubscribe@dartlang.org.

[Bob Nystrom]

On Wed, Feb 22, 2017 at 12:48 PM, Alex Tatumizer <tatu...@gmail.com> wrote:

Consider the analogy with "final".

You can declare an instance variable like "final int x=0", and later remove "=0" (relying on constructor to initialize it) - without triggering any warnings, or need in extra markup like "lateinit"

It's the same situation.

Sure, but I think that's an edge case. It's rare (and used to be an error) for a field to be initialized both at the declaration and in the constructor.

In the normal case, you can't remove the one initializer for the field and have that silently change the runtime behavior. It just produces a static error.

Coming from different angle: suppose you provide a library for others to use, and people are looking at your method declarations. They don't want to see "lateinit in

foo([lateinit int x]) - because "lateinit" doesn't add any information to THEM. From library writer's viewpoint, annotation this might serve some (minimal) purpose, but not for the reader.

True, though there are other cases where stuff in the parameter declaration is relevant to the library writer, but not the caller. For example, "this." in constructor parameters is an implementation detail and not part of the API itself.

I think the least disruptive approach would be really treat them all as lateinit by default, but whenever compiler cannot prove that initialization really happens before use, require exclamation mark whenever the variable is accessed (exclamation mark triggers explicit runtime check for null). E.g.

int x;

... x may be initialized after declaration, but compiler cannot prove it

int y=x! // need to explicitly invoke runtime check

If you don't put exclamation mark there, compiler will complain - at every point of access.

Yes, we'll probably have some syntax like this too where you can forcibly assert that you think something should not be null. I'm just trying to figure out if there are patterns that are so common that doing that assertion feels tedious and where it would make sense to have something more baked in.

Better type promotion covers a lot of those cases around local variables, but doesn't help with fields. If we had sealed fields, we could extend that to final sealed fields easily. But for mutable fields, static analysis can't really help you. Something like lateinit would help there (with the understanding that it does add runtime checking).

Cheers!

– bob

[Bob Nystrom]

On Wed, Feb 22, 2017 at 1:13 PM, 'Brian Slesinsky' via Dart Core Development <core...@dartlang.org> wrote:

int x;

... x may be initialized after declaration, but compiler cannot prove it

int y=x! // need to explicitly invoke runtime check

If you don't put exclamation mark there, compiler will complain - at every point of access.

Seems like you can either use another variable or extract a helper method?

For lazy initialization, you might have something like this:

Foo ensureFoo() {

  _foo ??= new Foo();

 return _foo;

}

That doesn't work exactly because you'd get an error returning _foo, which has type Foo? from a method declared to return Foo. But this would, I think:

Foo ensureFoo() {

  _foo ??= new Foo();

 return _foo ??= new Foo();

}

Or even:

Foo? _foo;

Foo get foo => _foo ??= new Foo();

There are definitely patterns you can apply. The question is whether the need is so common that its worth adding language support so you don't need to.

 

I'd be interested in hearing what initialization patterns Bob ran into in dart_style.

I only poked at it a little bit, so I don't have a good summary. Once I get some other libraries converted, I'll try to take another stab at it. In the meantime, you can always try out the branch yourself. You just need to:

1. pull down the repo
   
2. check out the branch
   
3. build the SDK
   
4. point the Dart IntelliJ plug-in add your built SDK 
   
5. open dart_style in IntelliJ

(You can of course just run the command line analyzer on it too, but I find seeing it in the IDE is nicer.)

Cheers!

– bob

[Bob Nystrom]

On Wed, Feb 22, 2017 at 1:23 PM, Kasper Peulen <kasper...@gmail.com> wrote:

Looking at this lateinit, I wonder if something similar could work with final. Where this would mean:

`lateinit final int i;`

That "i" can only be initiated once, to a non null value. I often have to rely on factory contructors to just get my fields final.

Using lateinit seems cleaner, and that could even work outside of the constructor right?

It could, yes, but it implies that every access of the field has a runtime check to see if it was initialized first, so there's a perf cost.

– bob

[Alex Tatumizer]

> ... but it implies that every access of the field has a runtime check to see if it was initialized first, so there's a perf cost.

I think this cost is negligible. The branch will always be predicted correctly, so we are talking about 1 cycle, and even that is in a pipeline (so it's a fraction of the cycle).

There's also a way to do it in hardware - e.g. use "test addr" instruction, and set things up so that null corresponds to non-existing addr. This will save a bit of space in instruction cache and branch prediction cache.

(Though it depends on OS and H/W architecture, so the trick might, or might not, be feasible).

Dart has much more freedom to do tricks on a low level than Kotlin (b/c JVM).

Runtime checks are necessary anyway if you want consistent behavior on lists of non-null values: compiler will have a hard time figuring out (statically) what is initialized and what isn't.

[Leaf Petersen]

On Wed, Feb 22, 2017 at 7:09 PM, Alex Tatumizer <tatu...@gmail.com> wrote:

> ... but it implies that every access of the field has a runtime check to see if it was initialized first, so there's a perf cost.

I think this cost is negligible. The branch will always be predicted correctly, so we are talking about 1 cycle, and even that is in a pipeline (so it's a fraction of the cycle).

This is definitely not true when compiling to javascript, where much more expensive tests are needed to preserve Dart semantics.

Even when compiling to native this claim is suspect.  Unless you use implicit null check tricks (rely on OS trap handling to handle the null de-reference) you generally have a code size penalty if nothing else.  More importantly though, the presence of null checks complicates things for the optimizer, and hence for any fixed optimization budget reduces its effectiveness.  Code motion, CSE, constant folding, register allocation, SIMD vectorization, inlining, and really almost every other optimization suffer from the additional control flow paths and the non-local control flow that happens on the exceptional branch.  

-leaf

 

There's also a way to do it in hardware - e.g. use "test addr" instruction, and set things up so that null corresponds to non-existing addr. This will save a bit of space in instruction cache and branch prediction cache.

(Though it depends on OS and H/W architecture, so the trick might, or might not, be feasible).

Dart has much more freedom to do tricks on a low level than Kotlin (b/c JVM).

Runtime checks are necessary anyway if you want consistent behavior on lists of non-null values: compiler will have a hard time figuring out (statically) what is initialized and what isn't.

--

[Lasse R.H. Nielsen]

If the compiler can determine that a variable is initialized before use on all paths, it can drop initialization checks.

The question is then whether we want to make programs where that is not the case into errors. That would require the spec to specify some sort of flow analysis to detect "definitely initialized variables" (only matters for non-nullable ones without initializers).

Having to have null checks is effectively what we have now, so it won't be worse than that (well, maybe, if we expect a different error to be thrown, but let's just assume we don't).

That also means that a lazily initialized non-nullable variable is effectively a nullable variable, just one where you are not allowed to assign null to it, and where it throws on read if it's null, instead of on use. I'm not sure I want that extra kind of variable.

Instead I'd probably prefer to just require that the author makes the variable nullable, because it is. The fact that it has some life-time where it isn't null, and that overlaps with the code where you actually use it, is not specific to variable initialization. It can happen for any nullable variable that you ensure its non-null-ness, and then use it as non-nullable.

E.g.:

if (x != null) {

   ...

}

or 

assert(x != null); 

x as Foo;  // non-null Foo.

...

Both of these should eventually type-promote x to non-nullable (I hope).

The other option is to require initialization. That's easy for primitive values, because you can always initialize to 0 or false or "", but for more complex classes, you'll effectively have to have a null-object for the type. That's likely as big an overhead for the user as allowing null.

I assume this is mainly for instance variables or global variables.

For local variables, it's easy to just declare a new variable:

  var notNullX = x ?? notNullValue;

or just type promote with a single check.

  notNullValue as NotNullType;

I'm sure there are cases with loops and distinct initialization in if-branches that makes that problematic, but you can just make it nullable if the variable really is null.

For instance and global variables, it is extremely hard to detect that a variable is initialized before use.

For instance variables, the one chance is to see that it leaves the constructor as non-null.

The workaround I would use there is to not initialize it in the body, and instead use two constructors, the initialing one and the one that does dependent computations.

(If there isn't two fields that need values depending on the same computation, you can just do: 

  Foo(x, y) : z = _compute(x, y) 

The problem is when you need to do something like

  Bar() : _x = new Completer, y = _x.future;

There you need an extra constructor: 

  Bar() : this._(new Completer()); Bar._(this._x) : y = _x.future;

(That's the solution to needing local variables in initializer fields).

I don't think I'd like the user to have to write something at the declaration to allow a partially non-null variable. Even if it's as simple as `int! x; // lazily non-null`.

/L

--

Lasse R.H. Nielsen - l...@google.com  

'Faith without judgement merely degrades the spirit divine'
Google Denmark ApS - Frederiksborggade 20B, 1 sal - 1360 København K - Denmark - CVR nr. 28 86 69 84

[Bob Nystrom]

On Wed, Feb 22, 2017 at 11:01 PM, 'Lasse R.H. Nielsen' via Dart Core Development <core...@dartlang.org> wrote:

If the compiler can determine that a variable is initialized before use on all paths, it can drop initialization checks.

The question is then whether we want to make programs where that is not the case into errors.

This is, I believe, the real key usability question.

If there are cases where we think the usable solution is to add runtime checks, I'm OK with that, but I want those cases to be very clear to users because those checks may fail. I want Dart users to have a very clear mental model of which parts of their code they need to pay attention to because they may fail at runtime.

That would require the spec to specify some sort of flow analysis to detect "definitely initialized variables" (only matters for non-nullable ones without initializers).

Yup. My impression from the prototype is that this would be really helpful. (It would also be helpful for strong mode in general, though there you can work around it by giving an explicit type annotation.)

 

That also means that a lazily initialized non-nullable variable is effectively a nullable variable, just one where you are not allowed to assign null to it, and where it throws on read if it's null, instead of on use. I'm not sure I want that extra kind of variable.

I'm not sure either, but it does seem to come up pretty frequently in code that I've looked at in the prototype. Also, Kotlin and Swift both have something roughly analogous, so they probably know what they're doing.

 

Instead I'd probably prefer to just require that the author makes the variable nullable, because it is. The fact that it has some life-time where it isn't null, and that overlaps with the code where you actually use it, is not specific to variable initialization. It can happen for any nullable variable that you ensure its non-null-ness, and then use it as non-nullable.

Yes, you can always live without this feature using various patterns. I think the question is whether the patterns are so common it's worth giving them direct language support.

 

E.g.:

if (x != null) {

   ...

}

or 

assert(x != null); 

x as Foo;  // non-null Foo.

...

Both of these should eventually type-promote x to non-nullable (I hope).

assert() is tricky because users expect assertions to have no runtime cost in a production build. But "as Foo" and "is Foo", and "!= null" should all definitely promote local variables. Promoting fields is a lot harder. There's no simple static analysis to tell if a field won't be mutated by some other method call between when it's tested and when it's used.

We could do simple analysis for final non-overridable fields. That covers a decent number of use cases (though not all) in code I've seen. But Dart currently has no way to express non-overridable fields. :(

 

The other option is to require initialization. That's easy for primitive values, because you can always initialize to 0 or false or "", but for more complex classes, you'll effectively have to have a null-object for the type. That's likely as big an overhead for the user as allowing null.

That's not a good solution in most cases. Even when you can come up with some easy dummy value to initialize it with, that's effectively just another kind of null unless that value is actually useful and correct.

I assume this is mainly for instance variables or global variables.

For local variables, it's easy to just declare a new variable:

  var notNullX = x ?? notNullValue;

or just type promote with a single check.

  notNullValue as NotNullType;

Yup. Using "as" feels verbose in some cases where the type name is long, but otherwise this works.

 

I'm sure there are cases with loops and distinct initialization in if-branches that makes that problematic, but you can just make it nullable if the variable really is null.

There are, but I haven't seen too many of them. I do think handling joins after if-branches would help. Stuff like:

foo(int? i) {

  if (blah) {

    i = 1;

  } else {

    if (i == null) throw "WAT";

  }

  i.length; // <-- OK, promoted here.

}

If you look in the prototype branch for "TODO(nnbd-flow)", you'll find a lot of cases like this. Loops are much less common (at least in the core libraries I've looked at so far).

 

For instance and global variables, it is extremely hard to detect that a variable is initialized before use.

For instance variables, the one chance is to see that it leaves the constructor as non-null.

Constructor initialization list or constructor body? Either way, even if the field is definitely non-null at exit, you can't assume later accesses to it won't return null. It's implicit getter could be overridden by a subclass. :(

The workaround I would use there is to not initialize it in the body, and instead use two constructors, the initialing one and the one that does dependent computations.

Yup, that works, though it feels tedious sometimes.

 

(If there isn't two fields that need values depending on the same computation, you can just do: 

  Foo(x, y) : z = _compute(x, y) 

The problem is when you need to do something like

  Bar() : _x = new Completer, y = _x.future;

There you need an extra constructor: 

  Bar() : this._(new Completer()); Bar._(this._x) : y = _x.future;

(That's the solution to needing local variables in initializer fields).

Really, you end up wanting access to the whole Dart language inside constructor initialization lists. (Witness: Flutter wanting assert() in there.)

If we were OK with definite assignment analysis, we could use that to allow you to initialize final fields inside constructor bodies and just require that all fields are definitely assigned before any implicit or explicit use of "this". Then you wouldn't need constructor initialization lists at all (modulo whatever shenanigans are needed for const constructors).

 

I don't think I'd like the user to have to write something at the declaration to allow a partially non-null variable. Even if it's as simple as `int! x; // lazily non-null`.

Yeah, I'm not crazy about it either, but it does seem to come up a bunch. You may wish to check out the nnbd prototype branch and try getting some code null-clean yourself to get a feel for it.

Cheers!

– bob

[Patrice Chalin]

On Monday, February 27, 2017 at 12:48:00 PM UTC-8, rnystrom wrote:

...

 

I don't think I'd like the user to have to write something at the declaration to allow a partially non-null variable. Even if it's as simple as `int! x; // lazily non-null`.

Yeah, I'm not crazy about it either, but it does seem to come up a bunch. You may wish to check out the nnbd prototype branch and try getting some code null-clean yourself to get a feel for it.

In case you're interested in some numbers (from the Java world) to qualify "it does seem to come up a bunch" ... in Appending I of DEP-30, we reference an empirical study of 700MLOC of Java code (Chalin et al., 2008): almost 60% of "nullable" declarations were monotonic non-null (i.e., eventually/lazily non-null) -- see Table 4 on page 25 of the cited paper. Of course, that was for Java, but I would not be surprised if the numbers were similar for Dart.

Cheers,

Patrice

Cheers!

– bob

[Bob Nystrom]

On Mon, Feb 27, 2017 at 1:03 PM, Patrice Chalin <pch...@gmail.com> wrote:

In case you're interested in some numbers (from the Java world) to qualify "it does seem to come up a bunch" ...

I am!

 

in Appending I of DEP-30, we reference an empirical study of 700MLOC of Java code (Chalin et al., 2008): almost 60% of "nullable" declarations were monotonic non-null (i.e., eventually/lazily non-null) -- see Table 4 on page 25 of the cited paper. Of course, that was for Java, but I would not be surprised if the numbers were similar for Dart.

Very useful, thanks!

One difference we'd have to deal with is that fields are virtual in Dart and there is no way currently to opt out of that. So even a monotonically non-null field can't be promoted after being checked for null. You could be in some subclass that overrides the field's getter to return null at some later point.

– bob

 

[Erik Ernst]

Indeed, it's an extremely interesting data point that so many variables are eventually non-null!

I think we have a quite attractive approach available here: "all hybrid" (explanation below).

We generally have the option to go a few ways: (1) An invariant is implied by a program element (e.g., a declaration modifier) and it is statically known to be established initially and maintained at each step, and (2) an invariant is implied by a program element, and it is checked dynamically at each point where the program semantics relies on it. (3) We can have a hybrid form.

In the domain of nnbd, as an example of a hybrid form, we have had the following on the table for a while: A variable `C x;` may be initialized to null (even though `C` is a non-null type), but it can only be assigned statically non-null values, and usages are either dominated by assignments, checked dynamically, or possibly prevented statically (because `x` is definitely not yet initialized).

That hybrid form will fit in smoothly with a traditional static approach: `C x = e;` would require `e` to have a non-null type (`C` is a again assumed to be a non-null type), and every assignment must have a statically non-null type. Of course, this form works well as `var x = e;` as well, where `C` is inferred.

The important point here is that we don't even have to make the distinction!

The latter (a statically enforced non-null variable) is a special case of the hybrid form, because all usages of the statically managed variable are dominated by some initialization/assignment with an expression whose value is statically known to be non-null (namely the initializing expression).

As usual, it's trivially possible to outlaw the hybrid form (say, using a lint), such that developers/organizations who want to be more strict can do that.

The rest of the world can continue to use the variables which are eventually-non-null-in-practice with a small effort to port them to nnbd.

So for the last issue, what happens if `x` is a field and there is an overriding setter (incl. field-overrides-field)? We could have `C? x;` overridden by `covariant C x;`, in which case the setter takes a covariant argument and assignments (in the superclass) of null would raise a run-time error. The other direction is not relevant, because we do not allow contravariant overrides on the return type (here: of the getter), so we cannot have a variable with a non-null type which is overridden in any way to allow null.

The hybrid model relies on static properties when available (if we know statically that a given variable `x` has been assigned a non-null value there will be no null-check in `x + 1`), so we need to consider when we know statically that a given field is non-null. That may be difficult to establish because any computational step which might invoke user-written code could potentially change the value of the field, but here we also benefit from the discipline in the hybrid model: As soon as we have established for a statically known receiver (say, `this` or `final B o`) that its field `x` is non-null, the static and dynamic checks will prevent it from becoming null again. For example, we will not have any problems avoiding the dynamic checks in situations like this one:

class B {

  C x; // Initially null, but assignments cannot make it null.

  foo() {

    if (x != null) {

      ... // Do whatever you want.

      x.baz(); // `x` is definitely still non-null.

    }

  }

  bar() {

    x.baz(); // Subject to a dynamic null check. The linter could flag it.

    x.qux(); // `x` is definitely still non-null.

  }

}

So, as far as I can see, the hybrid model would work very nicely for fields as well, even though there would be locations where we need an implicit dynamic null check, basically "when we start using that field". Developers who want to get rid of dynamic null checks would ask for stronger checking and make them explicit (or refactor in some other way to eliminate those checks).

 

– bob

 

--
You received this message because you are subscribed to the Google Groups "Dart Core Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to core-dev+unsubscribe@dartlang.org.

--

Erik Ernst  -  Google Danmark ApS

Skt Petri Passage 5, 2 sal, 1165 København K, Denmark

CVR no. 28866984

[Patrice Chalin]

SGTM. In fact, we did somewhat similar analysis in JML (under the assumption that a program was single-threaded).

A bit of trivia for language buffs (because this isn't really relevant to Dart). In the context of multithreaded Java programs, you need to make a local copy of a field's value when you want to use the value after the nullity test.  Interestingly, Eiffel introduced special if-statement syntax for such cases (combining a local variable declaration and an if-condition -- i.e. Fig. 7 is a Java for the Eiffel in Fig. 8: 

Cheers,

Patrice

– bob

 

To unsubscribe from this group and stop receiving emails from it, send an email to core-dev+u...@dartlang.org.

[Alex Tatumizer]

To put it succinctly, a major defect of declaration like "int? x" is that it makes it OK to assign null to x.

Which is rarely the case (I am skeptical about 40% cited above - I'd think it should be less than that).

That''s why "lateinit" is necessary. But the word "lateinit" is 1) ugly 2) too long for a case that occurs most of the time (compared with "?").

 

[Brian Slesinsky]

It seems like rather than complicating the language, it would be easier to change how you write Dart code a bit? For final fields, we write factory constructors and it seems to work out fine. Similarly, if a variable starts out as nullable and later becomes not null, how about using two variables or extracting a helper function?

--

You received this message because you are subscribed to the Google Groups "Dart Core Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to core-dev+unsubscribe@dartlang.org.

[Erik Ernst]

On Tue, Feb 28, 2017 at 6:27 PM, Patrice Chalin <pch...@gmail.com> wrote:

SGTM. In fact, we did somewhat similar analysis in JML (under the assumption that a program was single-threaded).

A bit of trivia for language buffs (because this isn't really relevant to Dart). In the context of multithreaded Java programs, you need to make a local copy of a field's value when you want to use the value after the nullity test.  Interestingly, Eiffel introduced special if-statement syntax for such cases (combining a local variable declaration and an if-condition -- i.e. Fig. 7 is a Java for the Eiffel in Fig. 8: 

Interesting! In a similar vein, we have discussed the following syntax several times (it's not necessarily going to happen, but we are aware of this idea):

if (e is T x) {

  ... // `x` is in scope, with type `T`, final, initialized to the value of e.

}

// But `is` is a boolean which may not fit, so maybe we want an `as` variant?

e as T y; // .. very similar to `T y = e`, but shows intent to re-type `e`.

// And then the nnbd stuff. Maybe we could do this?

if (e is! Null x) {

  ... // `x` in scope, final, with typeof(e)-but-non-null, init. to `e`.

}

To unsubscribe from this group and stop receiving emails from it, send an email to core-dev+unsubscribe@dartlang.org.

[Erik Ernst]

The hybrid model I proposed was very similar to `lateinit` except that it doesn't use a keyword, it just combines the lack of initialization and a non-null type to enable the lateinit semantics.

The point is that you'd be able to use `int x;` rather than `int? x`, and then the variable will be treated as non-null everywhere in its scope (which enables member accesses, e.g., `x.abs()`), but with dynamic checks whenever needed. 

The downside is that the dynamic checks will take time and may fail. But since this is all statically known it is trivially possible for tools to flag the situation, if a certain organization or developer prefers to use the more verbose variants which are statically safe.

--
You received this message because you are subscribed to the Google Groups "Dart Core Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to core-dev+unsubscribe@dartlang.org.

[Alex Tatumizer]

Suppose, just for the sake of argument, that runtime check costs zero.

Then rules are simple:

for variable declared as "int x" (uninitialized at declaration) compiler inserts:

runtime check on access

runtime check on assignment

for variable declared as "int? x" compiler inserts runtime check on access, but not on assignment.

There's no "third case", everything is taken care of.

(Then  question remains under what conditions compiler generates a warning, but it's a different question).

So it all boils down to whether runtime check is cheap or not. This can be determined only experimentally. My bet it's extremely cheap,

except when compiling to javascript. I'm not sure compilation to javascript should affect design choices at all, given the circumstances.

[Bob Nystrom]

On Wed, Mar 1, 2017 at 1:59 AM, Erik Ernst <eer...@google.com> wrote:

On Tue, Feb 28, 2017 at 6:27 PM, Patrice Chalin <pch...@gmail.com> wrote:

SGTM. In fact, we did somewhat similar analysis in JML (under the assumption that a program was single-threaded).

A bit of trivia for language buffs (because this isn't really relevant to Dart). In the context of multithreaded Java programs, you need to make a local copy of a field's value when you want to use the value after the nullity test.  Interestingly, Eiffel introduced special if-statement syntax for such cases (combining a local variable declaration and an if-condition -- i.e. Fig. 7 is a Java for the Eiffel in Fig. 8: 

Interesting! In a similar vein, we have discussed the following syntax several times (it's not necessarily going to happen, but we are aware of this idea):

if (e is T x) {

  ... // `x` is in scope, with type `T`, final, initialized to the value of e.

}

// But `is` is a boolean which may not fit, so maybe we want an `as` variant?

e as T y; // .. very similar to `T y = e`, but shows intent to re-type `e`.

// And then the nnbd stuff. Maybe we could do this?

if (e is! Null x) {

  ... // `x` in scope, final, with typeof(e)-but-non-null, init. to `e`.

Yup. For the record, the syntax is borrowed from C#.

Cheers!

– bob

[Bob Nystrom]

On Wed, Mar 1, 2017 at 7:33 AM, Alex Tatumizer <tatu...@gmail.com> wrote:

(Then  question remains under what conditions compiler generates a warning, but it's a different question).

This is the question I'm most interested in. The performance of a successful null check is not a huge concern for me. What I care about is which behaviors are static errors and which cases are runtime errors. It needs to be crystal clear for users where their code may be statically silent and can fail at runtime.

Cheers!

– bob

[Alex Tatumizer]

> This is the question I'm most interested in

There's not much freedom in how this can be done. Whenever compiler can't prove that non-nullable variable is initialized, there's a warning.

What's really interesting is what you expect user to do in this case.

If I'm not mistaken, Kotlin requires in this case either to annotate it as lateinit, or as nullable.

Both variants are bad IMO, and this is the whole issue.

I think you can use exclamation mark at the point of warning.

e.g. instead of "x+1" write "x!+1".

In Kotlin, you can't use exclamation mark for non-nullable, there's a whole ritual - first declare it nullable or lateinit, and THEN use exclamation mark.

Which is absurd..