Prototyping non-nullable types

[Bob Nystrom]

Greetings, Dartisans!

I've started hacking on a prototype implementation of non-nullable types in the Dart analyzer. I'll stress very heavily that this is a prototype. Code is nasty. All kinds of tests are breaking. There are sharp corners that will cut you.

It's based on Patrice's NNBD DEP. My goals for the prototype are to get some empirical data on the usability of making types non-nullable by default. We know it's technically possible to add them. The question is whether it makes Dart a more productive, enjoyable language to have them. (Non-nullable types may also be important for performance, but I am not investigating that now.)

My rough plan is:

1. Get analyzer treating types as non-nullable by default.
   
2. Get it reporting errors on uninitialized non-nullable state.
   
3. Enable "?" to define nullable types.
   
4. Do the right type relations between nullable and non-nullable types.
   
5. Report errors when downcasting from a nullable to a non-nullable type.

I think I have these working now, though I'm certain there are bugs and missing things, especially around function types and generics. There's some more core type system-y stuff to bang out:

• Handle nullable types as bounds in generic classes ("class Foo<T extends Bar?>").
  
• Add null-checking support to the type promotion rules.
  
• Incorporate nullable types into least upper bound calculations.
  

At that point, non-nullable types "work". Now, running analyzer on any codebase (including the SDK itself) should spew out errors where the code isn't correctly null safe. The next step is seeing what it feels like to get some code null safe. Next steps are:

• Start going through the core libraries and adding "?" for things that should be nullable.
  
• Start fixing errors by checking for null or handling it in some way.
  
• Assuming we can get enough core libs null-safe, move up a level and start converting some packages, frameworks, and application code.
  

Based on how painful that is, start looking at other language features to make nullable types more pleasant to work with. Things potentially like:

• A null assertion operator.
  
• A not-null type parameter modifier (C.3.3 in Patrice's DEP).
  
• Definite assignment analysis for uninitialized variables.
  

My hope is that by the end of this exercise, we'll have a pretty good feel for questions like:

• Should we treat Object as nullable or add a new non-nullable top type?
  
• How much work is it to get some code to be null-safe?
  
• How often does null safety find bugs?
  
• Which additional features do we need to improve the usability?
  
• How much work is it to get Angular and Flutter on this?
• What should we do around mocking?
• What are the tasks we need to complete to ship a real implementation?
  

I'll try to send out status updates periodically like this to let you know how its going. In the meantime, I encourage you to take a look at the branch and tell me what you think about the prototype or the proposal.

Cheers!

– bob

[Bob Nystrom]

You know what would probably help people understand what's going on here? An updated proposal.

Here you go: https://github.com/dart-lang/sdk/pull/28619

As you can see, almost all of the details line up with Patrice's DEP, but it's helpful for me to have a doc I can update as we go along and pin down answers to various questions.

Cheers!

– bob

[Natalie Weizenbaum]

Bob, would it be useful for community members to help you non-nullify the core libraries? Is there a repo where they can contribute to that effort?

--
You received this message because you are subscribed to the Google Groups "Dart Core Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to core-dev+unsubscribe@dartlang.org.

[Matthew Butler]

+1

To unsubscribe from this group and stop receiving emails from it, send an email to core-dev+u...@dartlang.org.

[Patrice Chalin]

Hi Bob et all,

Great news, that this is taking flight! Let me know if you think that it might be helpful to make updates to DEP30, as the report was quite extensive in exploring alternatives.

Having migrated other languages to NNBD before, I'd very strongly recommend doing so only with explicit support for G.2 Migration aids such as:

lexically scoped, non-inherited library, part and class level annotations [such as]: @nullable_by_default [or] @non_null_by_default.

If tools have only a single all-or-nothing top-level NNBD switch, that will result in an explosion of warnings that will make it difficult to work on adding nullity annotations to any project (including the core libraries).

For your current implementation of NNBD in the analyzer, which type hierarchy are you using --- e.g., a new root (B.2.1), or making Null a root (B.4.7), or something else?

Also, I noticed the following in the SDK Changlog for 1.22.0:

The Null type has been moved to the bottom of the type hierarchy. As such, it is considered a subtype of every other type.

Such a decision seems to be counter to supporting NNBD. How do you think it will impact work on the analyzer?

Cheers,

Patrice

[Leaf Petersen]

The Null type has been moved to the bottom of the type hierarchy. As such, it is considered a subtype of every other type.

Such a decision seems to be counter to supporting NNBD. How do you think it will impact work on the analyzer?

We spent a reasonable amount of time discussing the interactions of various strategies for making some form of bottom user visible, and how it would interact with a migration to NNBD.  The plan is that Null has been moved to the bottom of the existing Dart type hierarchy, and stands for the type which is inhabited only by the null object.  There is a true bottom which is uninhabited, but which users have no way of writing.  If we move to NNBD, then Null remains the bottom of the nullable hierarchy (by definition, if you like to think of it that way: T? == T | Null).  But it will no longer be the bottom of the entire type hierarchy, since it will not be below non-nullable types.  True bottom (perhaps call it "Nothing") would be below everything.  We may or may not wish to make true bottom a user visible type.  It is nice to have as a way to describe the type of computations which do not return (e.g. throw, or functions which always throw, etc), but is also added complexity.

I think, in the terminology of your DEP, this corresponds more closely to the idea of adding a new root (Anything) with Null and TrueObject as subtypes, where TrueObject is the non-nullable Object type.  I think one of the questions Bob wants to resolve is whether there is a need to expose Anything and/or TrueObject as user facing types, or whether defining the user facing Object type to be "Null | TrueObject" is sufficient.

Does that make sense? 

cheers,

-leaf

 

--
You received this message because you are subscribed to the Google Groups "Dart Core Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to core-dev+unsubscribe@dartlang.org.

[Bob Nystrom]

On Thu, Feb 2, 2017 at 5:35 PM, Natalie Weizenbaum <nw...@google.com> wrote:

Bob, would it be useful for community members to help you non-nullify the core libraries?

Probably not yet.

Keep in mind that the goal of the prototype is for the language team to get a better understanding of how much work it takes to non-nullify stuff. To really internalize that, I think we need to do a lot of that work ourselves. At least personally, that's what I'd prefer. I want a real hands-on feel for what non-nullable types are like to worth with.

Assuming the prototype goes well and we move to the real conversion of all of the world's Dart code to be null-safe, absolutely, it will be all hands on deck then. :)

 

Is there a repo where they can contribute to that effort?

The prototype work is happening on this branch: https://github.com/dart-lang/sdk/tree/prototype-nnbd

Cheers!

– bob

[Bob Nystrom]

On Thu, Feb 2, 2017 at 7:12 PM, Patrice Chalin <pch...@gmail.com> wrote:

Hi Bob et all,

Patrice! I was planning to email you this morning. :)

 

Great news, that this is taking flight! Let me know if you think that it might be helpful to make updates to DEP30, as the report was quite extensive in exploring alternatives.

That's one of the things I wanted to talk to you about. I wrote up a proposal here: https://github.com/dart-lang/sdk/pull/28619

That was mostly an intellectual exercise for me. I learn a lot by writing about something so hashing out that proposal forced me to think through a bunch of cases. It's also a little easier for me to evolve since it lives in the SDK repo and is just a plain Markdown file.

Having said that, your DEP is certainly the originator of all this and has a lot more detail. Do you have any thoughts on where it makes the most since to collaborate and what should be the single source of truth?

Having migrated other languages to NNBD before, I'd very strongly recommend doing so only with explicit support for G.2 Migration aids such as:

lexically scoped, non-inherited library, part and class level annotations [such as]: @nullable_by_default [or] @non_null_by_default.

If tools have only a single all-or-nothing top-level NNBD switch, that will result in an explosion of warnings that will make it difficult to work on adding nullity annotations to any project (including the core libraries).

Good point! Yes, right now the prototype is definitely in "explosion of warnings" mode, by design. I'm trying to make sure I don't miss reporting any warnings and hopefully have them all implemented. Then I'll start trying to go through and fix them.

For the prototype, I wasn't planning to add in support for disabling them on a per library/file/class basis. Instead, I figured I would just use diffs on that generated file in order to not get overwhelmed by the sea of errors.

If that doesn't work out, your suggestion is a good one. 

For the real implementation, yes, this is definitely something I see us wanting to tool. I made a note of it on the tracking bug so we don't forget.

For your current implementation of NNBD in the analyzer, which type hierarchy are you using --- e.g., a new root (B.2.1), or making Null a root (B.4.7), or something else?

This is one of the questions I'm hoping the prototype will help us answer confidently. Right now, I'm making as few changes as possible, so I am keeping Object as the root of the hierarchy because that's how it is today. See "Object is always nullable" here. This means there is no way to express "Any Object but not null."

If our goal is simply to statically ensure there are NoSuchMethod errors because of null, that is sufficient. Null supports all of the methods Object declares so if your code works with any Object, it really should be able to work with null as well.

However, we may find that users want to enforce a stronger contract and allow any kind of object yet still prohibit null. In that case, we can introduce a new top type like you suggest. My personal hunch is we won't need it, but I don't take a put a lot of stock in hunches. That's why we're prototyping. :) Real usability data > opinions and guesswork.

Also, I noticed the following in the SDK Changlog for 1.22.0:

The Null type has been moved to the bottom of the type hierarchy. As such, it is considered a subtype of every other type.

Such a decision seems to be counter to supporting NNBD. How do you think it will impact work on the analyzer?

Like Leaf said, this is a known thing. Moving Null to the bottom solves an immediate problem where Flutter wanted to use a Null type parameter to work as a bottom-ish type.

When we add non-nullable types, we may also add a new true bottom as well. (My hunch is we'll want it.)

Let me know if there's anything else I can do to coordinate better with you. I'm trying to move quickly and get hacking on stuff, but I don't want to step on any toes or waste time rediscovering something you already know.

Cheers!

– bob

[Patrice Chalin]

Hi Leaf,

Sure, it makes sense. The semantics had special rules for `null` while avoiding special cases for `Null`. Under the new 1.22 semantics, things are slightly more uniform (i.e., no need for special rules for `null` since `Null` now simply maps to \bottom).

I'm curious why the Flutter team couldn't have used:

const empty = const <dynamic>[]; // i.e. == const [];

instead of const <Null>[] (though I imagine that it maybe have been outside of the Flutter team's control).

Cheers,

Patrice 

[Bob Nystrom]

On Mon, Feb 6, 2017 at 3:19 PM, Patrice Chalin <pch...@gmail.com> wrote:

I'm curious why the Flutter team couldn't have used:

const empty = const <dynamic>[]; // i.e. == const [];

instead of const <Null>[] (though I imagine that it maybe have been outside of the Flutter team's control).

Flutter's case was slightly more elaborate. They had classes like:

class PopupMenuEntry<T> {

  T get value => null;

  ...

};

class PopupMenuDivider implements PopupMenuEntry<???> {

  ...

}

Then they wanted to be able to do:

<PopupMenuEntry<LeaveBehindDemoAction>>[

  ...

  new PopupMenuDivider(),

  ...

]

Using dynamic for "???" wouldn't work in strong mode. We don't allow a generic type with type argument dynamic to be assigned to places where other type arguments are expected since that breaks soundness. (Leaf can probably word that better than I can. :( )

Using Null here works, once we treat Null like a bottom type.

Cheers!

– bob

[Patrice Chalin]

On Friday, February 3, 2017 at 9:36:18 AM UTC-8, rnystrom wrote:

... I wrote up a proposal here: https://github.com/dart-lang/sdk/pull/28619

That was mostly an intellectual exercise for me. I learn a lot by writing about something so hashing out that proposal forced me to think through a bunch of cases. It's also a little easier for me to evolve since it lives in the SDK repo and is just a plain Markdown file.

I agree that it makes sense to keep such a planning/todo-list-like document as separate file that is more easily modifiable.

 

For the prototype, I wasn't planning to add in support for disabling them on a per library/file/class basis. Instead, I figured I would just use diffs on that generated file in order to not get overwhelmed by the sea of errors.

Sure, keeping things as simple as possible for the prototype makes sense.

...

For the real implementation, yes, this is definitely something I see us wanting to tool. I made a note of it on the tracking bug so we don't forget.

Good.

For your current implementation of NNBD in the analyzer, which type hierarchy are you using --- e.g., a new root (B.2.1), or making Null a root (B.4.7), or something else?

This is one of the questions I'm hoping the prototype will help us answer confidently. Right now, I'm making as few changes as possible, so I am keeping Object as the root of the hierarchy because that's how it is today. See "Object is always nullable" here. This means there is no way to express "Any Object but not null."

In the prototype that I had developed (in which I was also trying to make as few changes as possible), I let `Object` be the root of the non-nullable type hierarchy, and used `Object?` to represent a possibly null object. This is achieved by making Null a root (B.4.7) --- and hence the type hierarchy becomes a "forest", rather than a single tree ... or, if we insist on having a single tree, we can elect the (denotation of the type expression) `Object?` as the new root.

... Real usability data > opinions and guesswork.

+1

Let me know if there's anything else I can do to coordinate better with you. I'm trying to move quickly and get hacking on stuff, but I don't want to step on any toes or waste time rediscovering something you already know.

I'm open to suggestions. I think that it would be great to keep the DEP-30 document up-to-date so as to reflect our latest ideas / decisions (following experimentation) concerning how NNBD can best be realized in Dart.

I can grant you access to the repo if you'd like to be able to modify the DEP yourself. If Dan agrees, I'd be glad to contribute to updating the DEP myself.

That being said, I don't want to slow you down.

Let me know what you prefer, or even if you'd like for us to hold a short coordinating meeting.

Cheers,

Patrice

[Bob Nystrom]

On Mon, Feb 6, 2017 at 3:46 PM, Patrice Chalin <pch...@gmail.com> wrote:

On Friday, February 3, 2017 at 9:36:18 AM UTC-8, rnystrom wrote:

... I wrote up a proposal here: https://github.com/dart-lang/sdk/pull/28619

That was mostly an intellectual exercise for me. I learn a lot by writing about something so hashing out that proposal forced me to think through a bunch of cases. It's also a little easier for me to evolve since it lives in the SDK repo and is just a plain Markdown file.

I agree that it makes sense to keep such a planning/todo-list-like document as separate file that is more easily modifiable.

 

For the prototype, I wasn't planning to add in support for disabling them on a per library/file/class basis. Instead, I figured I would just use diffs on that generated file in order to not get overwhelmed by the sea of errors.

Sure, keeping things as simple as possible for the prototype makes sense.

...

For the real implementation, yes, this is definitely something I see us wanting to tool. I made a note of it on the tracking bug so we don't forget.

Good.

For your current implementation of NNBD in the analyzer, which type hierarchy are you using --- e.g., a new root (B.2.1), or making Null a root (B.4.7), or something else?

This is one of the questions I'm hoping the prototype will help us answer confidently. Right now, I'm making as few changes as possible, so I am keeping Object as the root of the hierarchy because that's how it is today. See "Object is always nullable" here. This means there is no way to express "Any Object but not null."

In the prototype that I had developed (in which I was also trying to make as few changes as possible), I let `Object` be the root of the non-nullable type hierarchy, and used `Object?` to represent a possibly null object. This is achieved by making Null a root (B.4.7) --- and hence the type hierarchy becomes a "forest", rather than a single tree ... or, if we insist on having a single tree, we can elect the (denotation of the type expression) `Object?` as the new root.

... Real usability data > opinions and guesswork.

+1

Let me know if there's anything else I can do to coordinate better with you. I'm trying to move quickly and get hacking on stuff, but I don't want to step on any toes or waste time rediscovering something you already know.

I'm open to suggestions. I think that it would be great to keep the DEP-30 document up-to-date so as to reflect our latest ideas / decisions (following experimentation) concerning how NNBD can best be realized in Dart.

Yeah, I agree.

 

I can grant you access to the repo if you'd like to be able to modify the DEP yourself. If Dan agrees, I'd be glad to contribute to updating the DEP myself.

Both of those sound fine to me.

 

That being said, I don't want to slow you down.

Let me know what you prefer, or even if you'd like for us to hold a short coordinating meeting.

How about I keep on trucking with the prototype like I am now? I'm under a lot of time pressure to reach a point where we can make a go/no-go decision. I'll send out period status updates so we can stay coordinated and you can point out dead ends before I go too far down them, hopefully.

Then assuming, we decide "go", we update the DEP with what I learned from the prototype and then use that moving forward to guide the real implementation. Sound good?

– bob

[Patrice Chalin]

SGTM