pre-SIP proposal discussion: Partial type inference, and named type arguments

[Ahmad Salim Al-Sibahi]

Hi everyone @ scala-debate,

While using Scala, I have noticed that it does not currently support partial application of type arguments when calling methods: one must either let Scala infer every argument or one must provide each individual argument oneself. This is seemingly inconvenient when calling methods which accepts multiple type arguments and Scala only fails to infer one of the arguments correctly; especially, when some of the other arguments have long type names. Another situation where this also presents an inconvenience, is when some of the type arguments are dependent on each other and providing the full type argument list thus leads to some duplication.

The following two examples illustrate the point:

import scalaz.\/._

object Application extends App {
  val nums = Set(1,2,3)
  val _ = nums.foldLeft(right(0)) { (acc, i) =>
    if (i < 0) left("negative number") // Error, found: String("negative number") required: Nothing
    else for (prevsum <- acc)
            yield (prevsum + i)
  }
}

Where it infers the value of B (the accumulator type) in foldLeft to Nothing \/ Int, and therefore is incompatible with the string used in the provided function.

Of course, one could solve this by providing the type arguments to right explicitly i.e. [String, Int]; however, since it could infer the second argument automatically, it seems inconvenient to have to type it in explicitly (imagine having a longer type Set[Map[String, Int]] on the right hand-side and you can see especially why).

Similarly, for dependent arguments like used in flatMap, there is duplication in type signature.

import scalaz.\/.
import scalaz.\/._

object Application extends App {
  val nums = Set(1,2,3)
  val a = nums.flatMap[String \/ Int, Set[String \/ Int]](a => Set(right(a))) // Note required duplication of String \/ Int argument
}

Therefore, it might be nice if possible to partially apply (either using _ or named arguments) type signature and let Scala take care of the rest.

In the first example, one should then be able to write right[String, _](0) or right[A = String](0) , and for the second example one could write nums.flatMap[String \/ Int, _] or nums.flatMap[String \/ Int, Set[_]] or nums.flatMap[B = String \/ Int].

I believe that this will strike a better balance than having to write everything explicitly, when Scalas automatic inference fails or produces the wrong result.

All (constructive) feedback and ideas are welcome. If people are interested, I shall go forth and write an official SIP.

Thanks in advance,

Regards,

AS

[Jason Zaugg]

On Thu, Aug 20, 2015 at 8:54 PM Ahmad Salim Al-Sibahi <asal.t...@gmail.com> wrote:

Therefore, it might be nice if possible to partially apply (either using _ or named arguments) type signature and let Scala take care of the rest.

In the first example, one should then be able to write right[String, _](0) or right[A = String](0) , and for the second example one could write nums.flatMap[String \/ Int, _] or nums.flatMap[String \/ Int, Set[_]] or nums.flatMap[B = String \/ Int].

I believe that this will strike a better balance than having to write everything explicitly, when Scalas automatic inference fails or produces the wrong result.

All (constructive) feedback and ideas are welcome. If people are interested, I shall go forth and write an official SIP.

Here's one previous discussion on this topic: "another matter of syntax". I recall the topic coming up once or twice since then, might be worth trawling the archives for any other discussions. 

I think that its an interesting idea that is worth fleshing out. I’d favour use of _, or maybe Predef.?, as the signal to infer the type, rather than named type parameters, which would be a bigger change to the language.

Here’s a quick prototype of the change that shows that the impact on the typechecker might not be too large. The backticked underscore is a way to sneak a wildcard through the parser, and aren’t a serious suggestion for the user facing syntax.

scala> def test[A, B >: A]: (A, B) = ???; def x = test[Int, `_`]
test: [A, B >: A]=> (A, B)
x: (Int, Int)

scala> val nums = Set(1,2,3)
nums: scala.collection.immutable.Set[Int] = Set(1, 2, 3)

scala> def right[L, R](r: R): Either[L, R] = Right(r)
right: [L, R](r: R)Either[L,R]

scala> def left[L, R](l: L): Either[L, R] = Left(l)
left: [L, R](l: L)Either[L,R]

scala> nums.foldLeft(right[String, `_`](0))( (acc, i) => left("nope"))
res0: Either[String,Int] = Left(nope)

scala> nums.flatMap[Either[String, Int], `_`](a => Set(right(a)))
res1: scala.collection.immutable.Set[Either[String,Int]] = Set(Right(1), Right(2), Right(3))

​

-jason

[Naftoli Gugenheim]

Another proposal of Paul's was based on something like currying, so you could have a function that lets you write f[Int][String](x) out just f[Int](x). I think the simple way of implementing it was nested apply's without parentheses, like def apply[A] = { def apply[B](x: B) = null }

This would allow for the same shortcutting, without (a) overloading _ more, or (b) giving an identifier like ? special meaning to the compiler.

--
You received this message because you are subscribed to the Google Groups "scala-debate" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-debate...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[martin odersky]

One argument against using `_' is that it already means "wildcard",
i.e. unbound type parameter.
So,

HashMap[Int, _]

and

HashMap[Int, _](1 -> "a")

would behave differently. And the latter would again be different from:

new HashMap[Int, _] ++ (1 -> "a")

I wonder whether this is this getting too confusing?

- Martin

--
Martin Odersky
EPFL

[Jason Zaugg]

I agree it is tough to add new meanings to _.

But I do think the facility is genuinely useful. Wdyt about using an identifier like `?` ?

[Dale Wijnand]

I agree that it would be nice as well.

I want to mention https://github.com/non/kind-projector which is also using '?'.

Dale

[Ahmad Salim Al-Sibahi]

Hi again,

Thanks for the answers.

@Jason I will take a look at previous discussion as well then.

@Everyone I think using `?`, `!` or similar identifier may perhaps be a better idea than `_`, because of the conflicts you had suggested.

                    Alternatively, one could just have explicitly named type arguments instead, which is not a problem since the names are often one or two letters long.

@nafg I think that currying sounds interesting and might solve a part of the problem; however, as far as I recall unification happens only in the same argument list in Scala (I would assume the same about type argument lists, when getting currying) and it might add some other challenges (such as having a fixing the order of arguments inferred automatically and the ones provided by the user). I guess it is more useful for a scenario where one wants to do things like kind-project (linked by Dale) than for simple partial inference.

Regards,

AS

[martin odersky]

On Fri, Aug 21, 2015 at 1:08 PM, Jason Zaugg <jza...@gmail.com> wrote:
> I agree it is tough to add new meanings to _.
>
> But I do think the facility is genuinely useful. Wdyt about using an
> identifier like `?` ?

Not sure yet. Maybe we'll want to reserve `?` so that T? would mean T
| Null. I'd also explore named type arguments as an alternative.

- Martin

[Simon Ochsenreither]

Not sure yet. Maybe we'll want to reserve `?` so that T? would mean T
| Null. I'd also explore named type arguments as an alternative.

I would prefer not going down that route. Giving special syntax for nulls will encourage people to start using it more extensively.
Currently null pretty much doesn't exist in Scala, and I think that's going to change if we provide "first-class" syntax. (See procedure syntax.)

[David Barri]

I would prefer not going down that route. Giving special syntax for nulls will encourage people to start using it more extensively.
Currently null pretty much doesn't exist in Scala, and I think that's going to change if we provide "first-class" syntax. (See procedure syntax.)

Absolutely! I'd like to emphatically echo Simon's point here. 

[Ahmad Salim Al-Sibahi]

Hi,

Alternatively, one could use no symbol in places where automatic inference is needed a la C#.

E.g. one could have 

• right[String,](0) instead of the previously proposed right[String,_] or right[String,?]
• nums.flatMap[String \/ Int, ] instead of the previously proposed nums.flatMap[String \/ Int, _]
• nums.flatMap[String \/ Int, Set[]] instead of the previously proposed nums.flatMap[String \/ Int, Set[_]]
• etc. 

More concretely:

If we have a construct T which takes a single argument then T[] states that the argument should be automatically inferred, for two arguments T[,] states that both arguments should be inferred,

T[A,] states that the second argument should be inferred (and the first given A), T[,B] states that the first argument should be inferred (and the second given B) and T[A,B] is fully specified. This of course, generalises to all sizes of type argument lists.

Although, a setback would be that it would be harder to parse (both for machines and humans) and becomes overwhelming when you have more than 3 arguments, e.g. T[,,,,,] can seem a bit hard to understand.

Regards,

AS

[Simon Schäfer]

It is not just about parsing, but also about error message generation. The compiler has no chance to know if you just forgot to add a parameter or if you intentionally left it when there is a type mismatch.

Regards,

AS

[Jon Pretty]

Hi Simon,

I think the point here is not that it would encourage people to use nulls, but would handle them in an acceptably-concise way everywhere they exist that we can't do anything about them, e.g. Java APIs.

Complementing this would be a new global assertion that non-? types like `String` and `List[Int]` could never be null (assigning `null` to them would be an error), and this would be an extremely valuable enhancement. `T` would coerce to `T?` for all `T`, but not the other way round, unless you import or define a "dangerous" implicit, presumably. We could have that implicit emit a warning, if we wanted.

So, for example, the signature of `System.getProperty` would become `def getProperty(key: String?): String?`, and that `?` would serve as a constant reminder that we need to explicitly handle these `null`s, probably with something like the `Option(...)`, or perhaps some more concise convenience methods.

So, would this *encourage* people to use nulls? The default for any type you write down would be not to support `null`s, so I don't see why people would voluntarily add a `?` to a type just to make it more inconvenient to use.

But it probably *would* encourage the use of nulls instead of `Option`s. So, is that a problem, now that our type system encapsulates that nullability well? We've just got a typesafe, boxing-free `Option`!

If this is the direction it's going, that gets a big +1 from me.
​

Cheers,

Jon

--

Jon Pretty

Propensive Ltd

@propensive

[Denys Shabalin]

On Fri, Aug 28, 2015 at 3:25 PM, Jon Pretty <jon.p...@propensive.com> wrote:

So, for example, the signature of `System.getProperty` would become `def getProperty(key: String?): String?`, and that `?` would serve as a constant reminder that we need to explicitly handle these `null`s, probably with something like the `Option(...)`, or perhaps some more concise convenience methods.

In Swift nil-able T? types have Option-like interface by default. So you can't even call a method without handling the null case. Mapping over an option has built-in syntax called "optional chaining". I think that's very much state of the art in this area and it would be great to have some of that goodness in Scala.

[Jon Pretty]

Hi Denys,

That's roughly what I was thinking, except that there are some Java methods which might expect a `null` to be passed to them, so the interface would need to support this. We could have to have some sort of "whitelist" facility for Java methods which are known not to accept or return `null`s, but that's a lot more complexity (and far from necessary for this to be useful).

First-class option-chaining sounds great, in the context of a language that's spent over a decade suffering from *every* reference type being an unchecked option...

The big problematic area I can see, which Martin may have already solved (or sufficiently constrained) in Dotty, is guaranteeing that all non-nullable references are initialized before they're accessed.

Cheers,

Jon

[martin odersky]

On Fri, Aug 28, 2015 at 5:20 PM, Jon Pretty <jon.p...@propensive.com> wrote:
> Hi Denys,
>
> That's roughly what I was thinking, except that there are some Java methods
> which might expect a `null` to be passed to them, so the interface would
> need to support this. We could have to have some sort of "whitelist"
> facility for Java methods which are known not to accept or return `null`s,
> but that's a lot more complexity (and far from necessary for this to be
> useful).
>
> First-class option-chaining sounds great, in the context of a language
> that's spent over a decade suffering from *every* reference type being an
> unchecked option...
>
> The big problematic area I can see, which Martin may have already solved (or
> sufficiently constrained) in Dotty, is guaranteeing that all non-nullable
> references are initialized before they're accessed.

Doing this statically seems to be very hard. I mean, it's easy to do
something simple, but it would likely be too restrictive. As soon as you want
to give some expressive power, the complexities mount. See (*), which is a
good paper, but it also shows how hard it is.

My current thinking is to do it at runtime on access (maybe controlled
by a switch similar to -Xcheck-init). The big advantage is that since
`null` is no longer a legal value, checking initialization status is
just null checking, so no separate bitmap is needed and the test is
very fast. In essence, we replace NPE by "NotInitializedError".

(*) Manuel F¨ahndrich and K. Rustan M. Leino. Declaring and checking
non-null types in an object-oriented language. In Proceedings of the
2003 ACM SIGPLAN Conference on Object-Oriented Programming Systems,
Languages and Applications, OOPSLA 2003, volume 38, number 11.

Cheers

- Martin

>
> Cheers,
> Jon
>
> On 28 August 2015 at 16:03, Denys Shabalin <denys.s...@epfl.ch> wrote:
>>
>> On Fri, Aug 28, 2015 at 3:25 PM, Jon Pretty <jon.p...@propensive.com>
>> wrote:
>>>
>>>
>>> So, for example, the signature of `System.getProperty` would become `def
>>> getProperty(key: String?): String?`, and that `?` would serve as a constant
>>> reminder that we need to explicitly handle these `null`s, probably with
>>> something like the `Option(...)`, or perhaps some more concise convenience
>>> methods.
>>
>>
>> In Swift nil-able T? types have Option-like interface by default. So you
>> can't even call a method without handling the null case. Mapping over an
>> option has built-in syntax called "optional chaining". I think that's very
>> much state of the art in this area and it would be great to have some of
>> that goodness in Scala.
>
>
>
>
> --
> Jon Pretty
> Propensive Ltd
> @propensive
>

[David Barri]

 - Scala would become a null safe language
 - There would be an alternative to Option that does not require
boxing, which in turn would make lots of operations faster.

Woah, that's a pretty neat idea. Is this something planned for Dotty?

On 28 August 2015 at 23:22, Jon Pretty <jon.p...@propensive.com> wrote:

Hi Simon,

I think the point here is not that it would encourage people to use nulls, but would handle them in an acceptably-concise way everywhere they exist that we can't do anything about them, e.g. Java APIs.

Complementing this would be a new global assertion that non-? types like `String` and `List[Int]` could never be null (assigning `null` to them would be an error), and this would be an extremely valuable enhancement. `T` would coerce to `T?` for all `T`, but not the other way round, unless you import or define a "dangerous" implicit, presumably. We could have that implicit emit a warning, if we wanted.

So, for example, the signature of `System.getProperty` would become `def getProperty(key: String?): String?`, and that `?` would serve as a constant reminder that we need to explicitly handle these `null`s, probably with something like the `Option(...)`, or perhaps some more concise convenience methods.

So, would this *encourage* people to use nulls? The default for any type you write down would be not to support `null`s, so I don't see why people would voluntarily add a `?` to a type just to make it more inconvenient to use.

[Simon Ochsenreither]

I think having String|Null is a perfectly fine way to express nullable things. I'm a bit concerned because we spend the last years cleaning up the language, and adding special syntax sugar would completely undo this progress.
Saving 4 characters is not worth it in my opinion, considering it also adds yet another way to describe the same thing.

I don't think that nulls will be required that often in practice, even when working with Java APIs. All the lessons learned from other languages are that littering Java APIs with |? (or |Null) just doesn't scale in practice, because it's hardly String|Null we are dealing with but ArrayList[String|Null]|Null and friends.

I think working with Java will require some language import which says "don't bother me with null-checking things from Java APIs", because nullability/non-nullability is nothing that can be retrofitted into the Java world. The APIs were never designed this way, and it will show in every little corner.

[martin odersky]

On Sat, Aug 29, 2015 at 12:10 AM, David Barri <japg...@gmail.com> wrote:
> - Scala would become a null safe language
> - There would be an alternative to Option that does not require
> boxing, which in turn would make lots of operations faster.
>
> Woah, that's a pretty neat idea. Is this something planned for Dotty?
>

Certainly not the first release. These are thought experiments about
what could be down the road.

Cheers

- Martin

[martin odersky]

On Sat, Aug 29, 2015 at 7:30 AM, Simon Ochsenreither
<simon.och...@gmail.com> wrote:
> I think having String|Null is a perfectly fine way to express nullable
> things. I'm a bit concerned because we spend the last years cleaning up the
> language, and adding special syntax sugar would completely undo this
> progress.
> Saving 4 characters is not worth it in my opinion, considering it also adds
> yet another way to describe the same thing.
>
> I don't think that nulls will be required that often in practice, even when
> working with Java APIs. All the lessons learned from other languages are
> that littering Java APIs with |? (or |Null) just doesn't scale in practice,
> because it's hardly String|Null we are dealing with but
> ArrayList[String|Null]|Null and friends.
>

No, "?" would be a postfix operator). You'd write

ArrayList[String?]?

As to commonality of usage, note that T? would also be the new, more
efficient way to write Option[T]. So I expect it would be quite widely
used!

> I think working with Java will require some language import which says
> "don't bother me with null-checking things from Java APIs", because
> nullability/non-nullability is nothing that can be retrofitted into the Java
> world. The APIs were never designed this way, and it will show in every
> little corner.
>

Very true, and that is where effect checking comes in. If null
dereference is an effect we can apply the normal effect scoping rules
to it. For instance, there might be an import

import canThrowNull

which essentially gives you a capability to use operations that can
cause a NPE. So then

getProperty("myProp").char(0)

would be allowed (and could throw a NPE). But the import would signal
loud and clear that unsafe stuff happens in its scope. So people will
gradually want to reduce such imports (at least I hope so).

Cheers

- Martin

[Julien Richard-Foy]

Comparing `A | Null` vs `Option[A]`: with `Option[A]` you have a lot of methods, e.g. `getOrElse`, on the other hand with `A | Null` how would you handle your value excepted by pattern matching on it? Will we have a “boxed” `A | Null` type that will provide all the useful methods of `Option[A]`?

[martin odersky]

On Sat, Aug 29, 2015 at 1:07 PM, Julien Richard-Foy <juli...@gmail.com> wrote:
> Comparing `A | Null` vs `Option[A]`: with `Option[A]` you have a lot of
> methods, e.g. `getOrElse`, on the other hand with `A | Null` how would you
> handle your value excepted by pattern matching on it? Will we have a “boxed”
> `A | Null` type that will provide all the useful methods of `Option[A]`?

I was thinking extractor for pattern matching and implicit value class
to add the operations.

[Simon Ochsenreither]

No, "?" would be a postfix operator). You'd write

    ArrayList[String?]?

I'm not sure I'm getting this. Whether one uses the ? or the |Null syntax doesn't change the issue that nullability can exist deeply nested in some generic structures, and it's extremely hard to get rid of it there.
 

As to commonality of usage, note that T? would also be the new, more
efficient way to write Option[T]. So I expect it would be quite widely
used!

Would we get similar operators for Either, Try, Validation etc.?
If not, my main concern about this is that having nicer syntax for T|Null/Option[T] means people will prefer using it even if Either, Try or Validation would be more appropriate.
If we look at languages which provide syntax sugar like the ? operator, we can see that alternatives like Either, Try, Validation stopped existing there. (And I don't think that's because Option can support all use-cases, but because due to the nice syntax, people just use it for everything.)
 

Very true, and that is where effect checking comes in. If null
dereference is an effect we can apply the normal effect scoping rules
to it.

Agree! That's exactly what I was thinking. If we ever want to change null-handling we will need such options to give people the option to migrate their code gradually.

Thanks and bye,

Simon

[Rex Kerr]

On Sat, Aug 29, 2015 at 12:03 PM, Simon Ochsenreither <simon.och...@gmail.com> wrote:

No, "?" would be a postfix operator). You'd write

    ArrayList[String?]?

I'm not sure I'm getting this. Whether one uses the ? or the |Null syntax doesn't change the issue that nullability can exist deeply nested in some generic structures, and it's extremely hard to get rid of it there.

But this is true of everything.  Option is hard to get rid of there.  So is Either.  If your types tell you it's there, maybe you can go get it.  But the performance and conceptual overhead is too great if you have to type Option[T] instead of T? every time, given that Java interop is important.

I can't see it as anything but a win for at least our types to tell us that lurking somewhere in there is a null that hasn't been properly handled.

 

 

As to commonality of usage, note that T? would also be the new, more
efficient way to write Option[T]. So I expect it would be quite widely
used!

Would we get similar operators for Either, Try, Validation etc.?

Yes-ish?  Either[String, Foo] would be just String | Foo in many cases.  Try is already really compact (Try[Foo]).  The name "Validation" is awkwardly long, but one can find a shortcut.

 

If not, my main concern about this is that having nicer syntax for T|Null/Option[T] means people will prefer using it even if Either, Try or Validation would be more appropriate.

That's already true when you have to specify the other type.  You can't make things easier than they already are; the only thing easier than String | Foo would be String Foo.

The first battle to fight is at least represent optional types as optional when you can.  Right now it's tempting to use T <: AnyRef with null meaning "no value" in place of Option[T] because it's so much harder to write Option[T] (for some actual T where you don't need to write <: AnyRef).

 

If we look at languages which provide syntax sugar like the ? operator, we can see that alternatives like Either, Try, Validation stopped existing there.

Can you provide specific examples?  At least in the case of Swift, the core API is pretty small, and blogs are full of examples of Either and Result and so on using enums.

Either way, I don't think that languages should do very much of making useful and often-correct things harder to do because they might be overused.  A few cases can be targeted when there's a better way (e.g. is/as -> isInstanceOf/asInstanceOf to encourage people to pattern match), but mostly I think the onus should be on the designers to make the other features compelling enough so they stand on their own merit, not just are acceptable given how awkward and painful the alternatives are.

  --Rex