Optional parameters (a language proposal)

[Ryan Hendrickson]

Hi all!

I've been kicking around this idea for an addition to Scala's core syntax for about a month, and I've started work on a patch to the compiler to implement it. I'd very much like to see if anyone else besides me has any interest in/feedback on this idea, and hopefully to promote it to a SIP.

In a nutshell: I want to allow optional parameters (kind of like default arguments, but different—complementary to them, in a sense) to functions, that behave like Option the same way that repeated/vararg parameters behave like Seq. This arises from a problem (of the class ‘useful, but could be more useful’) I have with default arguments. Read on for an explanation of the problem and what I'm trying to do about it.

(Apologies if something like this came up and was rejected when default arguments were being proposed for 2.8. I'm pretty new to the Scala community, and while I read back through most of the scala-debate archives and didn't see any discussion of a feature like this, I'm well aware that I could have just missed it.)

1 Motivation

Suppose you have a service that looks like this:

class CookieBaker {
  def bakeCookies(quantity: Int,
                  flavor: CookieFlavors.Value = CookieFlavors.ChocolateChip) =
    sys.error("Not implemented yet...")
}

The fact that the default flavor is chocolate chip is an implementation detail of the baker. Callers don't need to know it, and the CookieBaker class could change the flavor at any time without having to change other code that truly is agnostic to the flavor of cookies produced. This is as it should be.

Now consider another class that uses CookieBaker:

class KitchenManager(cookieBaker: CookieBaker, cakeBaker: CakeBaker) {
  def bakeForParty(partySize: Int,
                   cookieFlavor: CookieFlavors.Value = CookieFlavors.ChocolateChip) {
    cookieBaker.bakeCookies(partySize*3, cookieFlavor)
    cakeBaker.bakeCakes(partySize/4)
  }
}

Now we are in trouble: we have redundantly specified that the default cookie flavor is chocolate chip in two separate classes. Maybe that's okay; it could be that in the domain of this problem, the KitchenManager and the CookieBaker separately have independent ideas about what the default cookie flavor should be (and at the moment, they happen to coincide). But let's assume in this case, if the KitchenManager is not told what cookie flavor to provide, it would prefer to defer to the CookieBaker.

So you can revise KitchenManager in a couple of different ways, none of which are great:

class KitchenManager(cookieBaker: CookieBaker, cakeBaker: CakeBaker) {
  def bakeForParty(partySize: Int, cookieFlavor: Option[CookieFlavors.Value]) {
    cookieFlavor match {
      case Some(flavor) => cookieBaker.bakeCookies(partySize*3, flavor)
      case None => cookieBaker.bakeCookies(partySize*3)
    }
    cakeBaker.bakeCakes(partySize/4)
  }
}

class KitchenManager(cookieBaker: CookieBaker, cakeBaker: CakeBaker) {
  def bakeForParty(partySize: Int, cookieFlavor: CookieFlavors.Value) {
    cookieBaker.bakeCookies(partySize*3, cookieFlavor)
    cakeBaker.bakeCakes(partySize/4)
  }
  def bakeForParty(partySize: Int) {
    cookieBaker.bakeCookies(partySize*3)
    cakeBaker.bakeCakes(partySize/4)
  }
}

(If I'm missing a very clean way to do this in existing Scala, please let me know!)

The first solution is a little ugly to call (callers have to use Some or None, which is both extra clutter and not consistent with the more lightweight calling pattern for CookieBaker), and a little ugly to implement (you have to call cookieBaker.bakeCookies in both cases of the match, with very similar arguments—and about the only thing you can factor out of this would be a def quantity = partySize*3, which certainly doesn't make the method much easier to read). You could solve the first of these problems with implicits and either some new Optional trait which is a dumb wrapper for an Option (sad) or implicit conversions to/from Option itself (potentially a source of bugs/confusion elsewhere!); but I think the second problem is basically intractable (and grows exponentially with the number of default arguments under consideration!).

The second solution is ideal to call, but has the same redundancy/exponential code growth problems as the first, and what's worse, the growth affects actual method overloads rather than case clauses isolated inside a single method. Plus, it feels plain dirty to use default arguments at one layer of the software, and overloads at another, when talking about the same thing threaded between the two layers.

There are other hacks: you could use (shudder) null as the default argument to KitchenManager, and trust that null doesn't have some special significance to some other part of the software. You could, if you can edit the CookieFlavors enum, give it a Default value, but then everything that ever takes a CookieFlavors.Value would have to define what to do if given a Default. None of these solutions feel good and Scala-ish to me.

Here's the code I'd like to write (the bold red bits are new syntax that this language proposal would enable):

class KitchenManager(cookieBaker: CookieBaker, cakeBaker: CakeBaker) {
  def bakeForParty(partySize: Int, cookieFlavor: CookieFlavors.Value?) {
    cookieBaker.bakeCookies(partySize*3, cookieFlavor:_?)
    cakeBaker.bakeCakes(partySize/4)
  }
}
            
myKitchenManager.bakeForParty(5)
myKitchenManager.bakeForParty(10, CookieFlavors.Sugar)
val optFlavor: Option[CookieFlavors.Value] = getCookieFlavorPreference
myKitchenManager.bakeForParty(15, optFlavor:_?)

If it isn't obvious what the new ? syntax is intended to mean, consider it analogous to the * character in repeated parameters:

‘*’ is to scala.Seq as ‘?’ is to scala.Option

Just like * lets callers provide zero or more arguments for a parameter, which get wrapped up in a Seq, ? is intended to let callers provide zero or one arguments for a parameter, which gets wrapped up in an Option. The remainder of this document hammers out in pedantic detail what this means technically, but hopefully that analogy tells you enough to be able to understand the code and form an opinion on whether Scala would be better off with this feature.

Incidentally, I think of this improvement as two mostly independent features: optional parameters (the ability to declare a parameter with type T? as sugar for Option[T], with the feature that such parameters expect an argument of type T which may be missing) and option arguments (the ability to call a method expecting arguments of type T using arguments of type Option[T], if the language knows what to do if that argument is missing). The next two sections describe each of these separately, though their utility and the spirit of the */? analogy is I think most evident when both are considered together.

2 Optional Parameters

This proposed feature provides for the declaration/definition of functions with optional parameters with a syntax analogous to repeated parameters. The proposal builds on the work done with default arguments for Scala 2.8.

Parameter types can be suffixed with ‘?’ to indicate that a parameter is optional. Just as the type of a repeated parameter (annotated with T*) in the definition of the function is scala.Seq[T], the type of an optional parameter (annotated with T?) in the definition of the function is scala.Option[T]. When applying a function with one or more optional parameters, overload resolution, implicit conversions, and parameter binding proceed exactly as if each optional parameter a: T? were actually declared with a default argument as a: T = undefined, where undefined is a fictitious term of type T. After binding has succeeded, if optional parameter a is bound to an expression expr, that expression is lifted to scala.Some(expr); if a is not bound (where, for an ordinary parameter with a default argument, the compiler would insert a call to the default), the binding a = scala.None: Option[T] is used.

A function declaration/definition with optional parameters is valid if and only if the equivalent function with all optional parameters a: T? replaced with a: T = undefined would be valid. In particular, it is allowed to declare any number of optional or non-optional parameters in any order in a parameter section, but it is not allowed to have a parameter section with both optional parameters and a repeated parameter.

2.1 Syntax

The production of ParamType in SLS 2.9 chapter 4 is extended with the following clauses:

ParamType ::= Type ‘?’
            | ‘=>’ Type ‘?’

2.2 Integration with other features

By-Name Parameters An optional parameter declared with the by-name arrow ‘=>’ works such that every usage of the parameter within the body of the function re-evaluates the lifted argument expression (scala.Some(expr) or scala.None, depending).

Default Arguments Works exactly as if the optional parameter a: T? were declared as a: T = undefined. An optional parameter may not be declared with a default argument itself, but within a single parameter section, optional parameters and parameters with default arguments may coexist.

Implicit Parameters If an optional parameter a: T? is implicit, an implicit value matching T (not scala.Option[T]) is searched for. If an implicit value v matching T is found, the value of a inside the function is scala.Some(v); if none is found, the value of a inside the function is scala.None.

Overloading For the purpose of determining whether a member definition matches another (per definition 5.1.4 in SLS 2.9), an optional parameter a: T? is considered to have type Option[T]. In addition, at most one overloaded alternative of a method is allowed to have default arguments or optional parameters (or both).

Overloading Resolution As with default arguments, when multiple overloaded alternatives are applicable, the alternative which uses optional parameters is never selected.

Overriding If a member M matches a non-private member M′ of a base class, then in addition to the existing restrictions on M and M′ defined in section 5.1.4 of SLS 2.9, this additional restriction applies to M: a parameter of M must be optional if and only if it corresponds to an optional parameter of M′. (In other words, even though the definition of matching is amended to define T? to be equivalent to Option[T], a T? parameter cannot be overridden with an Option[T] parameter or vice versa.)

Repeated Parameters As with default arguments, repeated parameters and optional parameters may not coexist in a parameter section.

2.3 Example

The following code is a trivial use of optional parameters:

def show(i: Int?) = i match {
  case Some(value) => println("i = " + value)
  case None => println("i is unspecified")
}

show(5) // prints "i = 5"
show()  // prints "i is unspecified"

It is semantically equivalent to the following Scala 2.9 code:

def show(i: Option[Int]) = i match {
  case Some(value) => println("i = " + value)
  case None => println("i is unspecified")
}

show(Some(5))
show(None)

3 Option Arguments

This proposed feature provides for using a scala.Option[T] to determine at runtime whether to ‘take the default’ when calling a function with a parameter of type T that is either optional or has a default argument.

Analogously to the existing sequence argument feature, an expression of type scala.Option[T] can be marked to be an option argument with a _? type annotation. For the purposes of resolving overloads and parameter binding, option arguments are taken to have type T, except that an option argument can only be considered compatible with optional parameters or parameters with default arguments; all other rules apply as usual. If the option argument expr: Option[T] is successfully bound to a parameter a: T = default, then the actual value of a in the function is expr.getOrElse(default). If the option argument is successfully bound to an optional parameter a: T?, then the actual value of a in the function is expr (skipping the usual lifting performed for optional parameters as described in section 2 of this proposal).

3.1 Syntax

The production of ArgumentExprs as defined in chapter 6 of SLS 2.9 is amended to the following:

ArgumentExprs ::= ‘(’ [Exprs1] ‘)’
                | ‘(’ [Exprs ‘,’] PostfixExpr ‘:’ ‘_’ ‘*’ ’)’
                | [nl] BlockExpr
Exprs1        ::= Expr [‘:’ ‘_’ ‘?’] {‘,’ Expr [‘:’ ‘_’ ‘?’] }

Note that, for instance, someFunc(a:_?, b:_*) is syntactically invalid (and could never match a real function anyway, as repeated parameters and default argument/optional parameters can't coexist in a parameter section).

3.2 Integration with other features

By-Name Parameters An option argument to a function with a by-name parameter works such that every usage of the parameter within the body of the function re-evaluates the unlifted argument expression or the entire expr.getOrElse(default) expression, depending on whether the parameter is optional or has a default argument.

Overloading Resolution Because only one overload alternative can use default arguments or optional parameters, when applying an overloaded function with option arguments, only that alternative is considered for applicability. Option arguments do not allow for runtime switching between different overload alternatives.

3.3 Example

Continuing the example from 2.3:

val three = Some(3)
show(three:_?) // same as show(3)
show(None:_?)  // same as show()

----------------------------------------

This message is intended exclusively for the individual(s) or entity to
which it is addressed. It may contain information that is proprietary, 
privileged or confidential or otherwise legally exempt from disclosure. 
If you are not the named addressee, you are not authorized to read, 
print, retain, copy or disseminate this message or any part of it. 
If you have received this message in error, please notify the sender 
immediately by e-mail and delete all copies of the message.

[Paul Phillips]

On Thu, Feb 2, 2012 at 8:39 AM, Ryan Hendrickson
<ryan.hen...@bwater.com> wrote:
> (If I'm missing a very clean way to do this in existing Scala, please let me
> know!)

It is, if not "very clean", at least substantially cleaner than the
options you provided.

class CookieBaker {
def defaultFlavor: CookieFlavors.Value = CookieFlavors.ChocolateChip

def bakeCookies(quantity: Int, flavor: CookieFlavors.Value =

defaultFlavor) = ???
}

class KitchenManager(cookieBaker: CookieBaker, cakeBaker: CakeBaker) {
def bakeForParty(partySize: Int, cookieFlavor: CookieFlavors.Value =

cookieBaker.defaultFlavor) {
cookieBaker.bakeCookies(partySize*3, cookieFlavor)
cakeBaker.bakeCakes(partySize/4)
}
}

[Paul Phillips]

I see looking at more of the words that you don't want to do this, but
to me you seem to be working way too hard to avoid it. It's not that
much of an implementation detail.

[Ryan Hendrickson]

Yeah, that's definitely a valid solution for that example. What this doesn't address (and what I suppose I did a poor job of highlighting) is the idea of being able to distinguish between a caller providing a value that happens to be the default, however it is derived, and the caller providing no value. For that, I really do want a parameter that is an Option, but I also don't want the baggage of having to call it like an Option.

The CookieBaker/KitchenManager example is an oversimplified version of a problem I actually encountered, and in that problem I was altering a CookieBaker-like class that used a default argument to one that needed default behavior that wasn't easily separated out from the rest of the ‘baking’ process. If it had been separable, or even a simple function of the other parameters, I would have done something like what you suggested. But what I really needed in this case was an Option, except I didn't want to sacrifice the clean API my class's users were already used to, and I had to think about what the repercussions for this design choice would be all the way up the stack (KitchenManager and higher). That's what got me thinking about whether Scala could support a syntax for calling Option-parametered functions in as mentally effortless a way as it currently supports calling functions with repeated parameters or default arguments.

I don't know, the juice might not ultimately be worth the squeeze here. I think it's a neat idea, and I've managed to satisfy myself that it's an implementable one. But that's why I'm floating this, to see if the problem/solution resonates with anyone else.

[Justin du coeur]

On Thu, Feb 2, 2012 at 12:33 PM, Ryan Hendrickson <Ryan.Hen...@bwater.com> wrote:

I don't know, the juice might not ultimately be worth the squeeze here. I think it's a neat idea, and I've managed to satisfy myself that it's an implementable one. But that's why I'm floating this, to see if the problem/solution resonates with anyone else.

Just an end-user opinion, but I like it.  This is a pretty common anti-pattern in my experience, and I do find the approach rather elegant.

(Whether it's worth the effort for those building the language is another question, that I'm not qualified to opine on.)

[Paul Phillips]

On Thu, Feb 2, 2012 at 9:33 AM, Ryan Hendrickson
<Ryan.Hen...@bwater.com> wrote:
> Yeah, that's definitely a valid solution for that example. What this doesn't address (and what I suppose I did a poor job of highlighting) is the idea of being able to distinguish between a caller providing a value that happens to be the default, however it is derived, and the caller providing no value.

It won't win too many elegance awards, but this is one of the few
(almost the only) places I use null. The primary arguments for Option
over null are inapplicable because it never escapes the immediate
scope, and I trust myself to handle null for the distance of a block.

[Ittay Dror]

FWIW, +1

I think this is one of those features that can help sell Scala. It is easy to understand, easy to demonstrate and I think it will be easy for people to relate to it.

Ittay

[HamsterofDeath]

so you want a shortcut for what can be already expressed as

def method(x:Option[Foo] = None) = deeper(x.orElse(myDefault))
def deeper(x:Option[Foo] = None) = deepest(x.orElse(anotherDefault))

[Justin du coeur]

On Thu, Feb 2, 2012 at 2:33 PM, HamsterofDeath <h-s...@gmx.de> wrote:

so you want a shortcut for what can be already expressed as

def method(x:Option[Foo] = None) = deeper(x.orElse(myDefault))
def deeper(x:Option[Foo] = None) = deepest(x.orElse(anotherDefault))

Visual clutter aside, that doesn't look right -- the entire point is that method() shouldn't need a "myDefault".  It wants to either take a specified value or delegate the problem to deeper(), and not inject any semantics of its own.  It wants to be a pure passthrough for default argument x.  This isn't an unusual problem, especially as libraries evolve.

And yes, this can certainly be worked around.  But I agree with Ryan that it seems to be a gap with default arguments, that there is no straightforward and clean way to pass the defaulting up the call chain without getting a bit messy in the middle...

[HamsterofDeath]

like so? :)

Am 02.02.2012 20:55, schrieb Justin du coeur:

On Thu, Feb 2, 2012 at 2:33 PM, HamsterofDeath <h-s...@gmx.de> wrote:

so you want a shortcut for what can be already expressed as

def outer(x:Option[Foo]) = method(x)

def method(x:Option[Foo] = None) = deeper(x.orElse(myDefault))
def deeper(x:Option[Foo] = None) = deepest(x.orElse(anotherDefault))

[Ryan Hendrickson]

The problem is: if you set your functions up to take Options, they *must* take Options. You get the power to do what you wrote, Hamster, which is very desirable; but then you can't call method or deeper or deepest like

method(myFoo)

if you have a Foo; instead you have to say

method(Some(myFoo))

where the Some is adding no information (if you didn't have Some(thing), you'd say method() instead).

To experienced Scala programmers, the Some may be just a little bit of annoying boilerplate clutter---the price of power, easily glossed over. But when designing an API targeting the less experienced, it becomes part of a design trade-off: do I create an API with overloads or default arguments, which is more intuitive for users but less powerful; or do I create one with Option parameters, which requires users to always use and be aware of the power of the Option? The proposal lets me have my cake and eat it too, optimizing syntactically for the more common use case of just having a Foo and wanting to use it, and creating a lightweight _? syntax for the power-user case of wanting to pass down an Option[Foo].

[Daniel Sobral]

On Thu, Feb 2, 2012 at 18:43, Ryan Hendrickson
<Ryan.Hen...@bwater.com> wrote:
> The problem is: if you set your functions up to take Options, they *must* take Options. You get the power to do what you wrote, Hamster, which is very desirable; but then you can't call method or deeper or deepest like
>
>  method(myFoo)
>
> if you have a Foo; instead you have to say
>
>  method(Some(myFoo))
>
> where the Some is adding no information (if you didn't have Some(thing), you'd say method() instead).
>
> To experienced Scala programmers, the Some may be just a little bit of annoying boilerplate clutter---the price of power, easily glossed over. But when designing an API targeting the less experienced, it becomes part of a design trade-off: do I create an API with overloads or default arguments, which is more intuitive for users but less powerful; or do I create one with Option parameters, which requires users to always use and be aware of the power of the Option? The proposal lets me have my cake and eat it too, optimizing syntactically for the more common use case of just having a Foo and wanting to use it, and creating a lightweight _? syntax for the power-user case of wanting to pass down an Option[Foo].

That's very unscala-ish. It goes in the opposite direction of
everything Scala tries to accomplish. Scala prides in providing
general tools instead of special cases. For example:

* Numbers and operators -- that's not special syntax, it's just
methods that any other class can provide.
* Array access -- that's not special syntax, it's just apply and
update, that any other class can provide.
* Array initialization -- that's not special syntax, it's just an
apply factory on the object companion, that any other class can
provide.
* for -- that's not special syntax, it's just map, flatMap, filter and
foreach, that any other class can provide.
* The App trait is implemented through DelayedInit.
* String interpolation is implemented through StringContext.

Yes, tuples and functions have special syntax for types and literals,
though I really wish tuples were unified with parameter lists, just
like functions were kind of unified with methods. I think that was the
original goal at any rate, and the point is turning things that are
not objects into objects -- like functions do with methods, and tuples
would have done with parameter lists.

Option is none of that. It's not a non-object that we wish to elevate
into objecthood status. It would make sense if Option was a
null-handler, like in some of the newer languages. I personally think
it would have been desirable, though treating all Java APIs as
returning Option or a primitive would have been rather inconvenient.
Nevertheless, that is not the case, so that isn't a justification.

Yes, Option and default parameters has a problem. That's very evident
in that a common use case is where one *never* passes None, only
Some(x). Making special cases, however, is *not* a way to handle the
problem. Instead, try to abstract the problem and think of a solution
for the general case. Think, for instance, if you could use the
solution with Either, Validation, Box or RedBlackTree.

--
Daniel C. Sobral

I travel to the future all the time.

[Ryan Hendrickson]

Out of curiosity, then, how do you feel about repeated parameters and the * syntax? Seems like all the same objections apply; Seq is also not a non-object needing to be elevated to object status. But I find it difficult to imagine Scala without *; how would you create collections without it (aside from List construction with ::, which I'm certainly not knocking, but all those CompanionObject.apply methods do exist after all...)?

(One point, perhaps, that * has over proposed ? is the need for ease of interop with Java and '...'. Not sure if that has anything to do with the thrust of your argument, though.)

In my opinion, what makes * a nice feature of Scala and not some Java-compatibility or collection-bootstrapping wart is that it's *not* just a special syntax for doing those two particular things; it's a generally-usable feature of parameter sections that anyone can use to make their functions simpler to call. That's the spirit I'm trying to capture with ?--it solves the particular problem of passing default arguments down through layers, but I think it certainly has more general applicability than that.

I don't know; I hear what you're saying about general tools vs. special cases, and I also love that principle and how well Scala has been able to implement it. I just don't currently see a use case here that's more general than Option--I don't know what the equivalent desired behavior is for Either or Validation--and the Scala spec already makes a special case of Option in some respects (namely, as the return type of unapply methods). (Don't get me wrong; I intend to do some hard thinking about whether there really is a more general thing here, and if it exists, I'm all for doing that instead of what I proposed, as long as it solves a superset of the problems ? solves.)

[Daniel Sobral]

Ah, I see! On first reading, I thought you were proposing XXX? as a
general synonym for Option[XXX] -- probably because many languages use
it like that (though without the full power of Option). My fault --
you are being clear enough, now that my blinders are off.

Now that I realize you are speaking _specifically_ about passing
parameters, I have completely reversed my opinion. I think this is a
pretty good and enabling solution. I do have suggestions to make,
though.

It would have been better if there was some interface that Option
implemented, just like Seq itself is an interface. Such an interface
could be shared by many classes similar to Option, such as, for
example, Parsers.ParserResult, and, taking it as a basic model, it
would implement a functor map, isEmpty, get and getOrElse, and it
could provide a general extractor as well. Given how much this expands
scope, I'd put it as an optional extension of the SID.

There's one place were _* appears that isn't contemplated by this:
extractors. A last term with type @ _? could stand for lastOption of
the remaining sequence.

I have one question, though. This seems to be allowed by the spec:

def f(x: Int?, y: String?, z: Double?) = ???

In that case, what would happen if it were called with f(1, 1.0)? I
assume it would be an error, but I didn't see that being made explicit
(though, given how I misread the proposal the first time, I'm almost
afraid of making this question! :)

[Naftoli Gugenheim]

Would it not help a lot just to have an "OptionOrRaw" trait (similar to Lift's BoxOrRaw, note this is not copied from there, so they may do it different):

case class OptionOrRaw[A](underlying: Option[A])

object OptionOrRaw {

  implicit def fromOption[A](o: Option[A]): OptionOrRaw[A] = OptionOrRaw(o)

  implicit def fromRaw[A](r: A): OptionOrRaw[A] = OptionOrRaw(Some(r))

  implicit def asOption[A](oor: OptionOrRaw[A]): Option[A] = o.underlying

[Naftoli Gugenheim]

To elaborate, this would allow you to define methods like

def m(optional: OptionOrRaw[String]) = ???

that you can call without Option boilerplate like

m("xxx")

[Ryan Hendrickson]

> Ah, I see! On first reading, I thought you were proposing XXX? as a
> general synonym for Option[XXX] -- probably because many languages use
> it like that (though without the full power of Option). My fault --
> you are being clear enough, now that my blinders are off.

Oh wow. Okay, now I understand. If I had proposed *that*, I'd have told me off too! Glad we're on the same page now. :-)

> It would have been better if there was some interface that Option
> implemented, just like Seq itself is an interface. Such an interface
> could be shared by many classes similar to Option, such as, for
> example, Parsers.ParserResult, and, taking it as a basic model, it
> would implement a functor map, isEmpty, get and getOrElse, and it
> could provide a general extractor as well. Given how much this expands
> scope, I'd put it as an optional extension of the SID.

Hmm. Not sure I really understand what you're going for here. If there were an OptionLike trait that Option implemented, and that were the type of ? parameters inside the function, then it's not like the function would be able to specify that it wants an Option or a ParserResult or a what have you. The compiler would still have to actually wrap the raw value in a Some, or provide a None if absent. So this would only affect what could be passed into such a function with the :_? syntax, right? And even then, it's a tradeoff between converting your ParserResult to an Option before passing it in with :_?, versus converting your OptionLike into an Option inside the function if you are going to do something with it that actually expects an Option. I don't know; if OptionLike already existed and had uses elsewhere in the Scala universe, I would probably go with it as the type of ? parameters. But since it doesn't, I don't think it's right to create it and try to promote it as a sort of rider on this proposal. Do you think I'm missing something or thinking about that poorly?

(Also, ParserResult seems more Eitherish than Optionish to me--and something that makes me okay with * using an open type whereas ? uses a sealed one is that there are lots of different ways to think about Seq, both semantically and in terms of performance characteristics, while I'm not yet aware of a strong need for more than one implementation of the concept of a zero-or-one 'collection'.)

> There's one place were _* appears that isn't contemplated by this:
> extractors. A last term with type @ _? could stand for lastOption of
> the remaining sequence.

Ah, extractors! I'm glad you reminded me of this; I rejected extractors early on in my thought process due to a technical limitation that I incorrectly believed to exist, and then didn't properly revisit when I realized my error.

So, I think that if @_? is going to be supported anywhere, it's most important to first nail down what it means with respect to, e.g.,

case class Foo(i: Int, x: String?)

before we talk about what it might mean for sequence extractors. That is, if the following isn't supported:

Foo(0, "blah") match {
case Foo(_) => println("A")
case Foo(_, x) => println("B: " + x)
}
// prints B: blah

Foo(0, "blah") match { case Foo(_, x@_?) => println(x) }
// prints Some(blah)

then I think it would be too surprising for @_? to have any meaning in any use case. Does that make sense?

So what does that entail? Well, it means that the unapply method generated for Foo, which would otherwise look like

def unapply(f: Foo): Option[(Int, Option[String])]

needs to have as part of its signature the information that the second element of the tuple type is an optional parameter and not a vanilla Option. (This is the thing that I incorrectly thought wouldn't be straightforward; I didn't realize that it would be just as easy to use a special type in the signature data for a method's return type as it would be for a parameter.) But that leaves a choice: either we now support ? syntax in the return type of a method (so that it's not just limited to a feature of parameter sections), like so:

def unapply(f: Foo): Option[(Int, String?)]

OR we allow the compiler to generate an unapply method for case classes that would be impossible to define in user code:

def unapply(f: Foo): Option[(Int, <optional>[String])] // not valid Scala

I don't really feel great about either of those choices, although I could see myself being convinced either way. (In particular, if I understand your point of view about parameter sections wanting to be unified with tuple types, arguably the first choice is the right thing to do, and maybe we limit the ? syntax to be permissible only in parameter sections and also tuple type syntax... but man, I feel like I'm walking on pretty shaky ground now.) What do you think?

> I have one question, though. This seems to be allowed by the spec:
>
> def f(x: Int?, y: String?, z: Double?) = ???
>
> In that case, what would happen if it were called with f(1, 1.0)? I
> assume it would be an error, but I didn't see that being made explicit
> (though, given how I misread the proposal the first time, I'm almost
> afraid of making this question! :)

Yes, the principle I'm going for here is that an optional parameter and a parameter with a default argument should be syntactically indistinguishable to the caller (although not binary-compatible, for somewhat self-evident reasons). So that should be an error if and only if

def f_prime(x: Int = -9999, y: String = "-9999", z: Double = -9999.0) = ???
f_prime(1, 1.0)

is also an error. (Which, as it happens, it is.)

By the same principle, f(1, z = 1.0) would be perfectly okay. And conversely, f_prime(1, None:_?, 1.0) should also work.

[Ryan Hendrickson]

> (Also, ParserResult seems more Eitherish than Optionish to me--and
> something that makes me okay with * using an open type whereas ? uses a
> sealed one is that there are lots of different ways to think about Seq,
> both semantically and in terms of performance characteristics, while I'm
> not yet aware of a strong need for more than one implementation of the
> concept of a zero-or-one 'collection'.)

Well, okay, I take this back somewhat: a LazyOption[A] with a LazySome[A](x: () => A) would certainly be another interesting variation on Optionness, and distinct from passing in a regular strict Option by name (since you could answer isEmpty without evaluating the thunk). (Hmm, does that exist already?)

[Ryan Hendrickson]

> Well, okay, I take this back somewhat: a LazyOption[A] with a
> LazySome[A](x: () => A) would certainly be another interesting variation
> on Optionness, and distinct from passing in a regular strict Option by
> name (since you could answer isEmpty without evaluating the thunk).
> (Hmm, does that exist already?)

Heh, except I guess nothing's stopping someone from just saying

type LazyOption[A] = Option[() => A]

if they wanted that... sooo... never mind? :-)

[Ryan Hendrickson]

> To elaborate, this would allow you to define methods like
> def m(optional: OptionOrRaw[String]) = ???
> that you can call without Option boilerplate like
> m("xxx")
>
>
> On Thu, Feb 2, 2012 at 10:40 PM, Naftoli Gugenheim
> <nafto...@gmail.com> wrote:
>
>
> Would it not help a lot just to have an "OptionOrRaw" trait
> (similar to Lift's BoxOrRaw, note this is not copied from there, so they
> may do it different):
>
> case class OptionOrRaw[A](underlying: Option[A])
> object OptionOrRaw {
> implicit def fromOption[A](o: Option[A]): OptionOrRaw[A] = OptionOrRaw(o)
> implicit def fromRaw[A](r: A): OptionOrRaw[A] = OptionOrRaw(Some(r))
> implicit def asOption[A](oor: OptionOrRaw[A]): Option[A] = o.underlying
> }

Right. So this is what I meant when I said that you could work around this with implicits and a dumb wrapper for an Option. Some comments:

* If you want to also be able to call m(), you still need to define m as

def m(optional: OptionOrRaw[String] = OptionOrRaw.None) = ???

I think that's a lot of boilerplate for a pretty common pattern, versus

def m(optional: String?) = ???

* The code you suggest has this problem:

def tricky[A](optional: OptionOrRaw[A]) = ...
tricky(Some(3)) // Do I mean tricky[Int], or tricky[Some[Int]]?

It's not actually a compiler error; the fromOption conversion is preferred. But to eliminate the ambiguity, you have to explicitly call tricky(OptionOrRaw(Some(3))) if that's the case you want... it works, but it's not an API I'd be proud of.

* If I'm writing an API like this, users are now seeing an OptionOrRaw type which they have to learn about. Since there is no standard one of these in Scala libraries or scalaz or anything, they have to learn it anew for each library that uses this pattern. And it's almost never a type they would want to think in terms of or carry around in any way; it exists just as a workaround to implement this optional parameter pattern. That makes me sad.

* This still doesn't address at all the capability that the second half of my proposal enables: if someone else has gone and defined a function with a default argument, and I am holding an Option which determines whether I want to use the default or not, I still have to do this:

myOpt match {
case Some(x) => otherFunction(a, b, c, d, x)
case None => otherFunction(a, b, c, d)
}

which is just awkward (and grows exponentially with the number of default arguments).

[nuttycom]

That's an interesting use of null that I hadn't thought of before, and I agree that it's a possible solution, but it's still just a convention that must continue to be obeyed by derived libraries.

I have to say that I like this a lot; I've seen the pattern of wanting to conditionally delegate to the default argument values of the callee on many occasions, and I think that having a standard way to do this be a part of the language would be a nice improvement. This way, it's up to the consumer of a function that takes default arguments whether they want to defer the choice; with the language as it currently exists the direct consumer must *always* make the choice in one way or another.

+1

[Daniel Sobral]

Briefly, and about both extractors and the Maybe trait (hah!), I think
they need to be optional/alternatives on the SIP, to incite
discussion. The string interpolation SIP got tons of comments, and in
its current incarnation it is much better than the original proposal
because of it.

--

[Tony Morris]

f(t: Option[T])
g(t: T) = f(Some(t))
h = f(None)

I do it all the time. It is the most appropriate solution. Objectors to
the new function names may like to take note that they are very
different functions; as different as any others.

--
Tony Morris
http://tmorris.net/

[Ryan Hendrickson]

> f(t: Option[T])
> g(t: T) = f(Some(t))
> h = f(None)
>
> I do it all the time. It is the most appropriate solution. Objectors to
> the new function names may like to take note that they are very
> different functions; as different as any others.

Doesn't scale. Is this the most appropriate solution:

class GuiWidget(backgroundColor: Option[Color], foregroundColor: Option[Color], fontFamily: Option[String], fontSize: Option[Int])

def makeWidget = new GuiWidget(None, None, None, None)
def makeWidgetWithBG(bg: Color) = new GuiWidget(Some(bg), None, None, None)
def makeWidgetWithFG(fg: Color) = new GuiWidget(None, Some(fg), None, None)
def makeWidgetWithBGAndFG(bg: Color, fg: Color) = new GuiWidget(Some(bg), Some(fg), None, None)
// and twelve* more very different functions...

This is a real problem, I think. Workarounds exist but none of them are elegant, and ignoring the problem by just passing in Options all the time is safe but obnoxiously redundant. I'm proposing a way for Scala to do better, reusing patterns and concepts already in the language, and without (I think) sacrificing much if any clarity or purity. I'm not completely sure from your email that you disagree with any of this, but if so, I'd like to hear you (or any lurking naysayer) explain why.

---

(*) And that's not even counting combinations of lifted and unlifted parameters, like

def makeWidgetWithOptionalFGAndFontFamilyAndFontSize(foregroundColor: Option[Color], fontFamily: String, fontSize: Int)

I don't think any reasonable person would propose defining all 81 combinations of {present, absent, Option'd}^4, but just remember that you essentially get all of them for the price of one with _? syntax, with the same mechanics that Scala users already have an intuition for with _*, and with none of the ambiguity that comes with various implicit workarounds.)

[Trond Olsen]

Those example looks like they need default values rather than optional ones.

[Gary Pampara]

Couldn't agree more. Default and named parameters go a long way to
stop you from overloading.

[Ryan Hendrickson]

> So, the problem I have with this argument is that I find it is
> indistinguishable from saying, "having many functions does not scale",
> then proceed to demonstrate that there exist many possible functions.
> The counter-argument to this is that these particular functions related
> to Option deserve special significance above any other, regular,
> distinct, completely distinct, different, unrelated functions. I argue
> that no such special significance is deserved. This is what I mean when
> I say, "take note that they are very different functions; as different
> as any others."
>
> By different here, I mean unrelated in any meaningful way. I have never
> seen it explained in a way that provides that meaning. Handwaving it
> away under the assumption that this significance is deserved by
> credulity, then proclaiming "doesn't scale" it not compelling -- not
> that I think you are doing this -- but I have seen it and I hope to
> avoid that situation so I mention it only for that reason.
>
> By the way, there is actually a really neat solution to improving the
> practical implications of this situation, but not in the general case,
> which is all we are discussing at this point.

Okay, good, thanks for clarifying. The 'very different' remark makes more sense now.

Let me start by pointing out that, in case the text of my proposal was too long-winded or not clear enough, I am not talking about changing the rules for every function that takes an Option as a parameter. You don't just get to walk up to a function that takes an Option[A] and treat it like it takes an A. The analogy is with the * in

def foo(values: Int*): String

Having that feature doesn't mean that every function that takes a Seq can be used with varargs syntax.

Imagine a Scala without *, temporarily. If you proposed adding it, and I were to reply that I work around this with

def f(t: Seq[T])
def g = f(Nil)
def h(t0: T) = f(t0 :: Nil)
def i(t0: T, t1: T) = f(t0 :: t1 :: Nil)

and thus don't see a need for *, I think it would be somewhat missing the point to talk about g, h, and i all being different unrelated functions (unless I am still misunderstanding your remark). Of course they are distinct, but the point is about making f simpler to call with less boilerplate syntactic baggage in the common case, not about any relations that might exist between functions constructed to work around the problem. Certainly not all functions that take a Seq benefit from using a * instead. But the ones that do, really do.

(Now that's not quite fair, obviously, because each * parameter would need to be supported by an infinite number of overloads in the absence of * to really get an equivalent effect, whereas as you and I have both demonstrated, the number required to match the behavior of a ? parameter is at least finite (though exponential in the count of such parameters). And also there are additional Java-compatibility reasons for supporting varargs. So certainly having ? is not as important as *. In my experience, though, uses for this feature would still be pretty common.)

We already have something that comes very close to removing this baggage: default arguments. But to use default arguments, the implementation of your function f: Option[A] => B must satisfy f(None) == f(a_def) for some (possibly computed) a_def in A, so that you can write f as

def f(a: A = a_def): B

The proposal is 50% just a clean way to remove that implementation restriction. (The other 50% is about what you give up when you rewrite f: Option[A] => B in this way, whether you use today's default arguments to do it or the proposed ? syntax: the ability to actually pass in an Option[A] when that's what you have, and not have to treat f like it actually is two distinct functions that you call in two branches of a fold-like control structure.)

[Ittay Dror]

Gary Pampara wrote:

Couldn't agree more. Default and named parameters go a long way to
stop you from overloading.

Indeed, the thread was started by showing how default & named parameters are not optimal when used by client code which also needs to keep the default values. Here it is: http://groups.google.com/group/scala-debate/browse_thread/thread/7fbd59907af20215

[Tony Morris]

On 07/02/12 12:12, Ryan Hendrickson wrote:

So, the problem I have with this argument is that I find it is
indistinguishable from saying, "having many functions does not scale",
then proceed to demonstrate that there exist many possible functions.
The counter-argument to this is that these particular functions related
to Option deserve special significance above any other, regular,
distinct, completely distinct, different, unrelated functions. I argue
that no such special significance is deserved. This is what I mean when

I say, "take note that they are very different functions; as different
as any others."

By different here, I mean unrelated in any meaningful way. I have never

seen it explained in a way that provides that meaning. Handwaving it
away under the assumption that this significance is deserved by
credulity, then proclaiming "doesn't scale" it not compelling -- not
that I think you are doing this -- but I have seen it and I hope to
avoid that situation so I mention it only for that reason.

By the way, there is actually a really neat solution to improving the
practical implications of this situation, but not in the general case,
which is all we are discussing at this point.

--
Tony Morris
http://tmorris.net/

[Adriaan Moors]

By the way, there is actually a really neat solution to improving the
practical implications of this situation, but not in the general case,
which is all we are discussing at this point.

I guess your (constructive) existence proof didn't fit in the margin of your email,

if you find some space somewhere to write it down, I'd be interested to see it.

[martin odersky]

It's an interesting proposal. Overall, I am not very keen about the :
_* concept. But it was necessary to do it because otherwise there
would have been no way to forward from a method that takes a sequence
to a method with varargs. Consider:

def f(x: T*) = ...

def g(x: Seq[T]) = // no way to forward this to f!

On the other hand, the way I understand it, the present proposal
introduces shorter syntax for something that can already be expressed:

def f(x: T = default)

def g(xo: Option[T]) = xo match {
case Some(x) => f(x)
case None => f()
}

That's a lot of boilerplate? Sure. But the point is, it can be done,
as opposed to the varargs situation. So the argument to add extra
syntax to handle this case is weaker.

Cheers

-- Martin

[Ittay Dror]

I think using 'f' and 'g' for the method names is misleading. In the general case, the methods will be
def makeWidget(background: Background = defaultBackground)

def makeWidgetWithOptions(background: Option[Background])

so you have two methods one of which with an awkward name. clients need to use the awkward name, which is not very DSLish.

I also like this feature because it is easy to sell. I think most non-Scala people will find this feature much more compelling than implicits. I think implicits are a much much more powerful feature, but conceptually harder to reason about and see the value. This feature completes handling of default values, which is a known issue in Java classes, and will be easier for people to relate to.

Cheers

 -- Martin

[Dennis Haupt]

> -------- Original-Nachricht --------
> Datum: Tue, 07 Feb 2012 14:53:45 +0200
> Von: Ittay Dror <ittay...@gmail.com>
> An: martin odersky <martin....@epfl.ch>
> CC: scala-...@googlegroups.com
> Betreff: Re: [scala-debate] Optional parameters (a language proposal)

i don't think so. after a few years of java, i wanted to use extension functions anywhere (without even having known the concept), but i almost never had a need for optional parameters. even now, i rarely use them for anything except factories and builders.
i never encountered your use case.

[Ryan Hendrickson]

> On the other hand, the way I understand it, the present proposal
> introduces shorter syntax for something that can already be expressed:
>
> def f(x: T = default)
>
> def g(xo: Option[T]) = xo match {
> case Some(x) => f(x)
> case None => f()
> }
>
> That's a lot of boilerplate? Sure. But the point is, it can be done,
> as opposed to the varargs situation. So the argument to add extra
> syntax to handle this case is weaker.

That's the meat of my argument for :_?, yes. The gravy is that, while finite, the amount of boilerplate required compounds with each new default argument. The equivalent g for

def f(t: T = deft, u: U = defu, v: V = defv, w: W = defw)

requires sixteen case clauses, all to express a concept that really 'ought' to be a one-liner, like almost any other function forwarding to another. (Apologies for using the handwavy word 'ought' to essentially beg the question. Always happy to elaborate if that sentiment doesn't resonate, but I don't want that elaboration to distract from the point if it does.)

So yeah, it can still be done (2^n < \infty), and so the argument is weaker. But for APIs that are going to want to use lots of default arguments, I don't know if it's that much weaker.

[martin odersky]

But it is not what Scala is about. Scala is emphatically not about
adding syntax to remove boilerplate. That's not in its genes. It's
about finding a small syntax kernel of powerful abstractions that
avoid boilerplate from the start. So, while adding features might be
an easy sell in the short run, it would dilute the idea of what Scala
is supposed to be.

Cheers

-- Martin

[Christopher Sachs]

On Feb 6, 6:12 pm, Ryan Hendrickson <Ryan.Hendrick...@bwater.com>
wrote:

> > f(t: Option[T])
> > g(t: T) = f(Some(t))
> > h = f(None)
>
> > I do it all the time. It is the most appropriate solution. Objectors to
> > the new function names may like to take note that they are very
> > different functions; as different as any others.
>
> Doesn't scale. Is this the most appropriate solution:
>
>   class GuiWidget(backgroundColor: Option[Color], foregroundColor: Option[Color], fontFamily: Option[String], fontSize: Option[Int])
>
>   def makeWidget = new GuiWidget(None, None, None, None)
>   def makeWidgetWithBG(bg: Color) = new GuiWidget(Some(bg), None, None, None)
>   def makeWidgetWithFG(fg: Color) = new GuiWidget(None, Some(fg), None, None)
>   def makeWidgetWithBGAndFG(bg: Color, fg: Color) = new GuiWidget(Some(bg), Some(fg), None, None)
>   // and twelve* more very different functions...
>
> This is a real problem, I think. Workarounds exist but none of them are elegant, and ignoring the problem by just passing in Options all the time is safe but obnoxiously redundant. I'm proposing a way for Scala to do better, reusing patterns and concepts already in the language, and without (I think) sacrificing much if any clarity or purity. I'm not completely sure from your email that you disagree with any of this, but if so, I'd like to hear you (or any lurking naysayer) explain why.

If you need this many parameters, why not encapsulate them in a
configuration object? In your example I'd rather create a Style object
first and then pass it to makeWidget. And you could use the same Style
object to create multiple widgets.

class GuiWidget(implicit style: Style = GuiWidget.defaultStyle)

object GuiWidget {
def defaultStyle: Style = ???
}

case class Style(backgroundColor: Option[Color], foregroundColor:

Option[Color], fontFamily: Option[String], fontSize: Option[Int]) {

def withBackgroundColor(bg: Color) = copy(backgroundColor =
Some(bg))
def withForegroundColor(fg: Color) = copy(foregroundColor =
Some(fg))
def withFontFamily(family: String) = copy(fontFamily = Some(family))
def withFontSize(size: Int) = copy(fontSize = Some(size))
}

Then you would create widgets by modifying the class's default style.

makeWidget(GuiWidget.defaultStyle.withBackgroundColor(_).withFontFamily(_))

In this case I think a widget's default style *should* be exposed--
it's not just an implementation detail. I would also mark the style
parameter to GuiWidget as implicit so that you can (optionally!)
introduce a local implicit style and have it picked up by all your
makeWidget calls.

implicit def largeTypeStyle = GuiWidget.defaultStyle.withFontSize(48)
makeFooWidget()
makeBarWidget()

Factoring out configuration parameters makes it much easier to evolve
your code over time and lets you do things like "cascade"
configuration objects if you need that flexibility.

I'm curious why you might prefer overloaded method calls to this
approach?

--
Chris Sachs

[Ryan Hendrickson]

Oh, I don't! I like that approach a lot. It isn't always appropriate (I wouldn't want to build a configuration object for every method with default arguments), but as you imply, a widget's style is enough of a thing on its own that for this example this would work quite well. And it has the advantage of scaling linearly with the size of the configuration object, unlike some of these other techniques we've been kicking around.

The sixteen overloaded makeWidget methods were, in hindsight, a strawman I put forth because I thought Tony was proposing that any time I wanted to make it easier for my API's users to call a function with Options or with raw arguments, I define additional functions to facilitate those cases. It's something I might do if I had only one parameter, but certainly not something I'd do for anything more complex.

Anyway: given that Scala does have default arguments, which are very useful and clean, not every library author is going to use a configuration object when a default argument would do. And every time a library author makes that choice, if I'm using that library, I still have the problem of how to call that function if I want to dynamically choose between taking the default or provide an argument. The advantage of making this a language feature is having a standard solution to this problem that works with the language features and patterns that people are already using and used to. (Not that I'm saying that that's the necessary and sufficient condition for something to become a language feature; just that it represents the unique advantage that the proposal has over any other technique by virtue of being a language feature.)

[Ittay Dror]

On Feb 7, 6:54 pm, martin odersky <martin.oder...@epfl.ch> wrote:

I don't see this differently than case classes & pattern matching.
both can be "hand coded".

You once wrote a blog about the levels of Scala where you
distinguished between application programmers and library designers.
While the small syntax kernel appeals to library designers, optional
parameters will appeal to application programmers,. The thing is, to
make Scala popular, you need to appeal to application programmers
first, since they form the majority (because libraries are there to be
used by many). E.g., I love how scala unifies the handling of arrays,
strings and collections. but in all honesty, i've never, ever, heard
anyone complaining about the fact they are separate things in Java.
People accept this and move on.

I want such features to be the bait that attracts people to Scala.

>
> Cheers
>
>  -- Martin

[Kevin Wright]

Strings, not so much.

But I've often grumbled about the impedance barrier between Arrays and collections in Java.  The same thing goes for Enumeration and Iterator.

[Ittay Dror]

Kevin Wright wrote:

Strings, not so much.

But I've often grumbled about the impedance barrier between Arrays and collections in Java.  The same thing goes for Enumeration and Iterator.

people that already love scala don't count. that's the whole point ;-)

[Kevin Wright]

Oh no, these are things I grumbled about before I was using Scala.

They number amongst the reasons why I migrated in the first place.