Reasoning Behind Box
Daniel Spiewak
rationale for the Box ADT. I'm most distressed by the fact that it
seems to be masquerading as a drop-in Option replacement, and yet the
mathematical properties of the ADT are widely divergent. What's more,
the API is very, very different, leading to a great deal of confusion
whenever I'm working with code which touches both ADTs (as I often
do). Things like `or` vs `orElse`, `open_!` vs `get` and so on are
very confusing. The implicit conversion Box[A] => Option[A] doesn't
help either, since it means I can easily forget and use Option methods
on Box values, accidentally triggering a conversion.
The whole `or` vs `orElse` thing is particularly annoying since I
spend a fair amount of time wading through research which discusses
the `orElse` function in its abstract monadic sense. Given the
standard nature of the name (dating back long before Scala), I'm
surprised that Box went with something else.
I'm sure there are good reasons for all of this, I would just like to
know what they are. :-)
Daniel
Naftoli Gugenheim
I guess open_! is in keeping with the metaphor of a box (or originally, a can). The _! is Lift's way of saying, Danger! And I guess 'or' is just shorter. (Lift tends to put practicality before academic functionalness etc.)
-------------------------------------
Daniel Spiewak<djsp...@gmail.com> wrote:
Daniel
--
You received this message because you are subscribed to the Google Groups "Lift" group.
To post to this group, send email to lif...@googlegroups.com.
To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/liftweb?hl=en.
Heiko Seeberger
--
You received this message because you are subscribed to the Google Groups "Lift" group.
To post to this group, send email to lif...@googlegroups.com.
To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/liftweb?hl=en.
--
Heiko Seeberger
Company: weiglewilczek.com
Blog: heikoseeberger.name
Follow me: twitter.com/hseeberger
OSGi on Scala: scalamodules.org
Lift, the simply functional web framework: liftweb.net
Naftoli Gugenheim
I'm not so sure David will want to rewrite the entire lift anyway...
-------------------------------------
Heiko Seeberger<heiko.s...@googlemail.com> wrote:
Daniel,
Heiko
> liftweb+u...@googlegroups.com<liftweb%2Bunsu...@googlegroups.com>
Heiko Seeberger
Either -- but it's more verbose.
I'm not so sure David will want to rewrite the entire lift anyway...
Daniel Spiewak
that it would be a lot more verbose. I would argue that the
difference isn't *so* substantial if you sprinkle in a type alias and
some implicit conversions (so that you can map/flatMap/filter over
something of type Either[Option[A], String]).
I am aware of the Failure case, and that's why I argue that Box
*isn't* a drop-in Option replacement. API naming differences are
inconsequential, but adding another case to the ADT is a pretty
dramatic difference. However, Lift seems to use Box *everywhere*,
even in places where it is only using it as if it were Option.
Daniel
On Feb 26, 1:17 am, Heiko Seeberger <heiko.seeber...@googlemail.com>
wrote:
Marius
1. A rather richer API than Option
2. The Failure case that you are already aware of
=> a logical existence of Box-ification.
I fail to see what is the problem that we use Box and not Option
especially that you can seamlessly convert one to the other. Is there
a law that we should have been using Option? Are the methods naming
differences so dramatic that induces such confusion ? - I do not feel
this in practice.
It sometimes seems to me that people are view-ing Option as an
absolute term - a complete Maybe monad that everyone should obey.
Br's,
Marius
David Pollak
Thanks for asking the question and I hope the discussion will go better than the last time we had an in depth discussion on the topic. I'm going to respond in a Prologue, History, Reactions, Code Examples and Conclusion.
Prologue
I'm not a classically trained CS guy. I've never taken an FP-related CS course in my life. I've only done any real FP-related coding since November 2006. I've only taken 3 CS courses in my entire life (360 assembler [almost got thrown out because the teacher hated the fact that I always found bugs in her homework assignments during the class in which she handed out the assignments], data structures and algorithms [got accused of cheating on mid-term because my sorting algorithm ran 10x faster than the next fastest... turns out I randomized the incoming data {avoids O(n^2) issue in QuickSort} and created faux-pointers and sorted those rather than the structures {this was in Pascal}], and a database theory class [almost got thrown out because the comments in my code were considered "unprofessional"... rescued my grade by writing a hand SQL parser rather than using Flex/Bison]). The reason I say this is that I'm not going to discuss ADTs, mathematical properties, or anything else. I'm not qualified or capable of having that kind of discussion.
I'm also lazy and pragmatic. If a library works for me, I use it. If a library requires a lot of working-around, I ask for fixes and if I don't get them, I write my own. I did this with the NextStep Text object... it didn't have what I needed for Mesa, the Next guys weren't happy about giving what I needed... I wrote my own... then the Next guys rolled a lot of my concepts into the NS Text object. The same thing happened with the NextStep font chooser... in fact, Mesa's font chooser was the first piece of open source software I wrote... but I digress.
History
Back when I was the only Lift committer (okay, maybe SteveJ had commit rights back then... I don't remember) and I was working on the first Lift-based app, I was experiencing a nasty issue. This was summer of 2007. This was circa Scala 2.4 and before Either was in the language. One of the common patterns I kept running across was iterating through a series of pieces of user-input (both from the UI and from the REST interfaces) and having to return a not-answer (None) and the reason for the inability to complete the operation. It was about the same time, moved away from pattern matching against Options in favor of using Options in for comprehensions. The first thing I tried was to use Options in conjunction with exceptions (yeah, I was still lost in Java-think) such that in the case of a None for a particular operation (loading an item from the database, etc.), I'd throw an exception, the caller would catch the exception and present the right thing to the user/caller. It was ugly.
I posted to the Scala mailing list and, well... there were not a lot of good suggestions other than use Either (e.g., write my own Either library). The problem with Either is that it was not usable in for comprehensions and pattern matching was becoming increasingly deprecated. One of our advisors (http://www.lukehohmann.com/ ) was particularly keen on keeping a low McCabe complexity number for code (see http://en.wikipedia.org/wiki/Cyclomatic_complexity ). For comprehensions offered a very low complexity where pattern matching explodes complexity.
So, I finally bit the bullet and wrote my own set of classes that offered the kind of functionality that I was cobbling together with Option. I originally called the class Can (a short name for something that contains or does container a unit of something... a can of soup, a can of beans.) I patterned Can after Option (btw, Option is one of the worst names in the Scala libraries... explaining what an Option is to a Java or Ruby guy was one of my biggest stumbling blocks... Maybe would have been a much better name, but I digress.)
Can became a better Option. In addition to having Full/Empty (Some/None), it also had Failure... a subclass that could contain information as to why the Can did not contain anything. From an object hierarchy standpoint, Can and Option are identical at the first children: Full == Some. EmptyCan == None. The other problems I saw with Option were:
- The get method was too easy for Java (and Ruby) developers to "get wrong". Get is *FREAKING DANGEROUS*. It's an exception waiting to happen. It should not be used except in the most extreme situation. But with a nice, tasty, comforting, access-worthy name like "get", it just begs to be invoked. Something like 30% of our production-time defects resulted from mis-using Option.get. This needed to be changed... so something that says "danger" like "open_!" Oh, yeah, the "!" says "hmmm... why is that ! there? Should I be using this method?"
- orElse orElse orElse... who named this method. In terms of method naming the "Else" is a useless waste of 4 characters... the camel case takes the eye away from the things that are being tested... there's no reason for the Else part.
- Once I was fixing the naming problems with Option, I spent a lot of time working on getting the names to be more concise and descriptive.
- Change from Can to the much better name: Box
- Adding the FailureParam subclass such that more information (e.g., the HTTP return code) could be included in the Failure.
Reactions
I proposed to the Scala team that they include the Failure subclass as part of Option. They declined. Mainly for two practical reasons: thousands of Some/None pattern matches would be broken and Martin has always wanted to optimize Option at the compiler level to be ref/null so there would be no necessary boxing of Options. I took another run at Martin with the idea 14 months ago when 2.8 was just on the drawing board and Martin once again declined.
Most FP trained folks who have looked at Box have vomited over the mathematical differences between Option and Box. Tony Morris is the most notable example of this... he was so unable to control his rage at the abomination that is Box that we had to ban him from the Lift list (he's the first of two people that share that dubious honor).
PaulP almost vomited his sushi all over the table one day when we were discussing Box. He offered a suggestion that map/flatMap/foreach/filter be added to Either so the Left side would continue to process and the Right side would not. This feature was not added to 2.8 through Beta1.
Many newbies are confused by the existence of Box and Option. Many folks, like you, don't like the dichotomy. In a perfect world, the Scala team would adopt Box and deprecate Option... but the world is not perfect.
Code Examples
I hope this will help you understand the use cases for Box and the reason that everything in Lift returns a Box (except for a couple of places where the compiler requires Option... e.g. parameters in XML literals).
user <- User.currentUser
req <- S.request
id <- S.param("id")
idNum <- tryo(id.toLong)
record <- Model.find(idNum) if user.canView(record)
respType = calcRespType(req) openOr XmlType
resp <- record.formatAs(respType)
} yield resp
Now, the above code is nice, clean, has a low McCabe number and could be written with either Option or Box. But, if we want to add data about why the request could not be completed, we can just do:
user <- User.currentUser ?~ "No session" ~> 401
req <- S.request ?~ "Outside of request" ~> 500
id <- S.param("id") ?~ "Parameter 'id' missing"
idNum <- tryo(id.toLong) ?~! "Parameter 'id' invalid"
record <- Model.find(id).filter(user.canView) ?~ "Model not found"
respType = calcRespType(req) openOr XmlType
resp <- record.formatAs(respType) ?~ "Unable to format" ~> 500
} yield resp
The above code, which has a low McCabe number and is readable, yet it is the very essence of a response. Not just the content of the response, but the reason for the lack of a response.
It's not feasible to change Option to Box "only when you need it" because you almost always need it... or at least I do.
I use Box in Goat Rodeo because it's allows me to pass a ton more information about why an asynchronous operation did not result in the expected data. Having Failure and FailureParam allows for sending back all manner of information including timeout information, transaction rollback information, etc.
I could go on all day about where/how I use Box in my code, but I don't think it's an exaggeration to say that in my code, every place where there'd be a checked exception in Java, there's a Box. That means that in my app-level code, 50%+ of my methods use Box in some way or another... and in ways that Option cannot be used.
Conclusion
I am sorry that Lift's Box frustrates you... and I do not mean to minimize your level of frustration. Daniel, I have a ton of respect for you in terms of your coding, your questions, your understanding of a lot of CS concepts, and your excellent blog posts. But, I ask, borrowing a line from Caddy Shack, "Danny... be the Box..." Abandon your notions of Option/Maybe. Embrace Box. Upcast from Option to Box at the earliest possible time in your code. Noodle on it for 6 or so weeks... then I'm betting you'll ask the Scala team why they don't deprecate Option and replace it with Box.
Thanks,
David
PS -- if you want us to add orElse and a couple of other methods (other than get) to Box, I'm all for that. I don't want Box's improved (IMHO) names to cause consternation.
--
You received this message because you are subscribed to the Google Groups "Lift" group.
To post to this group, send email to lif...@googlegroups.com.
To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/liftweb?hl=en.
--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Surf the harmonics
Daniel Spiewak
> Back when I was the only Lift committer (okay, maybe SteveJ had commit
> rights back then... I don't remember) and I was working on the first
> Lift-based app, I was experiencing a nasty issue. This was summer of 2007.
> This was circa Scala 2.4 and before Either was in the language. One of the
> common patterns I kept running across was iterating through a series of
> pieces of user-input (both from the UI and from the REST interfaces) and
> having to return a not-answer (None) and the reason for the inability to
> complete the operation. It was about the same time, moved away from pattern
> matching against Options in favor of using Options in for comprehensions.
> The first thing I tried was to use Options in conjunction with exceptions
> (yeah, I was still lost in Java-think) such that in the case of a None for a
> particular operation (loading an item from the database, etc.), I'd throw an
> exception, the caller would catch the exception and present the right thing
> to the user/caller. It was ugly.
A valid point. I would argue that this is the "correct" way to solve
this problem, but it's obviously not the cleanest.
> So, I finally bit the bullet and wrote my own set of classes that offered
> the kind of functionality that I was cobbling together with Option. I
> originally called the class Can (a short name for something that contains or
> does container a unit of something... a can of soup, a can of beans.) I
> patterned Can after Option (btw, Option is one of the worst names in the
> Scala libraries... explaining what an Option is to a Java or Ruby guy was
> one of my biggest stumbling blocks... Maybe would have been a much better
> name, but I digress.)
+1 For the record, the Option name comes from ML, while Maybe comes
from Haskell (and its heritage). As I recall, Martin is a great fan
of ML, which is why most of Scala's functional features are heavily
inspired by that language. In fact, the only functional feature I can
think of which Scala got from Haskell would be typeclasses.
> - The get method was too easy for Java (and Ruby) developers to "get
> wrong". Get is *FREAKING DANGEROUS*. It's an exception waiting to happen.
> It should not be used except in the most extreme situation. But with a
> nice, tasty, comforting, access-worthy name like "get", it just begs to be
> invoked. Something like 30% of our production-time defects resulted from
> mis-using Option.get. This needed to be changed... so something that says
> "danger" like "open_!" Oh, yeah, the "!" says "hmmm... why is that !
> there? Should I be using this method?"
Agreed. This is maybe an education thing. My biggest problem with
open_! is the highly unidiomatic use of mixed alpha-symbolic method
name. Yes, I know Lift does it everywhere, but I can't think of any
other Scala library which does the same. Maybe I'm being too
pedantic, but I cringe every time I see it in my code. :-)
> - orElse orElse orElse... who named this method. In terms of method
> naming the "Else" is a useless waste of 4 characters... the camel case takes
> the eye away from the things that are being tested... there's no reason for
> the Else part.
The name for this method was devised a *long* time ago. I don't
remember exactly when, but it was a part of research examining monads
in the abstract mathematical sense. As a result, the goal was to name
the function something which would be unambiguous in the proverbial
"global context". They probably thought of "or", but that's not
specific enough. Thus, orElse. Since almost all languages which have
monads also use Hindley-Milner type inference, it's not surprising
that the name has stuck. Scala is really one of the few languages
which *can* go with something else.
> I proposed to the Scala team that they include the Failure subclass as part
> of Option. They declined. Mainly for two practical reasons: thousands of
> Some/None pattern matches would be broken and Martin has always wanted to
> optimize Option at the compiler level to be ref/null so there would be no
> necessary boxing of Options. I took another run at Martin with the idea 14
> months ago when 2.8 was just on the drawing board and Martin once again
> declined.
I have to say, I'm with Martin on this one.
> Most FP trained folks who have looked at Box have vomited over the
> mathematical differences between Option and Box. Tony Morris is the most
> notable example of this... he was so unable to control his rage at the
> abomination that is Box that we had to ban him from the Lift list (he's the
> first of two people that share that dubious honor).
I'm in rather interesting company, since I had a similar reaction when
I first saw Box. Maybe not quite so violent, but in that general
vein. To be clear, I have absolutely no problem with it as a separate
ADT unto itself. In fact, were I in your shoes, I may have even come
up with something similar. However, I would have made it a proper ADT
with four instances: Full(...), Empty, Failure(...) and
ParamFailure(...).
There are two aspects of Box which would unsettle a FP purist:
- The use of inheritance to break the ADT properties (Empty & Failure
<: EmptyBox, and ParamFailure <: Failure)
- The fact that it is billed as a "superior Option". It's not Option,
it's Box. This distinction is likely why it was refused inclusion
into Scala 2.8.
To repeat: there's nothing wrong with Box in and of itself, but I
prefer if it were a four instance ADT rather than a bizarre hybrid-
inherited thingy. There's really no changing that now though without
breaking API compatibility, so I'm not advocating any sort of
modification. What's done is done.
> PaulP almost vomited his sushi all over the table one day when we were
> discussing Box. He offered a suggestion that map/flatMap/foreach/filter be
> added to Either so the Left side would continue to process and the Right
> side would not. This feature was not added to 2.8 through Beta1.
I wouldn't add map/flatMap/foreach to Either, since it seems a bit
contrived. However, it might be interesting if, instead of Box, you
used something like this (re-using the Can name for clarity):
type Can[A] = Either[Option[A], String]
object Can {
def ! = if (a != null) Left(Some(a)) else Left(None)
}
implicit def canSyntax[A](can: Can[A]) = new {
def map[B](f: A => B) = can match {
case Left(opt) => Left(opt map f)
case Right(str) => Right(str)
}
case flatMap[B](f: A => Can[B]) = can match {
case Left(Some(x)) => f(x)
case Left(None) => Left(None)
case Right(str) => Right(str)
}
case filter(f: A => Boolean) = can match {
case Left(opt) => Left(opt filter f)
case Right(str) => Right(str)
}
}
You could add some more syntactic sugar in the form of factory
methods, deconstructors, etc. In the end, you would have something
that would be just as syntactically convenient as Box without
sacrificing the mathematical uniformity of a composite Either+Option.
> Many newbies are confused by the existence of Box and Option. Many folks,
> like you, don't like the dichotomy. In a perfect world, the Scala team
> would adopt Box and deprecate Option... but the world is not perfect.
I'm not sure I agree with your definition of "perfect", but ok. :-)
> Now, the above code is nice, clean, has a low McCabe number and could be
> written with either Option or Box. But, if we want to add data about why
> the request could not be completed, we can just do:
>
> for {
> user <- User.currentUser ?~ "No session" ~> 401
> req <- S.request ?~ "Outside of request" ~> 500
> id <- S.param("id") ?~ "Parameter 'id' missing"
> idNum <- tryo(id.toLong) ?~! "Parameter 'id' invalid"
> record <- Model.find(id).filter(user.canView) ?~ "Model not found"
> respType = calcRespType(req) openOr XmlType
> resp <- record.formatAs(respType) ?~ "Unable to format" ~> 500
>
> } yield resp
>
> The above code, which has a low McCabe number and is readable, yet it is the
> very essence of a response. Not just the content of the response, but the
> reason for the lack of a response.
To me, this looks like a lot of mysterious symbols and operators,
something I would have a hard time remembering without looking at the
documentation. I understand the desire for a cleaner usage API
though, so I won't begrudge it.
> I am sorry that Lift's Box frustrates you... and I do not mean to minimize
> your level of frustration. Daniel, I have a ton of respect for you in terms
> of your coding, your questions, your understanding of a lot of CS concepts,
> and your excellent blog posts. But, I ask, borrowing a line from Caddy
> Shack, "Danny... be the Box..." Abandon your notions of Option/Maybe.
> Embrace Box. Upcast from Option to Box at the earliest possible time in
> your code. Noodle on it for 6 or so weeks... then I'm betting you'll ask
> the Scala team why they don't deprecate Option and replace it with Box.
I have been noodling with it, and frankly I still use Option whenever
I can get away with it. I outlined my reasons above, so I won't waste
your time by repeating them. :-) I will freely admit that my dislike
stems more from the mathematical than the practical. Neither my
reasons nor my hypothetical solution (Either[Option[A], String]) are
rooted in any sort of pragmatics, so maybe it's best to ignore me as
just another raving FP lunatic. :-)
At the end of the day, Box is what it is. I'm glad I have a better
understanding of why it exists, and while I still (vehemently)
disagree with its design, I respect the process which created it.
Just be aware that, even as I am not the first FP purist to be annoyed
by this, I certainly won't be the last. I'm sure this will be an
ongoing discussion for as long as Lift endures, or at least as long as
FP renegades find their way into Lift's API.
> PS -- if you want us to add orElse and a couple of other methods (other than
> get) to Box, I'm all for that. I don't want Box's improved (IMHO) names to
> cause consternation.
I really would prefer orElse, getOrElse and friends. However, the
last thing I want is for you to clutter Lift's API with useless
aliases just because a small minority (i.e. myself and Tony Morris)
seem to feel it's necessary. :-) Stick with what's working and I'll
learn to live with it.
I would however *prefer* (and this is very much a personal preference)
that the implicit conversions to/from Option be deprecated. Well,
maybe it's ok to convert Option[A] => Box[A], but not the other way
around. This has just caused me too much confusion, especially
lately. I'm a firm believer that implicit conversions to pre-
established types should not be implicit at all, but explicit (Jorge
Ortiz's java-utils framework hit the nail perfectly on the head in
this department).
Daniel
Daniel Spiewak
> absolute term - a complete Maybe monad that everyone should obey.
Yeah, that's pretty much it. :-) Saying that you have a replacement
Option with totally different instances is like telling me that you
have a new definition for a derivative. Maybe/Option are just
implementations of the same abstract mathematical concept (like two
different ways of writing the *same* definition of a derivative). Box
is something which is obviously inspired by the Maybe monad, but seems
to have gone in a totally new direction with it. I'm not saying that
direction is bad, but it's certainly not anything like Maybe. It's
like looking at the problem of computing instantaneous rate of change
and coming up with a technique which maps things into a different
complexity domain. Your results may be valid, but you're certainly
not going to end up with differential calculus as we know it.
Daniel