Future and Option in for comprehensions

[Raul Raja]

I find myself in many cases where Future[Option[_]] is needed to pipe results from long running ops into other long running ops.

e.g. ReactiveMongo or any async driver where you need to query with the results of a first query that returns a Future[Option[_]]

I can achieve the final result with nested flatMaps and I have also looked into OptionT in Scalaz and a approach similar to this one https://github.com/twitter/storehaus/blob/develop/storehaus-core/src/main/scala/com/twitter/storehaus/FutureOps.scala.

But, Is there a way to combine the Option and Future monads in for comprehension like in the following example?

for {

  maybeUser <- getUser()  //Option[User]

  user <- maybeUser() //User

  messages <- getUserMessages(user) //Future[List[Message]]

} yield ...

The point is that the Option and Future monads do not combine well and this case results into nested Future[Option[Future]]

Any help is appreciated,

Thx

[Luis Ángel Vicente Sánchez]

Hi Raul,

I think that the OptionT approach is you best alternative. Why have you discarded that option?

Kind regards,

Luis

2013/8/11 Raul Raja <raul...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Raul Raja Martinez]

Didn't discarded it, just wondering if there was a better approach. Seems inconsistent you can throw monadic structures at for comprehensions but they don't always combine as expected based on the ones you combine. e.g. Option and List are fine together

[Som Snytt]

> Option and List are fine together

I once thought that about someone, and then it turned out the relationship was entirely one-sided.

This is because collections are cool and everyone wants to play with the cool crowd.

Although it's been said many times and many ways,

scala> val is = List(1,2,3)
is: List[Int] = List(1, 2, 3)

scala> val op = Some(4)
op: Some[Int] = Some(4)

scala> for (i <- is; o <- op) yield (i,o)
res0: List[(Int, Int)] = List((1,4), (2,4), (3,4))

scala> for (o <- op; i <- is) yield (i,o)
<console>:10: error: type mismatch;
 found   : List[(Int, Int)]
 required: Option[?]
              for (o <- op; i <- is) yield (i,o)
                              ^

[Raul Raja]

Som, thanks for the clever analogy. I have seen that case before.

Luis I will look into OptionT more in deep to achieve what I'm looking for.

I'm still of the opinion that for comprehensions while powerful is not obvious by looking at them what the result would be.

Asking other Scala devs outside of this list, they seem to find awkward how the different types may be used in a single comprehension yielding dispare results or resulting in cryptic compilation errors like in Som's example.

At a first glance it almost seems like for comprehensions may be used to chain operations and so it's expected by other devs when using them regardless of the monads used in the statement.

This example in SO is a clear case where in the answer you end up using OptionT for such use case

http://stackoverflow.com/questions/14385633/futureoption-in-scala-for-comprehensions

Thanks everyone for your help.

[Kevin Wright]

On 12 August 2013 09:31, Raul Raja <raul...@gmail.com> wrote:

Som, thanks for the clever analogy. I have seen that case before.

Luis I will look into OptionT more in deep to achieve what I'm looking for.

I'm still of the opinion that for comprehensions while powerful is not obvious by looking at them what the result would be.

Asking other Scala devs outside of this list, they seem to find awkward how the different types may be used in a single comprehension yielding dispare results or resulting in cryptic compilation errors like in Som's example.

Definitely confusing if you think about it like this, it's probably not the best way to help get a good intuition. :)

Think of the comprehension as working with just ONE kind of thing.  In the case of List/Option you can look at it as working with Iterables, which Option implicitly converts to.

This only works if the first generator in the comprehension is a collection type, or you explicitly call toList on the option (or otherwise convert it via type ascription, etc.)

Broadly speaking, look at the first generator and what type it expects for flatMap - all terms in the comprehension will be of that type.

--
Kevin Wright
mail: kevin....@scalatechnology.com
gtalk / msn : kev.lee...@gmail.com

quora: http://www.quora.com/Kevin-Wright

google+: http://gplus.to/thecoda

twitter: @thecoda

vibe / skype: kev.lee.wright

steam: kev_lee_wright

"My point today is that, if we wish to count lines of code, we should not regard them as "lines produced" but as "lines spent": the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger" ~ Dijkstra

[Oliver Ruebenacker]

Hello,

Seems to me any time some one explains what an Option is, they will
use the phrase "like a collection", causing confusion over how
collectionish an Option is.

Wouldn't it make it easier to understand for every one if Option
actually was a collection?

Take care
Oliver

Head of Systems Biology Task Force at PanGenX (http://www.pangenx.com)
Any sufficiently advanced technology is indistinguishable from magic.

[Kevin Wright]

On a theoretical level, yes, it would be easier.

On a practical level, it can't be done, at least not yet.

I've tried it, Paulp has tried it, for all I know even Martin has tried it.  Everyone I know who looked into this has come to the same conclusion: It can't be done without royally screwing binary compatibility, so it ain't happenin' at the moment.

[Oliver Ruebenacker]

Hello,

On Tue, Aug 13, 2013 at 4:54 AM, Kevin Wright <kev.lee...@gmail.com> wrote:
> On a theoretical level, yes, it would be easier.
>
> On a practical level, it can't be done, at least not yet.
>
> I've tried it, Paulp has tried it, for all I know even Martin has tried it.
> Everyone I know who looked into this has come to the same conclusion: It
> can't be done without royally screwing binary compatibility, so it ain't
> happenin' at the moment.

Interesting. Why is it so hard? What happens if you try?

Thanks!

Take care
Oliver

[Jason Zaugg]

On Tue, Aug 13, 2013 at 2:54 PM, Oliver Ruebenacker <cur...@gmail.com> wrote:

     Hello,

On Tue, Aug 13, 2013 at 4:54 AM, Kevin Wright <kev.lee...@gmail.com> wrote:
> On a theoretical level, yes, it would be easier.
>
> On a practical level, it can't be done, at least not yet.
>
> I've tried it, Paulp has tried it, for all I know even Martin has tried it.
> Everyone I know who looked into this has come to the same conclusion: It
> can't be done without royally screwing binary compatibility, so it ain't
> happenin' at the moment.

  Interesting. Why is it so hard? What happens if you try?

http://www.scala-lang.org/old/node/5903.html

-jason 

[Oliver Ruebenacker]

Hello,

What's wrong with Option.flatMap returning List?

[Kris Nuttycombe]

Consider, for example, the fact that map can be implemented in terms of flatMap and the Some constructor. flatMap is the more general operation you should be concerned with.

On Tue, Aug 13, 2013 at 1:40 PM, Kris Nuttycombe <kris.nu...@gmail.com> wrote:

Well, flatMap is a rather critically important method with respect to the monadic effect I'm talking about! 

On Tue, Aug 13, 2013 at 9:07 AM, Oliver Ruebenacker <cur...@gmail.com> wrote:

     Hello,

On Tue, Aug 13, 2013 at 10:34 AM, Kris Nuttycombe
<kris.nu...@gmail.com> wrote:
> The problem is the semantics of the monadic effect; the effect of Option is
> to impose zero-or-one semantics on the value in its context, whereas the
> effect of List is to impose zero-or-n semantics. These are not the same, any
> more than zero-or-one is the same as present-or-future (the Future monad) or
> at-least-one (NonEmptyList).
>
> Think of it this way: Option, by presenting a zero-or-one context, is
> promising its caller that there will never be more than one value. Allowing
> Option#flatMap to return List would make it impossible to enforce that
> contract through a sequence of operations. In short, it's a different type
> for a reason; if you want it to behave like a List, there's always .toList,
> which is nicely explicit.

  It makes perfect sense to me that, for example, Option.map should
return Option. But not flatMap. Certainly, we can't impose the
requirement that any method of Option returns Option. Why not let
flatMap be one of those that don't?

[Oliver Ruebenacker]

Hello,

Ah, my bad, I missed some subtlety. There are really two flatMap
methods, the one from Iterable and the one from Option. If Option
would become an Iterable, it would have both. And I think it should
be:

Option.flatMap[B](f: (A) => Option[B]): Option[B]
Option.flatMap[B](f: (A) => GenTraversableOnce[B]): Iterable[B]

If Option was an Iterable, it would also be a GenTraversableOnce.
Would that cause ambiguities?

Could we rename the current Option.flatMap to avoid collision with
Iterable.flatMap?

Take care
Oliver

On Tue, Aug 13, 2013 at 3:43 PM, Kris Nuttycombe

[Som Snytt]

> It can't be done without royally screwing binary compatibility.

This goes back to my previous analogy.

I still have an old gray sweatshirt in the bottom drawer that I take out occasionally just to reminisce.

It says, in bold font and early-aughts hubris: Iterable of one!

Years later, of course, it can only be understood as ironic commentary.

[Kevin Wright]

That reminds me...

I've been meaning to set up a website of collected Scala t-shirt slogans for a while (both existent and imagined).

I really must get round to it some day soon :)

[Kris Nuttycombe]

On Tue, Aug 13, 2013 at 1:54 PM, Oliver Ruebenacker <cur...@gmail.com> wrote:

     Hello,

  Ah, my bad, I missed some subtlety. There are really two flatMap
methods, the one from Iterable and the one from Option. If Option
would become an Iterable, it would have both. And I think it should
be:

  Option.flatMap[B](f: (A) => Option[B]): Option[B]
  Option.flatMap[B](f: (A) => GenTraversableOnce[B]): Iterable[B]

Recall that these two signatures will have the same erasure.

 

  If Option was an Iterable, it would also be a GenTraversableOnce.
Would that cause ambiguities?

  Could we rename the current Option.flatMap to avoid collision with
Iterable.flatMap?

You seem to miss my point that Option#flatMap[B](f: A => Option[B]): Option[B] is the *correct*  signature for the zero-or-one monadic effect, which is what Option is all about in the first place. Once again, what's wrong with calling .toSeq, .toList, .toStream and being nicely explicit about it? It is *vastly* more important that consistent semantics with respect to the monadic effect designated by the type be preserved, than it is for you to be able to avoid 6 characters of typing.

Direct implicit conversions between types (including the current implicit conversion that allows "for(i <- List(1, 2, 3), j <- Some(1)) yield ..." to work) are the source of many, many problems, and the solution to very few. Be explicit, and the future you will thank you. 

[Oliver Ruebenacker]

     Hello,

On Wed, Aug 14, 2013 at 11:08 AM, Kris Nuttycombe <kris.nu...@gmail.com> wrote:

On Tue, Aug 13, 2013 at 1:54 PM, Oliver Ruebenacker <cur...@gmail.com> wrote:

     Hello,

  Ah, my bad, I missed some subtlety. There are really two flatMap
methods, the one from Iterable and the one from Option. If Option
would become an Iterable, it would have both. And I think it should
be:

  Option.flatMap[B](f: (A) => Option[B]): Option[B]
  Option.flatMap[B](f: (A) => GenTraversableOnce[B]): Iterable[B]

Recall that these two signatures will have the same erasure.

  Oh, right. That answers my question about ambiguities. It now occurs to me that this must be a terrible thing for any method that takes a function, to have argument and return type of that function erased. Is there a way around that?

  If Option was an Iterable, it would also be a GenTraversableOnce.
Would that cause ambiguities?

  Could we rename the current Option.flatMap to avoid collision with
Iterable.flatMap?

You seem to miss my point that Option#flatMap[B](f: A => Option[B]): Option[B] is the *correct*  signature for the zero-or-one monadic effect, which is what Option is all about in the first place. Once again, what's wrong with calling .toSeq, .toList, .toStream and being nicely explicit about it? It is *vastly* more important that consistent semantics with respect to the monadic effect designated by the type be preserved, than it is for you to be able to avoid 6 characters of typing.

Direct implicit conversions between types (including the current implicit conversion that allows "for(i <- List(1, 2, 3), j <- Some(1)) yield ..." to work) are the source of many, many problems, and the solution to very few. Be explicit, and the future you will thank you. 

  I would say both signatures are justified. There should be a flatMap-method that returns Option, and there should be another that returns Iterable. That it is hard to have both under current circumstances sounds to me like an unfortunate technicality, not a deeper philosophical conflict. In principle, it could easily be solved by renaming one of them.

  If a monad is something that contains zero or more values, that sounds to me exactly like a collection. As I said, almost any one who explains what an Option is will say that it is "like a collection".

  Is flatMap the only point of conflict between Option and Iterable, or are there any others?

[Luis Ángel Vicente Sánchez]

  If a monad is something that contains zero or more values, that sounds to me exactly like a collection. As I said, almost any one who explains what an Option is will say that it is "like a collection".

A monad is not something that contains zero or more values; I'm not an category theory expert so I'm going to simplify a lot but a monad is nothing more than a trait with two operations:

trait Monad[M[_]] {
  def unit[A](a: => A): M[A]
  def flatMap[A,B](ma: M[A])(f: A => M[B]): M[B]
}

You can provide implementations of that trait for lots of different classes as long you obey monad laws.

The Option[A] type is a type that can contain a single value of type A or nothing. Maybe using the 'like a collection' metaphor is the fastest and easiest way to introduce Option type to people that would other way not understand why a zero-or-one type is useful. 

[Oliver Ruebenacker]

     Hello,

  OK, I misread you. I thought you were saying monads have zero or one value, but you just meant Option. In that case, I would say that monads in general may not be collections, but Option is one.

  The traits you (Kris and Luis) posted seem to confirm that if we had two different method names, like iterableFlatMap and monadFlatMap, then making Option be an Iterable would be trivial.

     Take care

     Oliver

--

You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Kris Nuttycombe]

I seem to not be getting my point across. Inheritance is not the correct solution here in any case; conversion to the correct type is the correct solution. It is vital that, for example, Future be able to declare flatMap, and that futures work in for-comprehensions just as well as Options and Lists and Function1s and so on.

This issue is much, much bigger than iterability. Stop thinking about it in that context, and please try out the exercises I posted; you have much to learn.

I apologize if this comes off as a bit snotty, but coming to a language that is almost a decade old, and asking for fundamental changes to the standard library without understanding the underlying design principles is a bit obnoxious. 

[Oliver Ruebenacker]

     Hello,

  I perfectly understand that you suggest that Option should not be Iterable, but instead provide conversion to iterable. You have stated that clearly.

  What I don't understand is why you suggest that.

  I also don't understand why you think that solving your, ahem, exercise would bring me any closer to an answer.

  Is there anything that prevents me from renaming the method "flatMap" in your traits into "monadFlatMap" and then add a method "flatMap" that acts like the flatMap of Iterable?

  Is it a design principle of Scala that the method must be called this way?

     Take care

     Oliver

 

[Luis Ángel Vicente Sánchez]

That means that you want Option to implement Iterable monadic flatMap instead of the monadic flatMap that really belongs to Option.

What stops you to do that is you should be able to implement Option's map method using Option's flatMap method. If we use Iterable flatMap then map should also return Iterable... that's a non-sense.

[Oliver Ruebenacker]

     Hello,

On Wed, Aug 14, 2013 at 1:50 PM, Luis Ángel Vicente Sánchez <langel...@gmail.com> wrote:

That means that you want Option to implement Iterable monadic flatMap instead of the monadic flatMap that really belongs to Option.

  No, I want Option to have two flatMaps, a monadic flatMap and an iterable flatMap.

 

What stops you to do that is you should be able to implement Option's map method using Option's flatMap method.

  Why do you need to be able to do this? You can still implement Option.map using monadic flatMap, if you like. 

If we use Iterable flatMap then map should also return Iterable... that's a non-sense.

  Option.map would still return Option. If Option is an Iterable, then that is compatible with Iterable.map.

     Take care

     Oliver

[√iktor Ҡlang]

On Wed, Aug 14, 2013 at 7:58 PM, Oliver Ruebenacker <cur...@gmail.com> wrote:

     Hello,

On Wed, Aug 14, 2013 at 1:50 PM, Luis Ángel Vicente Sánchez <langel...@gmail.com> wrote:

That means that you want Option to implement Iterable monadic flatMap instead of the monadic flatMap that really belongs to Option.

  No, I want Option to have two flatMaps, a monadic flatMap and an iterable flatMap.

Option(1).to[Iterable].flatMap

 

 

What stops you to do that is you should be able to implement Option's map method using Option's flatMap method.

  Why do you need to be able to do this? You can still implement Option.map using monadic flatMap, if you like. 

If we use Iterable flatMap then map should also return Iterable... that's a non-sense.

  Option.map would still return Option. If Option is an Iterable, then that is compatible with Iterable.map.

.to[Iterable]

Cheers,

√

--

Viktor Klang
Director of Engineering

Typesafe

Twitter: @viktorklang

[Kris Nuttycombe]

Forgot to reply-all

On Wed, Aug 14, 2013 at 12:00 PM, Kris Nuttycombe <kris.nu...@gmail.com> wrote:

On Wed, Aug 14, 2013 at 11:44 AM, Oliver Ruebenacker <cur...@gmail.com> wrote:

     Hello,

  I perfectly understand that you suggest that Option should not be Iterable, but instead provide conversion to iterable. You have stated that clearly.

  What I don't understand is why you suggest that.

  I also don't understand why you think that solving your, ahem, exercise would bring me any closer to an answer.

That's because you haven't solved it yet. :) In seriousness though, you need to recognize that for-comprehensions are *not just for traversing collection-like things*. They are general-purpose machinery for sequencing operations in a type-safe fashion, built around the idea of the monadic bind.

Here is an example of something that is not collection-like at all: 

val myGen = for {
  n <- Gen.choose(10,20)
  m <- Gen.choose(2*n, 500)
} yield (n,m)

 Should Gen (this is from https://github.com/rickynils/scalacheck/wiki/User-Guide) also implement your "iterable flatMap"? Of course not, this makes no sense.

Similarly,

val myFuture = for {

  a <- Future(/* some long-running computation */)

  b <- someOtherLongRunningComputationReturningFuture(a)

} yield b

makes *no sense* unless the consistency requirement for the monadic context is maintained. 

 

Monadic composition has *laws* which must be obeyed. In Scala, for-comprehensions desugar via a form of structural typing to flatMap (and map and filter) calls on the underlying types. Now, it is true that Scala's structural typing would make it possible to change Option in the way that you suggest, and in so doing you'd simultaneously make Option *utterly useless* for a much wider set of use cases.

The reason that the Monad typeclass is interesting is because knowing that something is a Monad allows you to completely abstract away the context of your computation. Here's a blog post you should read on this topic, if you have a genuine desire to understand: 

https://www.precog.com/blog/Precog-Copointed-The-Abstract-Future/

Kris

[Oliver Ruebenacker]

     Hello,

  I think now I understand: flatMap is special, because for-comprehensions call it. So there can be only one such method.

  Thanks for explaining!

  There is nothing in the declaration showing that Iterable.flatMap, Option.flatMap and Future.flatMap are in any way related. Wouldn't it be nice to have something that signals the special role of flatMap? Perhaps declare it via a trait ForComprehensible?

     Take care

     Oliver

[Raul Raja]

Getting back to the original question and In the same line of "sequencing operations in a type-safe fashion" and without getting into the internals on how for comprehensions work…

Would something like this make sense or be desirable in many cases?

def getUser : Future[Option[User]] = ...

def getMessages(u : User) : Future[List[Message]] = …

forLikeExpression {

maybeUser <- getUser()

user <- maybeUser

messages <- getMessages(user)

} yield messages

[Kris Nuttycombe]

If you use scalaz, you can get close:

import scalaz.std.option._

import scalaz.syntax.traverse._

for {

  maybeUser <- getUser

  userList  = maybeUser.toList

  messages <- userList.traverseM(getMessages)

} yield messages

[Raul Raja Martinez]

Kris,

Great! Thanks!

[Raul Raja Martinez]

This is what I finally came up with using Scalaz OptionT Monad transformers with a utility object that maps some common used values into Future[Option]

Here is an extract from some tests...

"Mix in heterogeneous values into a for comprehension" in {

      expect(for {

        a <- ? <~ Future(Option(1))

        b <- ? <~ Future(2)

        c <- ? <~ Option(3)

        d <- ? <~ 4

        e <- ? <~ Future(List(5, 6, 7))

      } yield (a, b, c, d, e), (1, 2, 3, 4, List(5, 6, 7)))

}

https://github.com/47deg/scala-commons-47/blob/master/src/test/scala/com/fortysevendeg/commons/scala/utils/tests/OptionTFutureUtilsTests/OptionTFutureUtilsTests.scala

And the actual trait

https://github.com/47deg/scala-commons-47/blob/master/src/main/scala/com/fortysevendeg/commons/scala/utils/OptionTFutureUtils.scala

It allows me to mix in any random operations into for comprehensions properly converting them to Future[Option] which them OptionT transforms into the actual value. This has simplified my code a lot since I no longer have to build a million nested flatmaps when dealing with Future[Option]

I'd love some feedback if there is a better way to do this since I'm just getting started learning ScalaZ Monad Transformers.

Cheers

--

--

Raúl Raja Martínez

Co-founder @ 47 Degrees

h: http://raulraja.com

w: http://47deg.com

t: http://twitter.com/raulraja