How to handle future of list of future of ...

[Normen Müller]

Hi, how to handle a future of a list of Http requests 

val requestsF: Future[List[Req]] = ...

after executing each request (`http(req...)`) whereas each Http request yields a future of JSON values:

val responsesF: Future[List[Future[JValue]]] =

  requestsF map ( requests => requests map (req => http(req OK as.json4s.Json)) )

  

and I actually want to further process the JSON values, for example, extracting some fields and, based on their values, spawn some new async Http calls. The first thing I came up with is

val tmp: Future[List[JValue]] = (responesF map (Future.sequence(_))).flatten

Now I could go on by `tmp flatMap { js => ... }`. But my question is, is the right-hand side of `tmp` the "proper" way to go?

Cheers, /nm

[Brian Maso]

I'm not sure what would be "improper" about it. Are you thinking the flattening might be forcing he evaluation of the nested Futures eagerly?

Stylistic thought: A "map/flatten" always makes me say "use flatMap". But in this case I had to explicitly provide the Future.sequence type parameters, or else the compiler couldn't figure out what the type parameters to Future.flatMap were supposed to be:

val tmp = responsesF flatMap (Future.sequence[JValue, List])

I'm not sure that's any easier to understand, but I don't think its any harder to understand either OTOH.

Brian Maso

[Normen Müller]

Hi, with "proper" I ask for if there isn't a more elegant way. In addition I am concerned about performance regarding first sequencing and then flattening. Best, /nm

[Derek Williams]

You can combine more of the commands using 'traverse' instead of 'sequence' and 'flatMap' instead of 'map' when generating 'responsesF', this will save you one intermediate collection:

val responsesF: Future[List[JValue]] =

  requestsF flatMap ( requests => Future.traverse(requests)(req => http(req OK as.json4s.Json)) )

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Derek Williams

[Johannes Rudolph]

On Thu, Feb 20, 2014 at 4:56 PM, Normen Müller <normen....@gmail.com> wrote:
> Hi, with "proper" I ask for if there isn't a more elegant way. In addition I
> am concerned about performance regarding first sequencing and then
> flattening. Best, /nm

Do you think performance of these local Future-operations matter if
the child Futures are executing HTTP-requests? This doesn't seem to be
code in the first place where you need to care for a bit of extra
latency from some local scheduling overhead, or is it?

--
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net

[Normen Müller]

Hi Derek, thanks for the hint! I actually came up with

val responseF: Future[List[JValue]] = requestsF flatMap { requests =>

  Future.sequence(requests map ( req => http(req OK as.json4s.Json) ))

}

what is equivalent as to the types. Best, /nm

[Normen Müller]

Hi Johannes, thanks for your reply.

Sorry, I don't get your point, though.

Basically, I am concerned about each *explicit* function call, i.e., I am always asking myself if this one is really needed or can I do better by utilizing more adequate combinators, for example. 

Best, /nm

[Vlad Patryshev]

Normen,

The science behind all this is that Future is a monad and an applicative functor, and List is Traversable.

If A is applicative and B is traversable, you can transform B[A[X]] into A[B[X]] by using, yes, traverse method (or function, whatever is available).

After you convert Future[List[Future[X]]] into Future[Future[List[X]]], you can flatten Future[Future[...]] into Future[...]] because Future is a monad.

:) Hope it helps.

A very good question, actually.

Thanks,
-Vlad

--

[Normen Müller]

Hi Vlad, thanks for your reply!

In a previous answer of mine in this thread I tried to clarify my motivation:

>Basically, I am concerned about each *explicit* function call, i.e., I am always asking myself if this one is really needed or can I do better by utilizing more adequate combinators, for example. 

So, as to your post, this would mean: do I really have to explicitly call `flatten` or use adequate library combinators instead. My question was not about how to flatten such a type or why can I do that, but rather if there isn't a more elegant way to do it.

First I ended up with what exactly you proposed: "[...] convert Future[List[Future[X]]] into Future[Future[List[X]]], [...] flatten Future[Future[...]] into Future[...]] [...]"

val responsesF: Future[List[Future[JValue]]] = ...

val tmp: Future[List[JValue]] = (responesF map (Future.sequence(_))).flatten

But finally:

val responseF: Future[List[JValue]] = requestsF flatMap { requests =>

  Future.sequence(requests map ( req => http(req OK as.json4s.Json) ))

}

I know there is a `flatten` behind the scenes but it can be easier read, say, is more straightforward and with respect to performance, I guess, standard library combinators are (almost) always better. 

Best, /nm

[Josh Suereth]

On Thu, Feb 20, 2014 at 4:20 PM, Normen Müller <normen....@gmail.com> wrote:

Hi Vlad, thanks for your reply!

In a previous answer of mine in this thread I tried to clarify my motivation:

>Basically, I am concerned about each *explicit* function call, i.e., I am always asking myself if this one is really needed or can I do better by utilizing more adequate combinators, for example. 

So, as to your post, this would mean: do I really have to explicitly call `flatten` or use adequate library combinators instead. My question was not about how to flatten such a type or why can I do that, but rather if there isn't a more elegant way to do it.

First I ended up with what exactly you proposed: "[...] convert Future[List[Future[X]]] into Future[Future[List[X]]], [...] flatten Future[Future[...]] into Future[...]] [...]"

val responsesF: Future[List[Future[JValue]]] = ...

val tmp: Future[List[JValue]] = (responesF map (Future.sequence(_))).flatten

But finally:

val responseF: Future[List[JValue]] = requestsF flatMap { requests =>

  Future.sequence(requests map ( req => http(req OK as.json4s.Json) ))

}

I know there is a `flatten` behind the scenes but it can be easier read, say, is more straightforward and with respect to performance, I guess, standard library combinators are (almost) always better. 

My prefered syntax for this is:

val responseF: Future[List[JValue]] = 

  for {
    requests <- requestsF
    results <-   Future.traverse(requests) { req => 

      http(req OK as.json4s.Json) 

    }
  } yield results

If you're doing a nested map, you should think about using traverse instead of sequence.

Also, for some reason, I prefer for syntax over others.

[Normen Müller]

Hi Joshua, thanks for the reply! Isn't sequence a specialised traverse, though? Just curious about your advice. Best, /nm

[Derek Williams]

Using a single 'traverse' is more efficient than using 'map' and 'sequence'. This is the same as using 'flatMap' instead of 'map' and 'flatten', except that 'traverse' is usually an even bigger win for efficiency and performance.

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Derek Williams