More thoughts on Asynchronous Callback syntax

[Vincent Marquez]

In my last email, I mentioned how I hated having to nest lambdas for asynchronous callbacks in my network code:
   obj1.connect("param", ()=>

      obj2.connect("asdf", ()=> 

          obj1.join(obj2, ()=>

                println("done for now")
            )

         )

      )

was godawful ugly.  So, I proposed using for comprehension to clean things up a bit, which turned out to be very similar to how Akka's futures work (though not quite the same)

https://groups.google.com/group/scala-user/browse_thread/thread/11f2e536b3ff47b5

for(res < - obj1.connect("param");
     res <- obj2.conncet("asdf");

     res <- obj1.join(obj2))

     println("done for now")

definitely looks better, but I don't always return something, and I can't put arbitrary code in.  So, my latest solution I whipped up rather quickly may mitigate some of my issues, but it also might be ugly.  It's certainly not too novel, borrowing a bit from akka and F#'s async workflows...

http://pastebin.com/cDstZ5da

The idea is to instead use this syntax:

obj1.connect("param") ~>
obj2.connect("asdf") ~>

obj1.join(obj2) ~>

println("done for now") run()

The final 'run' syntax is a bit weird, had some ideas on how to clean that up, but I was curious to get thoughts on the general approach...

--Vincent

[√iktor Ҡlang]

On Wed, May 30, 2012 at 8:40 PM, Vincent Marquez <vincent...@gmail.com> wrote:

In my last email, I mentioned how I hated having to nest lambdas for asynchronous callbacks in my network code:
   obj1.connect("param", ()=>

      obj2.connect("asdf", ()=> 

          obj1.join(obj2, ()=>

                println("done for now")
            )

         )

      )

was godawful ugly.  So, I proposed using for comprehension to clean things up a bit, which turned out to be very similar to how Akka's futures work (though not quite the same)

https://groups.google.com/group/scala-user/browse_thread/thread/11f2e536b3ff47b5

for(res < - obj1.connect("param");
     res <- obj2.conncet("asdf");

     res <- obj1.join(obj2))

     println("done for now")

definitely looks better, but I don't always return something, and I can't put arbitrary code in.

Doesn't seem that bad to me (considering that your API heavily side-effects), even parallelizing the connects:

obj1.connect("param") zip obj2.connect("asdf") flatMap { _ => obj1 join obj2 } foreach { _ => println("done for now" }

Cheers,

V

 

 So, my latest solution I whipped up rather quickly may mitigate some of my issues, but it also might be ugly.  It's certainly not too novel, borrowing a bit from akka and F#'s async workflows...

http://pastebin.com/cDstZ5da

The idea is to instead use this syntax:

obj1.connect("param") ~>
obj2.connect("asdf") ~>

obj1.join(obj2) ~>

println("done for now") run()

The final 'run' syntax is a bit weird, had some ideas on how to clean that up, but I was curious to get thoughts on the general approach...

--Vincent

--
Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[Vincent Marquez]

Thanks for the reply Victor. 

for(res < - obj1.connect("param");
     res <- obj2.conncet("asdf");

     res <- obj1.join(obj2))

     println("done for now")

definitely looks better, but I don't always return something, and I can't put arbitrary code in.

Doesn't seem that bad to me (considering that your API heavily side-effects), even parallelizing the connects:

obj1.connect("param") zip obj2.connect("asdf") flatMap { _ => obj1 join obj2 } foreach { _ => println("done for now" }

Using that way, I'm not sure I can cleanly add a print statement (or arbitrary function call that doesn't return something that is mappable) between obj1.connect and obj2.connect.  That's one of the reasons I thought my newer method (~>) might look cleaner. 

--Vincent 

[√iktor Ҡlang]

see andThen: http://doc.akka.io/api/akka/2.0.1/#akka.dispatch.Future

Cheers,

√

 

--Vincent 

[Josh Suereth]

Funny enough, I'm giving  a talk on this *exact* subject to the Pittsburgh techfest.  Not to spoil things until after the talk, I believe you want "Monadic workflows".   Key to this is the ability to lift behavior into the computational context (Future).

val behavior : Future[Unit] = 

  for {

    _ <- obj1.connect("param")

    _ <- obj2.connect("asdf")

    _ <- obj1 join obj2

    _ <- Future(println("Done for now")

  } yield ()

You are side-effecting *way* more than necessary here.   If connect returns a connected object, then you'd have:

val c1: Future[Connection1] = factory1.connect("param")

val c2: Future[Connection2] = factory2.connect("asdf")

val behavior: Future[Data] =

  for {

    (a,b) <- c1 join c2

    _ <- Future(println("Done joining queries"))

  } yield calculate(a,b)

Or something like that.

My example is using an asynchronous API on github and looks like the following:

trait GhApi {

  def projects(user: String): Future[Seq[Project]]

  def pullrequests(proj: Project): Future[Seq[PullRequest]]

  def watchers(proj: Project): Future[Seq[Watcher]]

}

trait StatisticsService {

  def api: GhApi

  def statistics(user: String): Future[Statistics] = {

      val projects = api.projects(user)

      def get[B](f: Project => Future[Seq[B]]): Future[Seq[B]] = for {

        ps: Seq[Project] <- projects

        items: Seq[Seq[B]] <- ps traverse f

      } yield items.flatten

      val pullRequests: Future[Seq[PullRequest] = get(api.pullrequests)

      val watchers: Future[Seq[User]] = get(api.watchers)

      (pullRequests zip watchers) map { case (prs, ws) =>

        Statistics(user, ws, prs)

      }

  }

}

Notice how the API chains together a bunch of asynchoronous calls and the entire function is asynchronous.   You have to use some concepts that you may not know the name for (Functors, Applicative Functors, Monads, Essence of Iteration).  However, it is possible, and it's not too ugly.

ALSO, if you abstract out the Future, you can wire in the execution context with implicits.   That is, you can make a "SingleThreaded" context where everything executes immediately for testing.

Some of us call this kind of behavior "monadic workflows", and they're pretty powerful, but very abstract.   I think we should probably stick with common conventions in scala (i.e. for-expressions) rather than custom methods and DSLs.

- Josh

[Tony Morris]

What type does connect return? I reckon you could tidy it further.

For your thought:

case class Param[F[_], X](k: String => F[X])

* has map if F has map
* has flatMap if F has flatMap

[Vincent Marquez]

I see the argument for using the scala convention of using for comprehension to construct a 'monadic workflow'.   

To answer Tony's question, connect in this case returns a Unit, (or a Future[Unit]), so I don't need to map over a new value. 

Has there been any talk on making 'monadic workflows' in Scala a little less cumbersome?  

val behavior : Future[Unit] = 

  for {

    _ <- obj1.connect("param")

    _ <- obj2.connect("asdf")

    _ <- obj1 join obj2

    _ <- Future(println("Done for now")

  } yield ()

seems to be the most idiomatic Scala way of what I'm trying to do (ignoring the fact that I need my Future to do something a wee bit different), but I do hate that I'd have to explicitly wrap non-Future returning calls in a future.   

That was the main problem I tried to solve with my syntax, along with not having to deal with Future[Unit] with the _ <- syntax.    I haven't spent a lot of time on the code I submitted,  but I do think that it could be adopted to also support the more 'monadic' paradigm, and could be made immutable.    In a vacuum, what are its flaws and merits?  Has anyone else with more brain power come up with alternative (to for comprehension) monadic workflow syntax for Scala?  

Thanks again to everyone who replied.  

--Vincent

On Wed, May 30, 2012 at 4:06 PM, Josh Suereth <joshua....@gmail.com> wrote:

On Wed, May 30, 2012 at 6:57 PM, Vincent Marquez <vincent...@gmail.com> wrote:

Thanks for taking the time to reply Josh.

On Wed, May 30, 2012 at 3:22 PM, Josh Suereth <joshua....@gmail.com> wrote:

Funny enough, I'm giving  a talk on this *exact* subject to the Pittsburgh techfest.  Not to spoil things until after the talk, I believe you want "Monadic workflows".   Key to this is the ability to lift behavior into the computational context (Future).

Cool.  When is this talk taking place.  I'll keep an eye out and hopefully it's recorded or you can put slides up.

June 9th.  Not sure on recordings.  I plan to give it more than once :) 

You are side-effecting *way* more than necessary here.   If connect returns a connected object, then you'd have:

If you're talking about my Future having side effects, I could make it immutable and the API should stay the same. 

If you mean my obj.connect() method must have some implicit side effects, then I fear we might get into a bit of a philosophical debate here.  :-) I don't mind it having side effects, the actual code I'd be using this for uses typed actors for any objects with internal state, so I should have thread safety from that. Tthe connect method would return a future that then sends an actor message to obj)

I got ya. You just want to asynchronously sequence a set of messages to a bunch of actors?   I think my early example (with for { _ <- ...}) shows that...

ALSO, if you abstract out the Future, you can wire in the execution context with implicits.   That is, you can make a "SingleThreaded" context where everything executes immediately for testing.

Great idea.  Thanks.

 

Some of us call this kind of behavior "monadic workflows", and they're pretty powerful, but very abstract.   I think we should probably stick with common conventions in scala (i.e. for-expressions) rather than custom methods and DSLs.

Given that I don't have to return a new connected obj (so for comprehension can be a bit cumbersome) do you still think an API to reduce the "{ }" for maps is a bad idea?  

It's now a matter of expectations and conventions in your company.   I think we (the scala community) are starting to use for comprehensions both for searching collections *and* for monadic workflows.   If you use a for-comprehension, you're using a more commonly understood scala convention.  Haskell has its do-notation, we use for-comprehensions.   If you create a new DSL for the same kind of thing, just know that you're setting up a new way to do the same thing that you'll have to teach everyone in your company.

Personally, I stick to for-comprehensions because you have to learn those quickly to get into scala.

[Luke Vilnis]

I agree the syntax is nasty - I would love to see an equivalent of F#'s workflow syntax. F# provides lots of different operators that are bound to keywords inside the workflow, the one that performs a "bind" and throws away the result is called "do!" (by analogy with "bind" which is written as "let!" - a sort of amplified "let"). Using F#'s syntax you would write your program as:

  async {

    do! obj1.connect("param")

    do! obj2.connect("asdf")

    do! obj1 join obj2

    do! Future(println("Done for now")

  }

which is pretty nice.

[Josh Suereth]

I'd love to see someone enhance for-expressions to allow more things.  If you google Comprehensive Comprehensions, you'll see some niceties you can do.  I also think it would be fun for more of haskell's do-notation features to show up, assuming you can unambiguously parse them.

- Josh

[√iktor Ҡlang]

On Thu, May 31, 2012 at 10:12 PM, Luke Vilnis <lvi...@gmail.com> wrote:

I agree the syntax is nasty - I would love to see an equivalent of F#'s workflow syntax. F# provides lots of different operators that are bound to keywords inside the workflow, the one that performs a "bind" and throws away the result is called "do!" (by analogy with "bind" which is written as "let!" - a sort of amplified "let"). Using F#'s syntax you would write your program as:

  async {

    do! obj1.connect("param")

    do! obj2.connect("asdf")

    do! obj1 join obj2

    do! Future(println("Done for now")

  }

which is pretty nice.

http://doc.akka.io/docs/akka/2.0.1/scala/dataflow.html#Dataflow_Delimiter

cheers,

√

[Derek Williams]

On Thu, May 31, 2012 at 2:07 PM, Vincent Marquez <vincent...@gmail.com> wrote:

val behavior : Future[Unit] = 

  for {

    _ <- obj1.connect("param")

    _ <- obj2.connect("asdf")

    _ <- obj1 join obj2

    _ <- Future(println("Done for now")

  } yield ()

seems to be the most idiomatic Scala way of what I'm trying to do (ignoring the fact that I need my Future to do something a wee bit different), but I do hate that I'd have to explicitly wrap non-Future returning calls in a future.   

As a workaround you can use '=' instead of '<-' to avoid wrapping:

val behaviour = for {

  ...

  _ <- obj1 join obj2

  _  = println("Done for now")

} yield ()

of course if your println should happen at the end of the for comprehension, you might as well put it after the yield: 

val behaviour = for {

  ...

  _ <- obj1 join obj2

} yield println("Done for now")

--

Derek Williams

[Tony Morris]

On 01/06/12 06:07, Vincent Marquez wrote:
> I see the argument for using the scala convention of using for
> comprehension to construct a 'monadic workflow'.

A 'monadic workflow' is best provided by library support. I think it is
agreed by most (all?) people who use these concepts heavily that F#
workflows are insufficiently useful. In particular, it is not possible
to abstract on the type constructor in which you are operation. To be
clear, that type constructor may be the composition of two or more type
constructors. In other words, it is not possible to implement sequence.
http://etorreborre.blogspot.com.au/2011/06/essence-of-iterator-pattern.html

This is game over in terms of what it is you are attempting to achieve,
because all I need to do is go through all the effort of altering the
language to suit your use-case, implement that use-case and while you
are smiling at me, request a slight difference to the use-case and
you'll have to throw your hands in air. You won't be able to achieve it
and I assure you there are an enormous amount of variations that arise
all the time.

> To answer Tony's question, connect in this case returns a Unit, (or a
> Future[Unit]), so I don't need to map over a new value.
>
> Has there been any talk on making 'monadic workflows' in Scala a little
> less cumbersome?
>
> val behavior : Future[Unit] =
> for {
> _ <- obj1.connect("param")
> _ <- obj2.connect("asdf")
> _ <- obj1 join obj2
> _ <- Future(println("Done for now")
> } yield ()

There is a mechanical test now that I can apply to your code and it
yields the result, "the program does nothing [footnote: side-effect]."
Any expression of this form:

_ <- unit(expr)

Where the unit function is the monadic unit (aka return/pure/point) for
the monad in which you are operating means your program is exactly
equivalent to this:

// I use ignore because underscore is a compiler-error; sufficient to
say, you are not able to use it
val ignore = expr

This is obviously a "nothing" program, but of course, that's not what
you are doing, because you are side-effecting. In other words, you want
to run in some kind of "first-class" environment for "side-effecting."
Scalaz has this if you are interested and I hear regular reports of its
reinventing and it is called IO.

As for your desire to tidy up the code itself, you want a supporting
combinator library. For example, for anything that has flatMap (such as
Future or the composition of IO with Future if you really want it nice),
you want a derived function of the type F[A] => F[B] => F[B] that runs
the effect in the first argument, ignores the result, then runs the
second argument. The code is obvious a => b => a flatMap(_ => b). This
is precisely your for-comprehension. This is in Scalaz.

As for "automatic lifting", that is, turning A into F[A] implicitly, I
strongly recommend against this. What you really want is library support
for coming it at some level in the type constructor stack. For example,
you might choose to run a comprehension in IO[Future[_]] and you have a
IO[A]. How do you put this value in? With a specific library function,
that's how -- in this case, you provide a trait that lifts IO values
into any environment and instance that for the thing that is the
composition of IO and Future

trait LiftIO[F[_]] {
def liftIO(x: IO[A]): F[A]
}

In practice, you'd write a data type that allows you to compose anything
with Future. That's because that thing too has a flatMap. It is often
called "FutureT."

In summary, language syntax for specific use-cases is less helpful than
library support for more use-cases, especially when that library support
does the specific use-case better than language support would. From the
perspective of having written it in the language, it is free money at
zero cost.

Happy to help you out with library support. You don't have to use
Scalaz. You can just reinvent the part you want -- reinvention is a
great method of discovery and besides, you wrote it!

> seems to be the most idiomatic Scala way of what I'm trying to do (ignoring
> the fact that I need my Future to do something a wee bit different), but I
> do hate that I'd have to explicitly wrap non-Future returning calls in a
> future.
>
> That was the main problem I tried to solve with my syntax, along with not
> having to deal with Future[Unit] with the _ <- syntax. I haven't spent a
> lot of time on the code I submitted, but I do think that it could be
> adopted to also support the more 'monadic' paradigm, and could be made
> immutable. In a vacuum, what are its flaws and merits? Has anyone else
> with more brain power come up with alternative (to for comprehension)
> monadic workflow syntax for Scala?
>
> Thanks again to everyone who replied.
>
> --Vincent
>
>
>
> On Wed, May 30, 2012 at 4:06 PM, Josh Suereth <joshua....@gmail.com>wrote:
>
>>
>> On Wed, May 30, 2012 at 6:57 PM, Vincent Marquez <
>> vincent...@gmail.com> wrote:
>>
>>> Thanks for taking the time to reply Josh.
>>>
>>> On Wed, May 30, 2012 at 3:22 PM, Josh Suereth <joshua....@gmail.com>wrote:
>>>
>>>> Funny enough, I'm giving a talk on this *exact* subject to the Pittsburgh

>>>> techfest <http://pghtechfest.com/>. Not to spoil things until after

--
Tony Morris
http://tmorris.net/

[Josh Suereth]

But it seems like we could leverage those abstractions better with additional language support.  I agree that putting as much into libraries is ideal.  However, macros open up a world of possible nicer syntax for this kind of development.  Nicer library-based syntax with good conveniences.

Some additions to for comprehensions can also help IMHO.  I agree that a library/user will know best.  But some things are better with language support, like first-class functions.

Possibly once we have type-level macros everything is easier?

[Tony Morris]

I agree, but I caution against cost of syntax, *especially* when that addition of syntax yields a worse result than library support.

For example, I could easily get behind:

for {
  expr
}

being equivalent to:

for {
  _ <- expr
}

because the syntactic cost is small and the benefit non-zero. I could also get behind introducing lenses on record and sum types, because the cost is small (zero compared to what exists today i.e. we've already paid the cost) and the benefit significant above zero. Other considerations: applicative functor comprehension, arrow comprehension, comonad comprehension.

However, I could not get behind something like F# workflows because they have a non-zero (significantly less than zero) cost with zero benefit above library support. Just write, discover or use existing the library support and you'll be far ahead already -- so why pay money to buy the opportunity to pay more money?

[Josh Suereth]

Perhaps we agree on things we'd like to see?  I'm mostly unfamiliar with F# workflows but I like the term for the general pattern.

Also comprehensive comprehensions seem nice.

[Vincent Marquez]

Thanks for the reply Tony.  

I do agree best case scenario would be to have the folks working on Scala 

add to for comprehension to allow

for( _ <- somethingMappable)

 be equivalent to 

for( somethingMappable ) 

which is what you said, though I'd also like 

for( _ = somethingNotMappable)

being equivalent to 

for( somethingNotMappable)

(I'm not sure what the term is for something that's mappable/flatmappable, functor perhaps?  You probably understand what I want never the less...)

This is obviously a "nothing" program, but of course, that's not what
you are doing, because you are side-effecting. In other words, you want
to run in some kind of "first-class" environment for "side-effecting."
Scalaz has this if you are interested and I hear regular reports of its
reinventing and it is called IO.

Yeah, I that's exactly what I want, and from what little I know of Scalaz,

it does sound a bit like IO now that you mention it...  

I'll take a look at Scalaz's IO, thanks again for the help.  

--Vincent

[Tony Morris]

On 06/06/12 13:19, Vincent Marquez wrote:
> (I'm not sure what the term is for something that's mappable/flatmappable,
> functor perhaps? You probably understand what I want never the less...)

"Things that are mappable and flatmappable" do not have a well
established name, however, with an additional unit function, that name
would be monad. Things with map are called functors.

scala> trait Functor[F[_]] { def fmap[A, B](f: A => B): F[A] => F[B] }
// things with map
defined trait Functor

scala> trait FlatMap[F[_]] extends Functor[F] { def flatMap[A, B](f: A
=> F[B]): F[A] => F[B] } // things with map & flatMap
defined trait FlatMap

Categories without a unit function (and so are not necessarily
categories) are called semigroupoids so we can give rise to the Kleisli
semigroupoid by taking "anything with flatMap."

scala> trait Semigroupoid[~>[_, _]] { def compose[A, B, C]: (A ~> B) =>
(B ~> C) => (A ~> C) } // No unit
defined trait Semigroupoid

scala> case class Kleisli[A, F[_], B](k: A => F[B])
defined class Kleisli

scala> def KleisliSemigroupoid[F[_]: FlatMap]: Semigroupoid[({ type
lam[a, b]=Kleisli[a, F, b] })#lam] = new Semigroupoid[({ type lam[a,
b]=Kleisli[a, F, b] })#lam] { def compose[A, B, C] = f => g => Kleisli(a
=> implicitly[FlatMap[F]].flatMap(g.k)(f k a)) }
KleisliSemigroupoid: [F[_]](implicit evidence$1:
FlatMap[F])Semigroupoid[[a, b]Kleisli[a,F,b]]

[Razvan Cojocaru]

If you take this to its obvious conclusion, sequence is not the only combinatory.

 

I’ve been playing with something here https://github.com/razie/gremlins

 

Some syntax samples include using + or --> but also seq/par and even CSP style syntax.

 

par {                                                                                                                                   

  seq {                                                                                                                                

     inc                                                                                                                                

     log($0)                                                                                                                           

     }                                                                                                                                  

  seq {                                                                                                                                

    inc                                                                                                                                 

    log($0)                                                                                                                            

    }                                                                                                                                   

  }"""

 

Or  CSP

 

v(c) (c ? P | c ! Q) 

 

all this is obviously based on a rather heavy DSL library which uses all kinds of conversions to wrap things and bind them.

 

Cheers,

RAzie

[Meredith Gregory]

Dear Razie,

One alternative is to use for-comprehensions as syntax.

for( y <- x ? ) { P( y ) } 

is the equivalent for sequencing: x?( y )P

and if you implement flatMap appropriately

for( v <- u ? ; y <- x ? ){ spawn{ P( v ) }; spawn{ Q( y ) } }

is the equivalent of parallel composition: u?( v )P | x?( y )Q

This approach has the advantage that it provides a natural syntax for fork-join patterns

for( v <- u ? ; y <- x ? ){ P( u, y ) }

which is much harder to express in process calculi that don't come with them natively.

Further, it's easy to do things like applied π-calculus.

Finally, now that i have over a year of observing teams of Java developers pick up this syntax and write complex distributed applications in much less time than it would otherwise take them, i think that there is some supporting evidence that people grok the for-comprehension notation a little more readily.

Obviously, there are trade-offs. If the default flatMap is interpreted as parallel, then sequences of basic i/o become a little more cumbersome to write.

for( y1 <- x1 !?; ... ; yN <- xN !? ){ spawn{ P1( y1, ..., yN ) }; ... ; spawn{ PM( y1, ..., yN )  } }

would be the equivalent of

x1( y1 )? ... xN( yN )?( P1( y1, ..., yN ) | ... | PM( y1, ..., yN ) )

With macros or whatnot it ought to be able to make a distinction between

for( y1 <- x1 ?; ... ; yN <- xN ? ){ ... }

and

for( y1 <- x1 ? |  ... | yN <- xN ? ){ ... }

which would again provide a very natural syntactic container for both semantics.

Best wishes,

--greg

--
L.G. Meredith
Managing Partner
Biosimilarity LLC
7329 39th Ave SW

Seattle, WA 98136

+1 206.650.3740

http://biosimilarity.blogspot.com