Concurrency semantics (aka task sequencing) in 0.13 syntax

[eugene yokota]

Hi,

I'm starting to use more sbt 0.13 syntax here and there, and it's an improvement over <<= in most simple cases

in terms of mental overhead and keyboard typing involved.

It *almost* feels like calling plain functions. The familiarity could backfire when used with side effects:

val startServer = taskKey[Unit]("start server.")
val stopServer = taskKey[Unit]("stop server.")
val integrationTest = taskKey[Unit]("integration test.")


startServer := {
  println("start")
  IO.touch(file("server.txt"))
}


stopServer := {
  println("stop")
  IO.delete(file("server.txt"))
}


integrationTest := {
  startServer.value
  test.value
  stopServer.value
}

When test task checks for "sever.txt", it may or may not exist.

Since startServer, test, and stopServer all run in parallel.

Is there an imperative macro that expands each expression into series of dependent tasks?

integrationTest := ordered {
  startServer.value
  println("hola!")
  test.value
  stopServer.value
}

-eugene

[Mark Harrah]

There is not. The macro is mainly just syntax and works with the previously existing methods. I'm pretty sure I emphasized these particular semantics in the changelog, but maybe not in the getting started guide. It is a syntax for applicative functors, not monads.

In the example below, `ordered` would not be a guarantee if it were built using existing functionality. If startServer depended on `test` for example, the order would be `test` and then `startServer` and the second `test` wouldn't do any work because it had already been added to the graph.

I've been trying to think of some new ordered-like primitive with the right semantics. The best I've come up with is something like `ordered` that just generates an error if they aren't executed in that order, such as in the example above.

-Mark

> integrationTest := ordered {
> startServer.value
> println("hola!")
> test.value
> stopServer.value
> }
>
> -eugene
>

> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to simple-build-t...@googlegroups.com.
> To post to this group, send email to simple-b...@googlegroups.com.
> Visit this group at http://groups.google.com/group/simple-build-tool.
> For more options, visit https://groups.google.com/groups/opt_out.

[eugene yokota]

On Monday, August 19, 2013 9:04:38 AM UTC-4, Mark Harrah wrote:

There is not.  The macro is mainly just syntax and works with the previously existing methods.  I'm pretty sure I emphasized these particular semantics in the changelog, but maybe not in the getting started guide.  It is a syntax for applicative functors, not monads.

"0.13.0 Changes" says:

> To refer to the value of other settings or tasks, use the value method on settings and tasks. This method is a stub that is removed at compile time by the macro, which will translate the implementation of the task/setting to the old syntax.

It wasn't clear if the order is preserved from that document.

In the example below, `ordered` would not be a guarantee if it were built using existing functionality.  If startServer depended on `test` for example, the order would be `test` and then `startServer` and the second `test` wouldn't do any work because it had already been added to the graph.

I've been trying to think of some new ordered-like primitive with the right semantics.  The best I've come up with is something like `ordered` that just generates an error if they aren't executed in that order, such as in the example above.

I am personally ok with `ordered` not being able to guarantee strict ordering since settings can depend on each other just by someone else rewiring them.

integrationTest := ordered {
  // setup
  startServer.value
  // middle
  val n = name.value
  val jn = (jarName in assembly).value
  val v = version.value
  doSomething(n, jn, v)
  // teardown
  stopServer.value
}

In the above, I am ok with version.value running before (jarName in assembly).value, as long as doSomething happens after startServer.value.

This reminds me of [scheduling/concurrency control][1], which is what I am actually after.

integrationTest := transaction {
  startServer.value
} andThen {
  val n = name.value
  val jn = (jarName in assembly).value
  val v = version.value
  doSomething(n, jn, v)  
} andThen {
  stopServer.value
}

-eugene

  [1]: http://en.wikipedia.org/wiki/Schedule_(computer_science)

[Mark Harrah]

It would, but what if stopServer.value happens before doSomething? You can't guarantee that.

> This reminds me of [scheduling/concurrency control][1], which is what I am
> actually after.

Schedule is a much better word than ordered because "schedule" is closer to what happens. However, I still think you'd need to generate an error if things don't happen in the order you expect (stopServer happening before doSomething, for example).

-Mark

> integrationTest := transaction {
> startServer.value
> } andThen {
> val n = name.value
> val jn = (jarName in assembly).value
> val v = version.value
> doSomething(n, jn, v)
> } andThen {
> stopServer.value
> }
>
> -eugene
>
> [1]: http://en.wikipedia.org/wiki/Schedule_(computer_science)
>

[eugene yokota]

On Thursday, August 22, 2013 8:04:55 PM UTC-4, Mark Harrah wrote:

> This reminds me of [scheduling/concurrency control][1], which is what I am
> actually after.

Schedule is a much better word than ordered because "schedule" is closer to what happens.  However, I still think you'd need to generate an error if things don't happen in the order you expect (stopServer happening before doSomething, for example).

integrationTest := transaction {
  startServer.value
} andThen { _ =>


  val n = name.value
  val jn = (jarName in assembly).value
  val v = version.value
  doSomething(n, jn, v)  


} andThen { _ =>
  stopServer.value
}

With my imaginary transaction macro, each block is executed as an independent task, so the ordering will be guaranteed:

val x1 = extracted.runTask(task1, state)._2
val x2 = extracted.runTask(task2(x1), state)._2
val x3 = extracted.runTask(task3(x2), state)._2
x3

Also I added `_ =>` in case someone wants values from the previous block.

So `integrationTest` task basically is off the grid with no dependencies, and tasks within a block will run more than once

if they showed up in another block. That's fine, and I think is a good tradeoff for having ordering.

It's sort of like trying express semicolons in batch mode `sbt startServer; doSomeTask; stopServer` (or human typing those tasks from the shell).

-eugene

  [1]: 5/22/2012 "Sequential chaining operators for Initialize" https://groups.google.com/d/msg/simple-build-tool/Yg17YxQ2su0/NCjwZx6IAIwJ

  [2]: 10/22/2012 "FYI found the way how to create wrapper for stb.Task[Unit] without dependsOn" https://groups.google.com/d/msg/simple-build-tool/7CMIQTCQdOY/u_uxTy0bWgMJ

[Mark Harrah]

By "off the grid with no dependencies" you mean nothing can depend on it, right? So, this is a new type of key. You'd probably want to be able to depend on it from other transactions, though?

> and tasks within a block will run more than once
> if they showed up in another block. That's fine, and I think is a good
> tradeoff for having ordering.

Given the semantics you are defining, where they are actually completely separate executions of the task engine, yes, that's fine.

> It's sort of like trying express semicolons in batch mode `sbt startServer;
> doSomeTask; stopServer` (or human typing those tasks from the shell).

How does this new type of task integrate with aggregation? For example, the semicolon example starts three task executions independent of the number of subprojects, but the new task approach seems like it would run 3 task executions for each subproject.

-Mark

> -eugene
>
> [1]: 5/22/2012 "Sequential chaining operators for Initialize"
> https://groups.google.com/d/msg/simple-build-tool/Yg17YxQ2su0/NCjwZx6IAIwJ
> [2]: 10/22/2012 "FYI found the way how to create wrapper for
> stb.Task[Unit] without dependsOn"
> https://groups.google.com/d/msg/simple-build-tool/7CMIQTCQdOY/u_uxTy0bWgMJ
>

[eugene yokota]

On Sat, Aug 24, 2013 at 12:57 PM, Mark Harrah <dmha...@gmail.com> wrote:

> So `integrationTest` task basically is off the grid with no dependencies,

By "off the grid with no dependencies" you mean nothing can depend on it, right?  So, this is a new type of key.  You'd probably want to be able to depend on it from other transactions, though?

I meant that it doesn't depend on tasks, and tasks don't depend on it.

I didn't think about the possibilities of transactions depending on another transactions, which could be interesting,

but if that adds work, I am ok with not having it until phase 2.

 

> It's sort of like trying express semicolons in batch mode `sbt startServer;
> doSomeTask; stopServer` (or human typing those tasks from the shell).

How does this new type of task integrate with aggregation?  For example, the semicolon example starts three task executions independent of the number of subprojects, but the new task approach seems like it would run 3 task executions for each subproject.

I didn't think about the aggregation. I'm guessing it would make sense to run first block in each subproject, wait till everything finishes, run second block in each subproject, etc similar to human typing it in. I'm open to other point of views.

-eugene