0.13.0-M2

[Mark Harrah]

0.13.0-M2 is now available. This is an early, unpolished preview of sbt 0.13.

* See the Changes page[1] for the features and changes in 0.13.
* Please only use the 0.13.0-M2 launcher[2] with 0.13.0-M2. This
avoids unnecessary complications while 0.13 is in development.
* sbt 0.13 is NOT binary compatible with 0.12. Plugins will need
to be recompiled before they can be used. Milestones will not
be binary compatible with each other, only release candidates will.
* Known incompatibilities are collected on the wiki[3]. Please report
anything you have to do to make a 0.12 plugin or build work with 0.13.
* Many commits from 0.13 were backported to 0.12.x so that 0.13 is
mainly a feature release. See also the commit history[4].
* There won't be any native packages until the first release candidate.

Going forward, there is one main task left. The remaining tickets[5] are
symptoms of the same basic problem and will be fixed together.

Once these are fixed, there will be a beta release[6]. This beta release
will be the time to rebuild plugins and and update projects to find
incompatibilities and problems so that they can be fixed before RC1. I
don't expect many source incompatibilities once the beta is out, but it
won't be binary compatible with later releases.

Thanks,
Mark

[1] http://www.scala-sbt.org/0.13.0/docs/Community/ChangeSummary_0.13.0.html
[2] http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.0-M2/sbt-launch.jar
[3] https://github.com/sbt/sbt/wiki/0.13-Known-Issues
[4] https://github.com/sbt/sbt/commits/v0.13.0-M2
[5] https://github.com/sbt/sbt/issues?milestone=2&state=open
[6] beta meaning feature complete and ready for outside testing

[oscarvarto]

I started using sbt more than a year ago. At the very beginning it was really hard to me to use it. I have learned a little bit more and now I love sbt. I must thank all the people involved in the development of sbt. I am specially thankful for the increasing documentation of the project.

[Mark Harrah]

Thanks for the feedback. 0.13 is aimed at making it easier for new users to learn it. The bigger changes towards this goal are listed on the 0.13 change summary page[1]. I am interested in feedback from those who made it past the learning curve on whether the changes improve the initial experience. If not now, the first Beta release is the best time to look at and try out these changes since it will be feature complete. Once RC1 is out, it will be too late for major changes and we'll be more or less stuck with whatever is in 0.13.0 for a while.

For those looking for how the new features look in the context of an sbt book, check out Josh's early access sbt in Action[2], which uses 0.13.

-Mark

[1] http://www.scala-sbt.org/0.13.0/docs/Community/ChangeSummary_0.13.0.html
[2] http://www.manning.com/suereth2/

[Paul Phillips]

"libraryDependencies += "org.scala-lang" % "scala-reflect" % scalaVersion.value"

Is the presence of .value a consequence of needing something to trigger the macro, or is it something you consider desirable? If this could be

  libraryDependencies += "org.scala-lang" % "scala-reflect" % scalaVersion

is there some reason for it not to be?

[Mark Harrah]

It is required for the expression to type check before macro expansion. scalaVersion has type Initialize[String] and scalaVersion.value has type String. .value is technically lying, but it is marked @compileTimeOnly so it can't accidentally escape to bytecode if used outside of the proper macro.

The async project has a similar macro system with await(...) taking the role of .value:

https://github.com/scala/async

It does also help with the implementation of the macro and makes it easier to reuse in different context. (I'll leave it at that handwaving explanation for now.)

I do consider it desirable as a marker that the behavior of scalaVersion is different than if it were a plain def or val. I'm also generally leery of an implicit conversion M[T] => T.

-Mark

> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to simple-build-t...@googlegroups.com.
> To post to this group, send email to simple-b...@googlegroups.com.
> Visit this group at http://groups.google.com/group/simple-build-tool?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Paul Phillips]

On Tue, May 7, 2013 at 1:57 PM, Mark Harrah <dmha...@gmail.com> wrote:

 I'm also generally leery of an implicit conversion M[T] => T.

Me too (similarly, people are always asking for and not getting that most favored of T => M[T]s, x => Some(x)) but right now all I can see is a rain of unhelpful ".value"s on my logic.

[Mark Harrah]

On Tue, 7 May 2013 14:02:10 -0700
Paul Phillips <pa...@improving.org> wrote:

I did think a fair bit about what name to use, including alternatives like get (shorter) and apply (just parens if so desired). Other suggestions for consideration are welcome. There are quite a few places collisions with existing methods are likely unfortunately (I think that was why 'get' wasn't used) and I tried to come up with a name that works in several situations.

Other names I could use a review on are for input tasks:

http://www.scala-sbt.org/0.13.0/docs/Extending/Input-Tasks.html#using-other-input-tasks

In general, I think larger task bodies should be separated out into a separate method regardless of the ultimate name/syntax chosen. The libraryDependencies example isn't one of them, but a contrived example:

someTask := someTaskImpl(aTask.value, bTask.value)

[Paul Phillips]

On Tue, May 7, 2013 at 2:10 PM, Mark Harrah <dmha...@gmail.com> wrote:

I did think a fair bit about what name to use, including alternatives like get (shorter) and apply (just parens if so desired).  Other suggestions for consideration are welcome.

Is there anything in the macro business which will prevent me from writing my own implicit so I can leave off the .value? When macros are involved one becomes pessimistic about maintaining full generality. For instance maybe it will "work" but the macro needs to be completely aware of it, because it will have to deconstruct the implicit wrapper. At least that's my experience with similar things.

If I won't be able to write my own implicit, then apply and a pair of parentheses would be the least offensive to the clarity of the build logic; but I can see how that might easily create other issues.

 http://www.scala-sbt.org/0.13.0/docs/Extending/Input-Tasks.html#using-other-input-tasks

I can at least comment on some typos: "evaluated" is missing in the first sentence of "Using other input tasks", and the documentation for the inputKey lies and demands a triple dash. Is it "evaluated" and not "eval" just because you want the past tense? That's a big plate of characters for tense. Personally I'd be fine with eval (and parse.)

In general, I think larger task bodies should be separated out into a separate method regardless of the ultimate name/syntax chosen.  The libraryDependencies example isn't one of them, but a contrived example:

Once we're in the world of "code" it matters less, but I still prefer for my builds to be a series of boring and transparent declarations for as long as I can manage it. To that end, any noise is noisy: there's no good amount which isn't zero.

 

[Jason Zaugg]

On Tuesday, May 7, 2013 11:26:11 PM UTC+2, Paul Phillips wrote:

On Tue, May 7, 2013 at 2:10 PM, Mark Harrah <dmha...@gmail.com> wrote:

I did think a fair bit about what name to use, including alternatives like get (shorter) and apply (just parens if so desired).  Other suggestions for consideration are welcome.

Is there anything in the macro business which will prevent me from writing my own implicit so I can leave off the .value? When macros are involved one becomes pessimistic about maintaining full generality. For instance maybe it will "work" but the macro needs to be completely aware of it, because it will have to deconstruct the implicit wrapper. At least that's my experience with similar things.

I think you can solve that problem with, wait for it, more macros. Make your implicit from M[T] => T a macro that expands to `key.value`, and the `:=` or `+=` macros won't see a trace of it.

Personally I prefer being explicit here.

-jason

[Paul Phillips]

On Tue, May 7, 2013 at 10:53 PM, Jason Zaugg <jza...@gmail.com> wrote:

Personally I prefer being explicit here.

Is there some use to which I will be putting "scalaVersion" instead of "scalaVersion.value" in build definitions such that the lines will be blurred? If there is, I would like to hear about it; if there isn't, I don't see how it's more explicit to add ".value" a couple or a couple hundred times - just noisier. Explicit, like a semicolon.

[ijuma]

Hi Mark,

Nice work. :)

On Tuesday, 7 May 2013 13:24:27 UTC+1, Mark Harrah wrote:

Going forward, there is one main task left.  The remaining tickets[5] are
symptoms of the same basic problem and will be fixed together.

As I understand the ScalaTest guys were making some changes to the test framework support in SBT. Do you know if that is still planned for 0.13?

Best,

Ismael

[Mark Harrah]

Hey Ismael,

On Wed, 8 May 2013 01:35:02 -0700 (PDT)
ijuma <ism...@juma.me.uk> wrote:

> Hi Mark,
>
> Nice work. :)

Thanks!

Yes, it was merged here:

https://github.com/sbt/sbt/commit/f7ce8334c3acd16561b15b1bd7e244342c5d0874

Bill will be following up more information shortly.

-Mark

> Best,
> Ismael

[Mark Harrah]

Yes, some examples:

1. Going back to the using other input tasks section[1], you call evaluated or parsed on an Initialize[InputTask[T]]

2. The API for getting multiple values[2] involves calling key.all(<filter>), where key is Initialize[...] (or Initialize[Task[T]])

3. Some methods, like runMainTask, accept an Initialize[T]. In an ideal (or maybe just different) world, these would be macros that would allow the same syntax as the := macro. However, that isn't practical for any reasonable number of methods for maintenance and readability reasons.

[1] http://www.scala-sbt.org/0.13.0/docs/Extending/Input-Tasks.html#using-other-input-tasks
[2] http://www.scala-sbt.org/0.13.0/docs/Detailed-Topics/Tasks.html#getting-values-from-multiple-scopes

[Mark Harrah]

> > http://www.scala-sbt.org/0.13.0/docs/Extending/Input-Tasks.html#using-other-input-tasks
> >
>
> I can at least comment on some typos: "evaluated" is missing in the first
> sentence of "Using other input tasks", and the documentation for the
> inputKey lies and demands a triple dash.

Thanks. I've fixed them locally.

> Is it "evaluated" and not "eval"
> just because you want the past tense? That's a big plate of characters for
> tense. Personally I'd be fine with eval (and parse.)

Yeah, they seemed to go better with "value" and "result". If it were "get" instead, parse and eval would fit better. I'm ok with throwing out semi-consistency for shorter names if others are, though.

-Mark

[Harald Meland]

On Tue, May 7, 2013 at 10:57 PM, Mark Harrah <dmha...@gmail.com> wrote:

On Tue, 7 May 2013 13:13:00 -0700
Paul Phillips <pa...@improving.org> wrote:

> "libraryDependencies += "org.scala-lang" % "scala-reflect" %
> scalaVersion.value"
>
> Is the presence of .value a consequence of needing something to trigger the
> macro, or is it something you consider desirable? If this could be
>
>   libraryDependencies += "org.scala-lang" % "scala-reflect" % scalaVersion
>
> is there some reason for it not to be?

It is required for the expression to type check before macro expansion.  scalaVersion has type Initialize[String] and scalaVersion.value has type String.  .value is technically lying, but it is marked @compileTimeOnly so it can't accidentally escape to bytecode if used outside of the proper macro.

I'm not at all sure I understood all of that, but if Paul's request is specific to usage in the revision parts of libraryDependencies, wouldn't overloading the GroupArtifactID.% method with one that accepts an Initialize[String] be a possible solution?

--
Harald

[Doug Tangren]

What bothers me with it not being explicit is two fold

1) When things go wrong its harder to figure out whats actually going on. I actually didn't mind the old 

<<= (dep1, dep2).map { 

  (val1, val2) => ...

}

syntax. It helped me understand that those settings were different. It was `informing` syntax telling me dep1 and dep2 were special. Semicolons are informing syntax.

Using `value` is sugar. It's there to lessen the required syntax but is still helpful in that its still informing syntax. Without it, for new users there is no way to tell the difference between initialized and uninitialized settings and thus its harder to wrap your head around whats going on when something fails and you are debugging the problem. Debugging problems isn't a strong point for new users and is often the source for posts on this list! The information that was previous available in the form of syntax would now removed.

2) I don't want my builds to take longer because of extra `macroization`. This point may be less informed because I haven't really used macros yet, but my assumption is that they will require more work on the part of the scala compiler. Making the compiler do more work is good for the library author ( Mark ) but adds tax to the library user ( me )! I want fast builds. period.

[Mark Harrah]

On Wed, 8 May 2013 08:18:17 -0400
Doug Tangren <d.ta...@gmail.com> wrote:

> 2) I don't want my builds to take longer because of extra `macroization`.
> This point may be less informed because I haven't really used macros yet,
> but my assumption is that they will require more work on the part of the
> scala compiler. Making the compiler do more work is good for the library
> author ( Mark ) but adds tax to the library user ( me )! I want fast
> builds. period.

I don't think it is possible to make a general statement on whether a macro requires more work on the part of the compiler or not. Macros are user code executed at compilation time so whether it is faster or slower largely depends on what the non-macro alternative is and how the macro implementation is written. There is an overhead from things like loading the classes for the macro implementation, but how much that matters in practice will also depend on the specific macro.

In the case of this particular macro system in sbt, the compiler is definitely not doing more work and reducing the work for the library author. On the contrary, the macro implementation was a lot of work to implement and it does quite a bit of work (in terms of variety/complexity and not necessarily time) when it is called. The implicits and type inference for the pre-0.13 syntax probably take more time to compile.

Finally, there are more benefits than just syntax. One example is that the error messages can be improved. Try using a task to define a setting using the .value approach. You'd see something like:

/home/user/demo/build.sbt:13: error: A setting cannot depend on a task
name := "demo-" + scalaInstance.value.actualVersion
^

instead of:

/home/user/demo/build.sbt:13: error: type mismatch;
found : sbt.Def.Initialize[sbt.Task[String]]
required: sbt.Def.Initialize[String]
name <<= scalaInstance.map("demo-" + _.actualVersion)
^
or:

/home/user/demo/build.sbt:13: error: value actualVersion is not a member of sbt.Task[sbt.ScalaInstance]
name <<= scalaInstance("demo-" + _.actualVersion)
^

Another example is that the source location of settings in .scala files, including Defaults.scala, can be recorded. Try setting libraryDependencies += something and then 'inspect libraryDependencies'. You'll see something like:

[info] Defined at:
[info] (sbt.Classpaths) Defaults.scala:945
[info] /home/user/demo/build.sbt:13

[Bill Venners]

Hi All,

Yes, we spent most of March and April focused on this enhancement. I wanted to re-introduce it, as it did evolve some since we last talked about it on the list (mostly because of discoveries we made during implementation, but also we added some enhancements to the enhancement). For historical perspective, the last time we were discussing it was in this thread:

https://groups.google.com/forum/#!msg/simple-build-tool/CYRuwgzyLvQ/-qFAup7hh8gJ

It was kind of waiting on the back burner for a long time. Right now the latest incarnation is in the new-framework-api branch of the scalatest clone of Mark's repo:

https://github.com/scalatest/test-interface/tree/new-framework-api/src/sbt/testing

So one thing you can see is we moved it to sbt.testing. Originally Mark's idea was this could be used by other build tools, but since no other build tool used it (and since we can't use scala-tools.org anymore), we moved it to sbt.testing.

One main change from the old framework API is there is now a notion of a "run" with both a beginning and an end. The beginning is when the runner method is called on Framework:

https://github.com/scalatest/test-interface/blob/new-framework-api/src/sbt/testing/Framework.java

That means a run is starting, and the Runner that is returned represents the lifetime of that run. That allows the test framework to do things at the beginning of a run, like create directories, which was not possible in the old framework API because there was no notion of "run."

https://github.com/scalatest/test-interface/blob/new-framework-api/src/sbt/testing/Runner.java

Sbt will invoke the task method to get Tasks to execute. Each Task represents some kind of test job to do, such as running a single test or suite of tests. If the framework wishes to "reject" a requested task, though, it can do so by returning a Task that does nothing. For example ScalaTest has a @DoNotDiscover annotation. If sbt discovers a test class that is otherwise runnable, except it has a @DoNotDiscover annotation, ScalaTest will just return a task that does nothing from the Runner.task method--unless the explicitly specified flag is set. That is in fact the purpose of explicitly specified. If you say: 

test-only com.mycompany.MasterSpec

Then you probably really want sbt to run that thing, even if it is marked with @DoNotDiscover. So in that case, explicitly specified will be set to true, and ScalaTest will return a task that will ignore the @DoNotDiscover annotation and actually run it.

When sbt is done invoking task, it will invoke done on the Runner. That is a signal to the test framework that sbt will not ask for any more tasks for this run, i.e., that sbt is "done" with this run. When the framework is done sending log messages for this run, it will return from done. That's how sbt knows when the framework is done actually running the tasks for that run. The done method has a String result, which is a test-framework specific summary. This allows sbt to show the test framework's summary instead of, or in addition to, the standard sbt summary.

The selectors passed to run can be used to indicate specific tests to run, which will be a new feature in sbt 0.13, and to enable more fine-grained rerunning with test-quick. We'll have to wait and see what syntax Mark comes up with for specifying tests to run, but you will be able to just specify a fragment of a test name. That will end up as a WildcardTestSelector, and the framework can just execute all tests whose names that include the "wildcard" string. Maybe something like:

test-only *FredSpec(kittens)

Would run any tests with "kittens" in the test name in any discoverable test class whose fully qualified name ends in FredSpec.

When the test framework fires Events:

https://github.com/scalatest/test-interface/blob/new-framework-api/src/sbt/testing/Event.java

It includes a Selector. This selector can be passed to task next time. The purpose of this is to allow sbt's test-quick command to be more fine-grained in rerunning than it could with the old Framework API. Previously sbt could rerun a whole test class (what ScalaTest calls a Suite). Now with the Selectors it can rerun just one or two tests inside a class, or even one test inside a test class nested inside a test class nested inside the test class that was actually run from sbt. Essentially you won't have to wait for tests that passed last time to rerun the tests that failed last time. (Some people have very long-running tests, so this should speed things up in that case.)

Another thing we added to the framework API was support for running tests in a forked JVM. Sbt added this feature in 0.12 if I'm not mistaken, but without any added help from the framework API. The remoteArgs method on Runner supports this by giving sbt a way to ask for an additional args to give to the framework in the remote JVM. What ScalaTest does with that is start a server socket and include an argument for the remote JVM that includes the IP address of that server socket, and that way the remote ScalaTest instance can communicate events back to the "local" one. This allows a run that includes both remote and local parts to be reported in a cohesive manner, for example, with one big HTML report. If a test framework doesn't want to do anything special for the remote JVM it can just return an empty string from remoteArgs.

That's a quick overview. The main goal of this effort was to enable all test framework features to be accessible from sbt. In the ScalaTest case, for example, there were a lot of features available when using ScalaTest's Runner directly from the command line that just weren't possible through sbt because of constraints in the original framework API. I think everything in ScalaTest should now be accessible in a convenient way via sbt.

The way we implemented it, sbt will look for an implementation of the new framework API, and if it finds that, will use it. Else it will look for the implementation of the old Framework API, and if it finds that, will use it. Test frameworks can upgrade to the new framework API at their leisure, since the old one will continue to work as it did before. Even after upgrading, test frameworks should support both for a good long time, so that old versions of sbt will still work fine with new versions of the test framework.

 

We would like to get feedback and suggestions, and help other Framework implementors try it out. I think posting to this sbt list thread is the best place.

Thanks.

Bill

Best,

Ismael

[nafg]

On Wednesday, May 8, 2013 6:23:27 PM UTC-4, Bill Venners wrote:

One main change from the old framework API is there is now a notion of a "run" with both a beginning and an end. The beginning is when the runner method is called on Framework:

https://github.com/scalatest/test-interface/blob/new-framework-api/src/sbt/testing/Framework.java

That means a run is starting, and the Runner that is returned represents the lifetime of that run. That allows the test framework to do things at the beginning of a run, like create directories, which was not possible in the old framework API because there was no notion of "run."

Does that solve https://github.com/guardian/sbt-teamcity-test-reporting-plugin#known-limitations?

 

[Bill Venners]

Hi Nafg,

I don't believe this enhancement will address that issue as is, because what is needed is an test framework event indicating a test is starting. The old framework API did not have that, and I didn't know this was desired until now, so nothing got added for that in the new framework API. Essentially sbt gets one event per test, which indicates the result status, as before. Here's ye olde Event:

https://github.com/scalatest/test-interface/blob/new-framework-api/src/org/scalatools/testing/Event.java

And here's the new Event:

https://github.com/scalatest/test-interface/blob/new-framework-api/src/sbt/testing/Event.java

I'm not sure if TeamCity needs just to know times? Or does it really want a separate event for test starting? I'm also not sure if Mark would want this in sbt anyway, as it would also need to make it into the TestListener interface somehow. ScalaTest certainly fires such TestStarting events, so if the framework API and sbt's TestListener had such a notion ScalaTest at least could easily provide the events. We can take a look into this and see how easy it would be to add, but as I said, I'm not sure that's something Mark would want sbt to deal with. At some point the events would be more test framework-specific, and the way to integrate with Team City and the like would be by hooking up your chosen test framework to it directly.

Bill

 

[Mark Harrah]

Hey Bill,

It seems reasonable to add to me. This has been request before a couple of times, the latest being here:

https://github.com/sbt/sbt/issues/734

-Mark

> Bill

[ijuma]

Thanks Bill and Mark.

Best,

Ismael

[Bill Venners]

Hi Mark,

OK, I think the question is whether a test starting event is needed, or just a duration after it is over. The easiest would be to simply add a duration to Event. Then sbt's TestListener would still get one event for each test, which is simple, but it would contain the actual test duration. I'd like to know what Team City and other CI tools like that actually need, though. Does anyone here know?

Thanks.

Bill

[Naftoli Gugenheim]

This is basically what they're doing: http://confluence.jetbrains.com/display/TCD65/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ReportingTests

--
You received this message because you are subscribed to a topic in the Google Groups "simple-build-tool" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/simple-build-tool/816bxcVfRuU/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to simple-build-t...@googlegroups.com.

[Naftoli Gugenheim]

By the way if there's a way to get test reports in Jenkins I would really like to have it. Apparently you have to output JUnit xml, which is described here: http://stackoverflow.com/a/9691131/333643

[ijuma]

On Friday, May 10, 2013 8:30:50 AM UTC+1, nafg wrote:

By the way if there's a way to get test reports in Jenkins I would really like to have it. Apparently you have to output JUnit xml, which is described here: http://stackoverflow.com/a/9691131/333643

This has been possible for a long time. You can use native support in ScalaTest or Specs2 to output the relevant XML file. Alternatively, you can use the SBT plugin junit_xml_listener. I used the SBT plugin and moved to native ScalaTest support once that became available recently (in the 2.0 milestone releases).

Best,

Ismael

[Naftoli Gugenheim]

On Fri, May 10, 2013 at 3:34 AM, ijuma <ism...@juma.me.uk> wrote:

You can use native support in ScalaTest or Specs2 to output the relevant XML file.

Do you have a link? I haven't managed to find it.

[Jason Zaugg]

On Friday, May 10, 2013 8:09:07 AM UTC+2, Bill Venners wrote:

Hi Mark,

OK, I think the question is whether a test starting event is needed, or just a duration after it is over. The easiest would be to simply add a duration to Event. Then sbt's TestListener would still get one event for each test, which is simple, but it would contain the actual test duration. I'd like to know what Team City and other CI tools like that actually need, though. Does anyone here know?

I expect a TestStarted event would be really useful for tools like Activator, that might eventually render the test lifecycle in a GUI.

-jason 

[Som Snytt]

There are versions of Par-Test that do this.  People like to see the test start, hang for a while, then they ctl-c in frustration.

That's much more satisfying than fiddling with test timeouts.

--

[Josh Suereth]

It's actually friendly enough, from a UI perspective to show pending tests.   I'd +1 the feature for activator

[Mark Harrah]

Ok, a test started even sounds good to me then.

-Mark

> -jason

[Bill Venners]

Hi,

By "pending" tests, do people mean tests that have started but not yet finished? That's what you'd get with a test starting event.

Bill

[Som Snytt]

There's a useful distinction between when a test is "submitted" (the term I inherited in Par-Test, as in submitted to the queue) and when it is "started" (begins to run).

I don't know current usage of "pending" here, so, speaking only for myself.

It occurs to me that the number of tests in the run state is not intrinsically limited, for instance by the degree of parallelism.  I had code that tried to keep N tests running, but if you're really trying to keep N threads busy, you might start more than N tests.  So it can be very useful to see exactly what's running.

[Mark Harrah]

On Mon, 13 May 2013 14:43:53 -0700
Som Snytt <som....@gmail.com> wrote:

> There's a useful distinction between when a test is "submitted" (the term I
> inherited in Par-Test, as in submitted to the queue) and when it is
> "started" (begins to run).
>
> I don't know current usage of "pending" here, so, speaking only for myself.
>
> It occurs to me that the number of tests in the run state is not
> intrinsically limited, for instance by the degree of parallelism. I had
> code that tried to keep N tests running, but if you're really trying to
> keep N threads busy, you might start more than N tests. So it can be very
> useful to see exactly what's running.

I might be missing the point, but what threads are actually running is VM level or lower information. Otherwise, I agree the distinction between submitted and started is useful. I believe this API as currently designed would describe submitted.

A test is mapped to an sbt task, which is then scheduled. Something like #592 would be needed to track when those tasks are submitted to an actual thread pool. (Whether they are running is again a VM level thing.) But, if the user specifies sequential tests, that is currently handled by the test framework, which is responsible for executing those tasks on the same thread and sbt would only see one task for all of the tests. This could be changed to break things into tasks like in the concurrent case*, but currently there is an overhead in the sbt task engine to that**.

Overall, it is a good point to consider, thanks.

-Mark

#592, task progress listener: https://github.com/sbt/sbt/issues/592

* Bill: I think this would be implemented by the framework initially returning a single Task that executes the test and then returns the next Task in the sequence.

** It is not a fundamental problem or anything, it is just that tasks are typically DAGs and not usually longish linear chains, so this isn't optimized. It would only matter for a lot of very short tests in a suite.

[Bill Venners]

Hi Mark,

In the ScalaTest case, the "test starting event would happen when sbt actually fires a thread into the task to execute it. It wouldn't happen when the task was returned. The time between that TestStarting and the completion of the test is what ScalaTest reports as a duration, and this is what people usually care about: i.e., how long did it take to execute the body of the test once that code was invoked. They don't usually care how long it took after a test was put on an execution queue that the thread pool got around to giving it a thread. This also means that if a task is executing multiple tests sequentially, that's you'd get multiple test starting and test completion events (as we now get multiple test completion events). A task start and end event is not really something that is related to the Framework API, as I think you are getting at with your link to 592.

My concern is that I don't see much evidence that a test starting event is actually needed other than some vague "it might be cool" feelings. The one actual "ticket" I saw had to do with the lack of a duration in a team city report, and that could potentially be addressed by simply adding a duration to the existing Event class:

https://github.com/scalatest/test-interface/blob/new-framework-api/src/sbt/testing/Event.java

To add a test starting event, we'd probably want to refactor and have two subtypes of Event. I'd rather not do that if if it is not actually needed. One thing is I'm not sure how easy it would be for other test frameworks to implement. I know JUnit has a test starting event, but I don't know how easy it would be for specs2 or ScalaCheck. It would be straightforward for ScalaTest, but again I don't like adding features unless there's a real, actual, real use case. Perhaps tonight I'll ask Chee Seng to look at the TeamCity requirement and see if it can be solved by adding a duration. If so, then I'd rather just add a duration to Event.

Bill

[Som Snytt]

Sorry I'm not up to speed on the sbt side, so please consume a grain of salt or your preferred buffering agent.

I did just "upgrade" partest with the idea that the scalac test suite should be uniform with the framework; the upgrade was for ScalaCheck tests run by partest; my other fork of partest has more fine-grained test events. So that's my background.  Probably you have all hashed this stuff over, so AIA for naive comments.

Here's the recent issue, nothing mind bending, just show me when something has hung so I don't hang around waiting for it:
https://issues.scala-lang.org/browse/SI-7416

The trunk partest only shows when a test finishes.

My fork of partest just collates the submit order to report when tests get run and then finish (or fail to finish soon).

The use case: run tests for issue 1234, the tool will figure out which tests those are, but I can't guess if that is one test or a hundred -- a more elaborate use case would be a tool that guesses which tests have to run after an incremental change -- so I really want to see "1000 tests pending," "these are running now", and "these are done (pass/fail)".

Here's a use case to logically distinguish submit and start events: I submit a bunch of tests, but they aren't started in submit order because the clever framework knows that B can only succeed if A succeeds, so it only runs A first, even if B precedes A on submit.

Another use case is that during test development, you need to know if a test is hanging up the suite.  That could be for any reason -- blocking i/o on a test thread, etc -- but sometimes you want to monitor the test suite closely.  Maybe there are unexpected interactions between tests.

--

[Bill Venners]

Hi All,

Chee Seng and I did some quick investigating tonight. First it was quick to see that team city is being sent a "testStarted" message here, either nothing for success, "testFailed", for an error, or "testIgnored" for skipped, and finally a "testFinished" event. 

https://github.com/guardian/sbt-teamcity-test-reporting-plugin/blob/master/src/main/scala/com/gu/TeamCityTestReporting.scala

I was unable to locate any documentation for the teamcityReport method. If someone knows where the Javadoc or other documentation is for that I'd appreciate a pointer. But regardless I'm not sure what it does with it, unless it is display timing. The weird thing about that is that it wouldn't be giving the time that the test framework itself came up with. So I think it would be less accurate anyway, unless there is an optional, unused here way to send the test framework's duration.

One issue Chee Seng brought up is that sbt currently buffers events for a test class and sends them all at once at the end. Here is the code where this happens:

https://github.com/sbt/sbt/blob/0.13/testing/src/main/scala/sbt/TestFramework.scala#L75

So if we were to add a test starting event to the framework API, then it might get buffered in that spot and all come out at once anyway. So that would have to be changed, but maybe there's a reason it is buffering that would break. Mark, do you remember the reason it buffers? One thing I could think of is as a simple way to minimize (though probably not eliminate) interspersing of events for different test classes.

So now it looks to me like adding a duration to Event may not help at all with team city. Adding a test starting could help, but only if sbt actually fired it right away. I'd like to consult the team city documentation to see what they really do with "testStarting", though my guess from the "limitation" description is team city actually times the tests that way.

Bill

[Bill Venners]

Hi som-snytt,

On Monday, May 13, 2013 7:56:59 PM UTC-7, som-snytt wrote:

Sorry I'm not up to speed on the sbt side, so please consume a grain of salt or your preferred buffering agent.

I did just "upgrade" partest with the idea that the scalac test suite should be uniform with the framework; the upgrade was for ScalaCheck tests run by partest; my other fork of partest has more fine-grained test events. So that's my background.  Probably you have all hashed this stuff over, so AIA for naive comments.

Here's the recent issue, nothing mind bending, just show me when something has hung so I don't hang around waiting for it:
https://issues.scala-lang.org/browse/SI-7416

The trunk partest only shows when a test finishes.

My fork of partest just collates the submit order to report when tests get run and then finish (or fail to finish soon).

The use case: run tests for issue 1234, the tool will figure out which tests those are, but I can't guess if that is one test or a hundred -- a more elaborate use case would be a tool that guesses which tests have to run after an incremental change -- so I really want to see "1000 tests pending," "these are running now", and "these are done (pass/fail)".

Here's a use case to logically distinguish submit and start events: I submit a bunch of tests, but they aren't started in submit order because the clever framework knows that B can only succeed if A succeeds, so it only runs A first, even if B precedes A on submit.

Another use case is that during test development, you need to know if a test is hanging up the suite.  That could be for any reason -- blocking i/o on a test thread, etc -- but sometimes you want to monitor the test suite closely.  Maybe there are unexpected interactions between tests.

All these make sense to me, but not necessarily as features for sbt to adopt. These sound like desirable features for a test framework, not a build tool. I think at some point if you want info about what's being tested you may want to get that directly from the test framework.

Bill

[Naftoli Gugenheim]

Hi Bill, I think you were looking into the teamcity integration approach that involves writing a plugin for teamcity, using their Java API. However there is a much simpler mechanism, which I referred to, where teamcity parses a script's stdout. If you follow the link http://confluence.jetbrains.com/display/TCD65/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ReportingTests you'll see that all you have to do is println something like ##teamcity[testFinished name='testname' duration='<test_duration_in_milliseconds>']. So you can send it an arbitrary duration value.

--
You received this message because you are subscribed to a topic in the Google Groups "simple-build-tool" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/simple-build-tool/816bxcVfRuU/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to simple-build-t...@googlegroups.com.

[Mark Harrah]

I think the point of partest using the test-interface is that it both avoids needing to write code for each test framework and is a stable interface.

-Mark

> Bill
>
>
>
> >
> >
> >
> >
> > On Mon, May 13, 2013 at 5:19 PM, Bill Venners <bi...@artima.com<javascript:>

> >>> #592, task progress listener: https://github.com/sbt/sbt/**issues/592<https://github.com/sbt/sbt/issues/592>

[Mark Harrah]

I think it was a few things and what you say is probably one of them. The biggest reason was probably connecting two different APIs or spanning two incompatible test interface versions. It is possible there is a thread-safety reason as well, but that can be addressed other ways of course.

> So now it looks to me like adding a duration to Event may not help at all
> with team city. Adding a test starting could help, but only if sbt actually
> fired it right away. I'd like to consult the team city documentation to see
> what they really do with "testStarting", though my guess from the
> "limitation" description is team city actually times the tests that way.

Ok, let's do both if possible. A duration on Event is easy to consume if that's all a client needs and the test started is good for interactive reporting.

-Mark

[Bill Venners]

Hi Mark,

OK. I'll enhance our branch to differentiate between test starting and test completed, and to include a duration in test completed.

Bill

[Bill Venners]

Hi nafg,

Ah, that's good to know. This justifies adding the duration for sure.

Bill

[Bill Venners]

Hi Mark,

It makes sense for partest to implement the Framework API, because then sbt can drive it. My point was that some framework-specific features are probably best accessed directly from the test frameworks themselves, else the Framework API would get bloated, and that would make it harder for test frameworks to implement it. There's quite a bit of of ScalaTest still that isn't accessible via the framework API, for example, features don't necessarily exist in other frameworks, such as the distinction between ignored, pending, and canceled tests. To ScalaTest that distinction matters, but to sbt they are all just "skipped."  I think that makes sense for the Framework API, because JUnit just has ignored and not canceled or pending; specs2 just has pending and skipped, which is like canceled; ScalaCheck I'm not sure has any way of skipping, etc. partest's distinction between submitted and started test jobs may really only make sense for partest, so that kind of thing is probably best accessed by integrating tools directly with partest rather than trying to cram more into the Framework API.

 

Bill

[Mark Harrah]

My understanding was that partest was the client and was using
frameworks like ScalaTest or ScalaCheck through this interface. I may
have misunderstood, however.

-Mark

[Mark Harrah]

On 5/14/13, Bill Venners <bi...@artima.com> wrote:
> Hi Mark,
>
> OK. I'll enhance our branch to differentiate between test starting and test
>
> completed, and to include a duration in test completed.

Sounds good, thanks!

-Mark

>> an email to simple-build-t...@googlegroups.com <javascript:>.

>> > To post to this group, send email to

>> > simple-b...@googlegroups.com<javascript:>.

[Bill Venners]

Hi Mark,

That's interesting. If so then there *was* another user of the Framework API besides sbt.

Bill

[Mark Harrah]

On 5/16/13, Bill Venners <bi...@artima.com> wrote:
> Hi Mark,
>
> On Thursday, May 16, 2013 12:20:46 PM UTC-7, Mark Harrah wrote:
>>

Yeah, I'm not sure, but I think it was only very recently.

-Mark

>> > email to simple-build-t...@googlegroups.com <javascript:>.

>> > To post to this group, send email to

>> > simple-b...@googlegroups.com<javascript:>.

[Som Snytt]

> Yeah, I'm not sure, but I think it was only very recently.

Yeah, I'm not sure either.  It was only by chance that I noticed ScalaCheck had a recent update and I was touching related code.

However, for the distant goal of resurrecting the sbt build for Scala that will run tests, as Bill says, the scalac test flavors will be frameworked.  Hopefully partest could become just a command-line front-end for sbt test?  Guessing.

I'm not a fan of lowest common denominator for an API, albeit with sensitivity to API obesity.

Unsupported nuances like "skipped" can be faked by a reasonable implementation.

Two examples from this week, there are tests specific to Avian VM, and someone contributed a test that would succeed trivially on Windows (having failed to run /bin/ls).  One really needs a way to communicate "I'm skipping this test" to whoever is running me.  I don't even care what "skipped" means or if anyone cares that I'm skipping it; that's their business.  You didn't cancel it, and it didn't fail, I'm just refusing to run it.

So, I think the API should aim for things people commonly need even if they don't happen to, as opposed to what they actually do exploit.

I'll try to be more concrete in future.

ATM, I'm trying to understand Mark's Process code, and it suddenly occurred to me, maybe my mind is not subtle.

[Mark Harrah]

Hey Bill,

Where do we stand with this? In particular, what is left to be done for the new test interface for a 0.13.0 release?

-Mark

On Tue, 14 May 2013 14:20:12 -0700 (PDT)

> > an email to simple-build-t...@googlegroups.com <javascript:>.
> > > To post to this group, send email to simple-b...@googlegroups.com<javascript:>.