filter example execution

[Mikael Högqvist]

Hi,

is there a way to filter which examples to execute when running tests
via sbt? In specs it was possible to tag tests with a name and then
use the --accept flag to decide which tests to run.

Thanks,
Mikael

[etorreborre]

Hi Mikael,

You can use the 'ex' option (http://etorreborre.github.com/specs2/
guide/org.specs2.guide.Runners.html#Arguments) like that:

test-only mySpec -- ex "this example"

This however doesn't work with the "test" task in sbt since it doesn't
accept command-line arguments.

Eric.

[Pete]

Eric,

Please excuse my newbie questions (both new to Scala and Specs/Specs2)
as I am joining Scala from the Java side of things. I am selecting the
test tools that our group will be using and Specs2 fits nicely as we
are using other BDD tools for other technologies (jasmine for js,
etc).

I see the differences you point out between Specs and Specs2 wrt test
selection at execution time. The tag approach in Specs seemed fairly
straight forward, give the specification a name and then filter/
exclude by that name at execution time.

I can see in Specs2 that the selector is now a regexp provided on the
command line and what I'd like to confirm is that when given i.e.
'test-only mySpec -- ex "this example"' the expression "this example"
is used to match against the name of the specification.

In our Java test environment we are using junit 4.x categories to
"tag" our tests. This is accomplished using the @Category annotation,
where one can provide a set of values to be used as a selector for a
test class or method. We group our tests into different categories,
"checkin", "regression", etc. but a single test can belong to multiple
categories.

For all our different technology we have imposed the same rules for
test definition/execution. Be able to mark a test class or test method
with one or more categories and then be able to selectively run those
tests based on the category given at runtime.

Now my question. Is this possible using Specs2? If so are there
concrete examples that you could point me at?

Regards,

Pete

[etorreborre]

Hi Pete,

> I can see in Specs2 that the selector is now a regexp provided on the
> command line and what I'd like to confirm is that when given i.e.
> 'test-only mySpec -- ex "this example"' the expression "this example"
> is used to match against the name of the specification.

No the "this example" expression is used to filter individual
examples, not the whole specification.
At the specification level you can define commands in sbt and filter
on specification names. Here is an example of my sbt Project file:

override def includeTest(s: String) = Seq("Spec", "Suite", "Unit",
"all").exists(s.endsWith(_)) && !s.endsWith("FeaturesSpec") ||
s.contains("UserGuide")
|| s.matches("org.specs2.guide.*")

Now on the subject of tags, I've had some hesitation to re-introduce
them in specs2 as I wondered if the existing features would not be
enough the get the work done:

- selection based on names in sbt
- creation of a specification including other specifications:

class checkin extends org.specs2.mutable.Specification {
include(spec1)
include(specs2)
// and so on
}

I can imagine adding tags to a specification and examples but the main
difficulty would be to pass the required tags to the sbt test command
because it accepts no arguments.
So the question is, if you had tags, how were you thinking of using
them with sbt?

Eric.

[Mikael Högqvist]

Hi Eric,

I tried this from the sbt console (0.7.4 with scala 2.8.1):

test-only mySpec -- ex "my example"

but it doesnt work. It runs all of the examples. But with def is =
args(ex = "my example") ^ ... in the MySpec code it works. I'm using
args(sequential = true) already, could this make a difference?

Thanks,
Mikael

[etorreborre]

Hi Mikael,

That's a bug indeed. This is fixed now and I'll publish an updated 1.1-
SNAPSHOT in the next 2 hours.

Eric.

[etorreborre]

Hi Pete,

One of my issues with tags which I didn't mention was the difficulty
of implementing that elegantly with immutable data.

Thinking about it again, this led me to find another solution for tags
with additional fragments. It looks like this:

class MySpec extends Specification with Tags { def is =

"this is a first example" ! e1^ tag("checkin")^
"this is a second example" ! e2^ tag("unit")^
p^
"and some integration tests" ^ section("integration")^
"test 1" ^
"test 2" ^
...
"test n" ^ section("integration")^
end
}

The Tags trait simply provides new "Tag" fragments and the fragment
selection algorithm will keep/reject fragments based on their position
relative to those tags:

* in an acceptance spec, a fragment before a Tag fragment is
considered "tagged" by this one (and subject to inclusion/exclusion)
* in an acceptance spec, the fragments "between" 2 similar "section"
Tags are considered as Tagged by the section label.
if there's no closing section tag, all the fragments till the end
are considered as tagged

This way you can also define your project wide tags:

trait MyProjectTags {
def checkin = tag("checkin")

def integration = tag("integration")
def integrationExamples = section("integration")
}

What do you think?

Eric.

On Mar 24, 5:06 am, Pete <cur.m.g...@gmail.com> wrote:

[Pete]

Eric,

Thanks for the replies.

I'l try to repeat what I understand as a possible solution.The Tag
trait would add the necessary methods to my spec so that one could
define what a tag and section name would be. By including the tag() or
section() in the fragments one establishes a marker where the tag/
section entry identifies which example(s) belong to a tag/section.

The point I am getting lost at is at runtime, where one would need to
filter (or select) the examples that match a given tag, lets say
"integration". Would this require a modification the the current
method of example filtering or a separate way of accomplishing this?

On another note, this project I am working on is heavy in java and
thus uses ant as its build environment. Will the existing and proposed
solution work in that environment as well.I am using
SpecificationWithJunit as my base class.

Once again, I apologize my lack of Scala knowledge. You know life,
have to get things done quickly regardless of one's experience.

Also, are you considering making this solution part of Specs2 or am I
on my own :)

Pete

[etorreborre]

Hi Pete,

> The point I am getting lost at is at runtime, where one would need to
> filter (or select) the examples that match a given tag, lets say
> "integration". Would this require a modification the the current
> method of example filtering or a separate way of accomplishing this?

There will be 3 ways of declaring which tags you want to execute:

- from the specification args (declared in the specification class)
- from the command line arguments (when using test-only in sbt, or
specs2.run on the command line)
- from -D arguments, which I guess will be the easiest way to pass
arguments from Ant

> On another note, this project I am working on is heavy in java and
> thus uses ant as its build environment. Will the existing and proposed
> solution work in that environment as well.I am using
> SpecificationWithJunit as my base class.
>
> Once again, I apologize my lack of Scala knowledge. You know life,
> have to get things done quickly regardless of one's experience.
>
> Also, are you considering making this solution part of Specs2 or am I
> on my own :)

Yes my plan is to make this available soon in 1.1-SNAPSHOT. It might
be difficult to do this
week-end but I hope to have finished that by next week.

Eric.

[etorreborre]

Hi Pete,

In the latest SNAPSHOT you can have a go at using tags by either:

- adding the org.specs2.specification.Tags trait in an acceptance
spec
- adding the org.specs2.mutable.Tags trait in a unit spec

I still have some documentation to do for the unit spec but there is
already some documentation in
the org.specs2.guide.SpecStructure class for the acceptance spec.

I'll do that after the migration and testing of specs/specs2 to the
latest Scala 2.9.0.RC1 but
I thank you for sending your feedback on tags in the meantime if you
have any.

Eric.

On Mar 26, 5:54 am, Pete <cur.m.g...@gmail.com> wrote:

[Pete]

Eric,

Thanks and I'll have a go at it.

Pete

[Pete]

Eric,

Just finished trying the Unit tagging and it works great.

Thanks.

[Pete]

Eric,

Continuing on the unit examples with sections I found that the section
piece of the tagging is not being picked when one includes the section
name.

Running the following spec (from you example in docs) with a -
Dinclude=("checkin") yields no tests run. Without the include or
running with one of the other tag values I do see tests being run from
this spec.

I also reviewed the tagging support with one of our developers. His
only concern is how the tag function breaks the readability of the
DSL. He was wondering if the tag function could be used as the section
function is, as a marker after the example rather than as before.

Thanks again for providing this functionality on such short notice.

package a.b.c

import org.specs2.mutable._
import java.io._

class MySpec extends SpecificationWithJUnit with Tags {

"this is some introductory text" should {
"and the first group of examples" in {
tag("feature 1", "unit")
"example 1" in success
tag("integration")
"example 2" in success
}
}

"and the second group of examples" in {
"example 3" in success
"example 4" in success
}
section("checkin")
}

[etorreborre]

Hi Pete,

> Dinclude=("checkin") yields no tests run. Without the include or

I think that this is because you included ("checkin"). Can you try to
just write -Dinclude=checkin? I tried it and it works for me.

> DSL. He was wondering if the tag function could be used as the section
> function is, as a marker after the example rather than as before.

There are actually several "tags fragments" enabling the tagging
functionality:

- Tag: tags the next fragment
- TaggedAs: tags the previous one
- Section: starts a section with the next fragment or ends a section
with the previous one
- AsSection: starts a section with the previous fragment or ends a
section with the previous one

In the case of a mutable.Specification I propose that we can write:

tag("integration") // create a Tag fragment
"this example" >> {

}
"this other example" >> {
...
} taggedAs("integration") // create a TaggedAs fragment

"this third example" >> {
...
} asSection("integration") // create an AsSection fragment
// and so on
// ...
// ..
"this last example" >> {
...
} asSection("integration") // create an AsSection fragment to close
the section

We could also allow the tagging of a single example like this:

"this other example" >> {
...
} tag("integration") // create a TaggedAs fragment

But newlines would be significant! Because

"this other example" >> {
...
}
tag("integration") // create a Tag fragment tagging the next one

What do you think? Maybe the last proposition is the best one,
compact, doing the job, and hopefully
not to ambiguous w.r.t the newline thing?

Eric.

[etorreborre]

Hi Pete,

I went ahead and implemented the latter suggestion for you to play
with.

So now, you can write:

tag("integration")

"this example" >> {
...
}

OR

"this example" >> {
...
} tag("integration")

Tell me what you think about that.

Eric.

[Pete]

Eric,

I'll give this a try and review the complete example with the
developer from your earlier post. I missed the nuances of asSection,
TaggegAs from the source. Yesterday I was able to get the section to
run by adding a section above the examples.

I believe the developer will like this solution. The newline issue
already grabbed us when trying to find out how and where to place
section markers.

In this version will:

} tag("unit")

be the same as
}
tag("unit")

Or will the latter end up marking the next fragment?

Pete

[Pete]

Eric,

I made sure I ran the ant junit run with -Dinclude=checkin with the
following spec and no examples were run

class TaggedSpecification extends SpecificationWithJUnit with Tags {
"this is some introductory text" >> {
"and the first group of examples" >> {

tag("feature 1", "unit")
"example 1" in success

"example 2" in success tag ("integration")
}
}
"and the second group of examples" >> {

"example 3" in success
"example 4" in success
} section ("checkin")
}

with the spec altered to the following specs example 3 and 4 run

class TaggedSpecification extends SpecificationWithJUnit with Tags {
"this is some introductory text" >> {
"and the first group of examples" >> {

tag("feature 1", "unit")
"example 1" in success

"example 2" in success tag ("integration")
}
}
section ("checkin")
"and the second group of examples" >> {

"example 3" in success
"example 4" in success
} section ("checkin")
}

class TaggedSpecification extends SpecificationWithJUnit with Tags {
"this is some introductory text" >> {
"and the first group of examples" >> {

tag("feature 1", "unit")
"example 1" in success

"example 2" in success tag ("integration")
}
}
section ("checkin")
"and the second group of examples" >> {

"example 3" in success
"example 4" in success
}
}

Not sure if it matters - this is with ant 1.7.1 Mac OS X 10.6.7

Thanks

Pete

[etorreborre]

Let's look at your examples:

1. example 1

> class TaggedSpecification extends SpecificationWithJUnit with Tags {
>   "this is some introductory text" >> {
>     "and the first group of examples" >> {
>       tag("feature 1", "unit")
>       "example 1" in success
>       "example 2" in success tag ("integration")
>     }
>   }
>   "and the second group of examples" >> {
>     "example 3" in success
>     "example 4" in success
>   } section ("checkin")
>
> }

In this case, running with include=checkin will not include anything
since the section is opened at the end but is not enclosing any
fragment. That being said
you would think that it should at least tag "example 4". I'll have a
look at that.

2. Example 2

> class TaggedSpecification extends SpecificationWithJUnit with Tags {
>   "this is some introductory text" >> {
>     "and the first group of examples" >> {
>       tag("feature 1", "unit")
>       "example 1" in success
>       "example 2" in success tag ("integration")
>     }
>   }
>   section ("checkin")
>   "and the second group of examples" >> {
>     "example 3" in success
>     "example 4" in success
>   }
>
> }
>

Running with include=checkin includes all the fragments after the
section opening, that's to be expected.

3. Example 3

> class TaggedSpecification extends SpecificationWithJUnit with Tags {
>   "this is some introductory text" >> {
>     "and the first group of examples" >> {
>       tag("feature 1", "unit")
>       "example 1" in success
>       "example 2" in success tag ("integration")
>     }
>   }
>   section ("checkin")
>   "and the second group of examples" >> {
>     "example 3" in success
>     "example 4" in success
>   } section ("checkin")
>
> }

The section is closed and running with include=checkin includes
example 3 and example 4 as expected.

I have some ideas on what to do with example 1 that I'm going to
investigate.

Thanks for your experiments,

Eric.

On Apr 1, 5:59 am, Pete <cur.m.g...@gmail.com> wrote:
> Eric,
>
> I made sure I ran the ant junit run with -Dinclude=checkin with the
> following spec and no examples were run
>
>

> with the spec altered to the following specs example 3 and 4 run
>

>

[etorreborre]

Ok I just published a SNAPSHOT where example 1 runs ok.

So if you put "section" on the same line as a "should" block, this
marks all the enclosed examples as part of the section.

E.

[Pete]

Eric,

Thank you for adding this support into specs2. We are already taking
advantage of this feature.

However, I have one more request wrt filters. The filter names
selected are quite ubiquitous and since we need to express these as
ant properties the likelihood of a name collision is of concern to us.

I was wondering if you could name these in a way where they are
isolated to the specs2 environment.

Pete

[etorreborre]

I hope this answers the question:
http://etorreborre.github.com/specs2/guide/org.specs2.guide.Runners.html#From+system+properties

You can name properties as -Dspec2.name=value to avoid collisions.

E.

[etorreborre]

Sorry, typo, the format is: -Dspecs2.name=value

On Apr 12, 7:15 am, etorreborre <etorrebo...@gmail.com> wrote:
> I hope this answers the question:http://etorreborre.github.com/specs2/guide/org.specs2.guide.Runners.h...