Exclude tag by default but allow manual include

[Ryan Boder]

Is there a way to have a tag that's excluded by default but I can include manually (for example with -n in SBT)?

I'd like to have a tag called Slow. These tests would not be run by default with test or test-only. But I could enable them manually when I want those test to run.

I've tried in build.sbt

testOptions in Test += Tests.Argument("-l", "Slow")

which does exclude the tests by default. However, when I try to include them by running

sbt "testOnly -- -n Slow"

the Slow tests still don't run. Is this possible? Is there a better approach for this?

[Bill Venners]

Hi Ryan,

On Fri, Jul 22, 2016 at 6:05 PM, Ryan Boder <ryan....@gmail.com> wrote:

Is there a way to have a tag that's excluded by default but I can include manually (for example with -n in SBT)?

I'd like to have a tag called Slow. These tests would not be run by default with test or test-only. But I could enable them manually when I want those test to run.

You can use ScalaTest's Slow tag:

http://www.artima.com/docs-scalatest-3.0.0-RC4/#org.scalatest.tags.Slow
http://www.artima.com/docs-scalatest-3.0.0-RC4/#org.scalatest.tagobjects.Slow$

 

I've tried in build.sbt

testOptions in Test += Tests.Argument("-l", "Slow")

which does exclude the tests by default. However, when I try to include them by running

sbt "testOnly -- -n Slow"

 

the Slow tests still don't run. Is this possible? Is there a better approach for this?

That behavior is as specified. The Scaladoc says:

If tags to include is specified, then only those tests whose tags are mentioned in the argument following -n and not mentioned in the tags to exclude, will be executed. 

That's here:

http://www.artima.com/docs-scalatest-3.0.0-RC4/#org.scalatest.tools.Runner$@specifyingTagsToIncludeAndExclude

So I had to chose which one of include/exclude overpowers the other, and I chose the wrong one for your use case.

The design of ScalaTest (in particular, the ability for users to override Suite's lifecycle methods) is supposed to allow people to do what they want if it isn't supported by the framework, so this will be a good "test case." It is not obvious to me how to do it, but I'll think about it and post again to this thread.

Bill

--
You received this message because you are subscribed to the Google
Groups "scalatest-users" group.
To post to this group, send email to scalate...@googlegroups.com
To unsubscribe from this group, send email to
scalatest-use...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/scalatest-users?hl=en
ScalaTest itself, and documentation, is available here:
http://www.artima.com/scalatest
---
You received this message because you are subscribed to the Google Groups "scalatest-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalatest-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Bill Venners
Artima, Inc.
http://www.artima.com

[Ryan Boder]

Hi Bill,

I didn't mean to imply it's behaving incorrectly. I appreciate the complexity of trying to fit many different use cases. I'm just curious whether there is a recommended way to get the behavior I'm looking for.

The most frequent use case on our team is a dev doing sbt test many times during development and wanting to run all tests - except for tests that take a long time. Those slow tests we normally would only run just before committing and in CI.

So in the most common case we would rather just type "test" instead of "testOnly -- -n Slow". I think we can achieve what we need by defining an alias.

Thanks for the tip about the Slow tag in scalatest. If you have a built-in Slow tag it might be worth considering having the default behavior exclude those and then have a testAll task includes the Slow tests.

Best Regards,

Ryan

[Bill Venners]

Hi Ryan,

On Sat, Jul 23, 2016 at 10:04 AM, Ryan Boder <ryan....@gmail.com> wrote:

Hi Bill,

I didn't mean to imply it's behaving incorrectly. I appreciate the complexity of trying to fit many different use cases. I'm just curious whether there is a recommended way to get the behavior I'm looking for.

No worries. I wanted to start by verifying for myself that this was a feature not a bug. I couldn't remember which of include and exclude I had decided should overpower the other. Nevertheless your use case is very reasonable.

One way that I can think of to do it is to override runTests in a trait that extends SuiteMixin and inspect the filter coming in. If the Slow tag is mentioned in both the include Set and the exclude Set, then make a new Filter that is exactly the same as the one coming in, but with Slow missing from both the include and exclude sets. Then you can do what you hoped would work in sbt. Exclude Slow tests by default:

testOptions in Test += Tests.Argument("-l", "Slow")

Then when you want to run them do:

sbt "testOnly -- -n Slow"

You would need to mix your SuiteMixin trait that overrides runTest into all of your test classes.

Bill

[Ryan Boder]

That looks like a good solution. Thanks! I guess if I'm going to make a SuiteMixin then I don't need to do anything in build.sbt. This seems to work.

import org.scalatest._
import org.scalatest.tags.Slow

trait ExcludeSlowByDefault extends SuiteMixin with Suite {

  override protected def runTests(testName: Option[String], args: Args): Status = {
    val tagName = classOf[Slow].getCanonicalName
    val filter = args.filter
    val isExcluded = filter.tagsToExclude.contains(tagName)
    val isIncluded = filter.tagsToInclude.isDefined && filter.tagsToInclude.get.contains(tagName)
    if (!isExcluded && !isIncluded) {
      val newFilter = Filter(filter.tagsToInclude, filter.tagsToExclude + tagName, filter.excludeNestedSuites, filter.dynaTags)
      super.runTests(testName, args.copy(filter = newFilter))
    } else {
      super.runTests(testName, args)
    }
  }

}

I noticed with the built-in Slow tag I have to type the fully qualified class name when I include it.

testOnly -- -n org.scalatest.tags.Slow

Is there a way to just type the class name? Like...

testOnly -- -n Slow

It's not a big deal at all, just curious.

Thanks again! I like this better than using aliases.

[Bill Venners]

Hi Ryan,

There's no way currently to drop the fully qualified name, but I wonder if we could do that as either an enhancement to Runner or at least the sbt integration. If no fully qualified name is given to a tag, then it is org.scalatest.tags? Or we could get more complicated and say you can specify a tag prefix as a ConfigMap parameter, or even as new command line parameter.

I do prefer to keep sbt and Runner in sync, so I think I'd lean towards the first idea: If a tag has no prefix we prefix org.scalatest.tags automatically.

Thoughts?

Bill

[Ryan Boder]

The first idea sounds simpler so I'd prefer that. We might just define our own slow tag for time being so we can name it "Slow" without a package name.

After some experimenting I think this is the best solution for us:

import org.scalatest._
import org.scalatest.tags.Slow

trait ExcludeSlowByDefault extends SuiteMixin with Suite {

  override protected def runTests(testName: Option[String], args: Args): Status = {

    val slowTagName = classOf[Slow].getCanonicalName
    val selectedTagName = "org.scalatest.Selected" // couldn't find the actual class?
    val filter = args.filter
    val isExcluded = filter.tagsToExclude.contains(slowTagName)
    val isIncluded = filter.tagsToInclude.isDefined && filter.tagsToInclude.get.contains(slowTagName)
    val isSelected = filter.tagsToInclude.isDefined && filter.tagsToInclude.get.contains(selectedTagName)
    val newTagsToExclude = if (!isExcluded && !isIncluded && !isSelected) {
      filter.tagsToExclude + slowTagName
    } else if (isExcluded) {
      filter.tagsToExclude - slowTagName
    } else {
      filter.tagsToExclude
    }
    val newFilter = Filter(filter.tagsToInclude, newTagsToExclude, filter.excludeNestedSuites, filter.dynaTags)
    super.runTests(testName, args.copy(filter = newFilter))
  }

}

It has a couple improvements over the one I posted above.

1) It works when running the test individually in IntelliJ (Selected in tagsToInclude).

2) It provides a way to un-exclude the excluded-by-default Slow tag with -l org.scalatest.tags.Slow.

It would be nice to parameterize it to ExcludeByDefault[A] so it could be used for tags other than slow. I tried it for a bit but ran into a problem determining the runtime class of the type parameter in the trait.

Oh well, this more than meets our needs as is.

Thank for the help!

Best Regards,

Ryan