Re: [scalatest-users] Suffixes via scalatest-maven-plugin?

[Bill Venners]

Hi Alex

On Mon, Apr 1, 2013 at 4:26 PM, Alex Popiel <tapo...@gmail.com> wrote:
> Hello, folks.
>
> I've been banging my head against a wall trying to exclude integration tests
> from the test phase when running in maven (and having them run only in the
> integration-test phase, which is after packaging). It looks like the
> command-line Runner has a nifty -q option that could be used for this (with
> a value of """(?<!Integration)(?:Test|Spec)"""), but there doesn't appear to
> be any way to use that option via the scalatest-maven-plugin. Was this a
> deliberate decision, or just an oversight? If the latter, how open would
> people be to a patch which adds such support?
>
I think it likely was an oversight, and yes we'd accept a patch. I'd
like to have all of Runner's functionality accessible through the
scalatest-maven-plugin. After dinner I'll double check that we don't
already support it and merely neglected to document it.

> Note that excluding via tags is insufficient, because of stuff that happens
> in the BeforeAndAfterAll relying on packaging having been done.
>
Can you elaborate on this? You can't use tags because tags don't
exclude beforeAll/afterAll code, just tests, and you are wanting to
exclude the beforeAll/afterAll also?

Thanks.

Bill

> Thanks,
> - Alex
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "scalatest-users" group.
> To post to this group, send email to scalate...@googlegroups.com
> To unsubscribe from this group, send email to
> scalatest-use...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest
> ---
> You received this message because you are subscribed to the Google Groups
> "scalatest-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scalatest-use...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
Bill Venners
Artima, Inc.
http://www.artima.com

[Alex Popiel]

Hello, Bill.

On Apr 1, 2013, at 6:43 PM, Bill Venners <bi...@artima.com> wrote:

Hi Alex

On Mon, Apr 1, 2013 at 4:26 PM, Alex Popiel <tapo...@gmail.com> wrote:

Note that excluding via tags is insufficient, because of stuff that happens

in the BeforeAndAfterAll relying on packaging having been done.

Can you elaborate on this? You can't use tags because tags don't
exclude beforeAll/afterAll code, just tests, and you are wanting to
exclude the beforeAll/afterAll also?

That is precisely it.  For my integration test, I want to start an instance of my service in beforeAll via the production start/stop script (which requires the classes to be packaged in a jar), and stop it similarly in the afterAll.  The actual tests in the integration test spec send real service requests to the thus-started service instance to make sure that (among other things) the service actually starts listening for requests on the configured ports, marshaling works, etc.  Since starting and stopping the service is expensive (firing up a new JVM, among other things), I don't want to do it for each test.

Since the normal test phase runs before packaging, even running the suite with all tests excluded via tags is... unhappy.

- Alex

[Bill Venners]

Hi Alex,

One question I have is do your integration tests need to be in the
same package as the the unit tests? Unit tests are usually put in the
same package so they can access non-public things. But integration
tests I would imagine you might want to prevent from accessing
non-public things. If so, then you could also do this currently by
just putting the integration tests in a different package (maybe add
.integ on to the current package names). Then you could just use the
-m and -w that exists today, or their equivalents in the Maven plugin.

We will add -Q support to the maven plugin if it isn't already there,
but the other thing were planning to add was a way to specify a plain
old regex that will be used to determine discovery. If that existed,
would that address your need in a desirable way?

Two other possible ways would be to add a feature to ScalaTest to
enable users to specify either "tags to not discover" or "classes to
not discover". You could tag integration tests with @Integration and
unit tests with @Unit, then specify to not discover the test classes
tagged as @Integration. Or you could make all your integration tests
extend com.mycompany.myproject.IntegrationSpec and all your unit tests
extend com.mycompany.myproject.UnitSpec, then say to not discover one
or the other. Do either of these seem useful to you?

Thanks.

Bill

[Alex Popiel]

Hello, Bill.

My front-end integration tests (which I described previously) don't
need to be in the same package, but many of my back-end integration
tests (where I start services that the code I am testing is calling)
do need to be in the same package. (Technically, I could put the test
suites into a different package, and just have utility routines in the
same package... but that just stepped from the slippery slope all the
way into the abyss of ugly.)

Under the covers, -q actually is just a partially-anchored regex. I
am in fact relying on that to be able to do a negative match. Also
note that -Q is a distinct predefined invocation of -q "Spec|Suite".

There already is notation to exclude tests with a given tag; extending
that to class discovery would, IMO, actually be cleaner than going
based on the class name via the regex stuff. Were that available, I
would use that instead of the patch I built this morning and am in the
process of testing prior to submission.

- Alex

[Alex Popiel]

This patch appears to work for me:

apopiel@cerebros:~/src/scalatest-maven-plugin$ svn diff

Index: src/test/scala/org/scalatest/tools/maven/PluginTest.scala

===================================================================

--- src/test/scala/org/scalatest/tools/maven/PluginTest.scala   (revision 5236)

+++ src/test/scala/org/scalatest/tools/maven/PluginTest.scala   (working copy)

@@ -123,6 +123,10 @@

     configure(_.tests = comma("a\\, bc", "b", "c")) should containSuiteArgs("-z", "a, bc", "b", "c")

   }

 

+  def testSuffixes {

+    configure(_.suffixes = "(?<!Integration)(Test|Spec|Suite)") should containSuiteArgs("-q", "(?<!Integration)(Test|Spec|Suite)")

+  }

+

   def testMembers {

     configure(_.membersOnlySuites = comma("a", "b", "c")) should containSuiteArgs("-m", "a", "b", "c")

   }

Index: src/main/java/org/scalatest/tools/maven/AbstractScalaTestMojo.java

===================================================================

--- src/main/java/org/scalatest/tools/maven/AbstractScalaTestMojo.java  (revision 5236)

+++ src/main/java/org/scalatest/tools/maven/AbstractScalaTestMojo.java  (working copy)

@@ -85,6 +85,12 @@

     String tests;

 

     /**

+     * Regex of suffixes to filter discovered suites

+     * @parameter expression="${suffixes}"

+     */

+    String suffixes;

+

+    /**

      * Comma separated list of tags to include

      * @parameter expression="${tagsToInclude}"

      */

@@ -380,6 +386,7 @@

             addAll(parallel());

             addAll(suites());

             addAll(tests());

+            addAll(suffixes());

             addAll(membersOnlySuites());

             addAll(wildcardSuites());

             addAll(testNGConfigFiles());

@@ -422,6 +429,10 @@

         return suiteArg("-z", tests);

     }

 

+    private List<String> suffixes() {

+        return compoundArg("-q", suffixes);

+    }

+

     private List<String> membersOnlySuites() {

         return suiteArg("-m", membersOnlySuites);

     }

This patch is original work, and may be used with whatever license is most convenient.  Thanks go to my employer (Marchex, Inc) for allowing me to work on this during business hours and contribute it back to the community.

I use this with the following Maven configuration:

      <!-- enable scalatest -->

      <plugin>

        <groupId>org.scalatest</groupId>

        <artifactId>scalatest-maven-plugin</artifactId>

        <!-- version>1.0-M2</version -->

        <version>1.0-M4-SNAPSHOT</version>

        <configuration>

          <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>

          <junitxml>.</junitxml>

          <filereports>WDF TestSuite.txt</filereports>

        </configuration>

        <executions>

          <execution>

            <id>test</id>

            <goals>

              <goal>test</goal>

            </goals>

            <configuration>

              <suffixes>(?&lt;!Integration)(Test|Spec)</suffixes>

            </configuration>

          </execution>

          <execution>

            <id>integration-test</id>

            <phase>integration-test</phase>

            <goals>

              <goal>test</goal>

            </goals>

            <configuration>

              <suffixes>(?&lt;=Integration)(Test|Spec)</suffixes>

            </configuration>

          </execution>

        </executions>

      </plugin>

All appears happy now. :-)

- Alex

[Bill Venners]

Hi Alex,

On Tue, Apr 2, 2013 at 10:30 AM, Alex Popiel <tapo...@gmail.com> wrote:
> Hello, Bill.
>

> My front-end integration tests (which I described previously) don't
> need to be in the same package, but many of my back-end integration
> tests (where I start services that the code I am testing is calling)
> do need to be in the same package. (Technically, I could put the test
> suites into a different package, and just have utility routines in the
> same package... but that just stepped from the slippery slope all the
> way into the abyss of ugly.)
>
> Under the covers, -q actually is just a partially-anchored regex. I
> am in fact relying on that to be able to do a negative match. Also
> note that -Q is a distinct predefined invocation of -q "Spec|Suite".
>
> There already is notation to exclude tests with a given tag; extending
> that to class discovery would, IMO, actually be cleaner than going
> based on the class name via the regex stuff. Were that available, I
> would use that instead of the patch I built this morning and am in the
> process of testing prior to submission.
>

OK. Would you prefer to annotate every integration test with
@Integration or something like that? Or do you already have a class or
trait that integration tests extend that unit tests do not?

Are you working on a patch for the scalatest-maven-plugin? We would
warmly welcome that.

Thanks.

Bill

[Alex Popiel]

Hello, Bill.

I do not currently have a common ancestor for all my integration tests, but being able to specify a trait as the distinguishing characteristic feels a bit more scala-ish than using an annotation everywhere.  My codebase is currently small enough that running through a few dozen files to mark them in whichever way ends up being published is not a problem.

- Alex

[Bill Venners]

Hi Alex,

OK. Thanks for your input. I think we'll add support for influencing
discovery based on both annotations and superclasses. Your use case
will fit the superclass one better most likely. The annotations one
will be more for situations in which a class is tagged with a test
tag, such as "Slow", and people want to just not discover classes
annotated like that when they want to exclude Slow tests.

Bill