How to exclude scenarios from being run on the CI server when using mstest

[mgilden]

I have just introduced SpecFlow on my current project. Our goal is to
specify acceptance criteria for user stories using SpecFlow before
implementing them. These acceptance tests will fail until all tasks
for the story are done, which will keep our CI server in the red
permanently. So I am looking to find a way to exclude work in progress
from being executed on the CI server.

What I am thinking is tagging scenarios under development with a @wip
tag and have the CI server know not to run these. Is there any way to
accomplish this, when using mstest?

Martin

[Jean-Charles Meyrignac]

If you use nUnit and the specflow.exe command line, you can generate a report with red/green and purple for unimplemented tests (I posted a message on this list explaining how to achieve it).

IIRC, Gallio is also able to colorize the unimplemented tests correctly, and is probably working with msTests, though I'm not sure if it's able to generate reports.

Another thing we adopted recently (this week !) is that we moved the unimplemented features into another part of our project.
And our product manager only adds the minimal required acceptance tests at the beginning of our iteration.

JC

[Xerxes Battiwalla]

We solve this problem with NUnit by leaving our step definitions with the default ScenarioContext.Current.Pending(), and once the feature is ready to be wired up, we literally just wire up the acc test as rqd. this way our CI is never red, and nor should it be - it's not a broken test!

I'm not sure how MSTest would react to the ScenarioContext.Current.Pending(), though...

-xerx

--
To unsubscribe, reply using "remove me" as the subject.

[Jonas Bandi]

Hi,

with both NUnit and MSTest you can use the @ignore tag:

@ignore
Scenario: All Strikes
  Given a new bowling game
  When all of my rolls are strikes
  Then my total score should be 300

This tag gets translated into a  IgnoreAttribute of the respective framework. TestRunners then do not run these tests.


With NUnit however I suggest using a custom tag (i.e. WIP):

@WIP
Scenario: One single spare
   Given a new bowling game
   When I roll the following series: 2,8,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
   Then my total score should be 29

The tag then gets translated into a NUnit Category when saving the feature-file in VS.

When running the tests with nunit-console you can use the the /exclude flag so skip those tests:

C:\Users\jba\Dev\Projects\SpecFlow-Examples\BowlingKata\BowlingKata-Nunit>nunit-console /labels /nologo /exclude:WIP Bowling.Specflow\bin\Debug\Bowling.Specflow.dll



For MSTest you can also use Test-Lists, although it is a bit tedious:
In your .vsmdi (Test List Editor) you can define a test list "non-WIP". Then you can put tests that should be executed into this test-list.
With MSTest you can then use the option /testlist to execute only tests in this list:

C:\Users\jba\Dev\Projects\SpecFlow-Examples\BowlingKata\BowlingKata-MsTest>MSTest /testmetadata:Bowling.vsmdi /testlist:Not-WIP

Unfortunately it seems not possible to exclude a test-list...
The VS2010 Version of MSTest seems also to support TestCategories (http://msdn.microsoft.com/en-us/library/ms182489%28VS.100%29.aspx). we have to look into this for future versions of SpecFlow.



I hope this helps? Any other ideas?
jonas

--
mail: jonas...@gmail.com
web: www.jonasbandi.net
blog: blog.jonasbandi.net
twitter: twitter.com/jbandi