executing multiple Thens

[powerdude]

Hello,

Have you considered a way to execute multiple Thens in one test run.
For instance, given:

Scenario 1: Account is in credit
Given the account is in credit
And the card is valid
And the dispenser contains cash
When the customer requests cash
Then ensure the account is debited
And ensure cash is dispensed
And ensure the card is returned

i'd like the last 2 tests to run even if the first Then failed: Could
you code up the Verify to catch the exceptions and output everything?

cliff

[Jason Diamond]

Hi Cliff,

Verify could definitely be written to not skip steps after the first
failing step.

I thought about allowing an option that could let you continue to
execute steps even if one failed, but I couldn't think of why I'd want
to do that.

Could you describe why you think it would be useful?

Also, all steps are treated equally. There's really no difference
between a given, when, or then except for the label. If you only
wanted it to continue when a then step failed, we'd have to check the
step type, but that wouldn't be difficult.

Thanks.

--
Jason

[cliff vaughn]

Sure I can provide more info.  Using the above example, i would like to if the cash and the card got dispensed, even if the account wasn't debited like I hoped.  It would reduce the number of times i have to run the tests if I could see all failures and fix them rather than get the first failure, update code, maybe get a second failure with the next then, update code, and finally maybe get a third failure, update code and then finally see a successful run

--
thanks

cliff

[Jason Diamond]

Hi Cliff,

I just checked in a new property that allows this. If you have
Subversion access and want to check the code out, you can give it a
spin. Otherwise, I'll put together another release either today or
tomorrow.

I called the property ContinueAfterFailedSteps. It's a little wordy so
if you have a better suggestion, I'd love to hear it. It defaults to
false, by default. To get the behavior you want, you could just set it
to true before calling any of the Verify methods.

--
Jason

[cliff vaughn]

Thanks Jason, i'll give it a go.  My suggestion would be to add a boolean property to Verify() since that's where it matters. 

Also, do you think it's possible to work on the output some.  Working with this fixture:

[TestFixture]
    public class CalculatorScenarios2 : Scenario
    {
        [SetUp]
        public void Setup()
        {
            Given(a_new_calculator);
            When(adding_the_numbers, 1, 2);

        }
        [Test]
        public void AddingTwoNumbers()
        {
            Then(the_result_should_be, 1);
            Verify();

        }

        [Test]
        public void AddingTwoNumbers2()
        {
            Then(TheResultShouldBe2, 2);
            Verify();

        }

        private Calculator _calculator;

        public void a_new_calculator()
        {
            _calculator = new Calculator();
        }

        public void adding_the_numbers(int a, int b)
        {
            _calculator.Add(a, b);
        }

        public void the_result_should_be(int result)
        {
            Assert.That(_calculator.Result, Is.EqualTo(result));
        }

        private void TheResultShouldBe2(int result)
        {
            Assert.That(_calculator.Result, Is.EqualTo(result));
        }
    }

I currently see this for a failed test when I use TestDriven.NET:


------ Test started: Assembly: Calculator.Tests.dll ------

  Given a new calculator
  When adding the numbers 1 and 2
! Then the result should be 1
  Expected: 1
  But was:  3

TestCase 'Calculator.Tests.Integration.CalculatorScenarios2.AddingTwoNumbers' failed:
  Expected: 1
  But was:  3
    C:\Users\cliff\Documents\Projects\Tests\Integration\GMailServiceTests.cs(102,0): at Tests.Integration.CalculatorScenarios2.the_result_should_be(Int32 result)
    c:\code\behaven\BehaveN\StepMethod.cs(124,0): at BehaveN.StepMethod.<>c__DisplayClass1`1.<GetAction1>b__0()
    c:\code\behaven\BehaveN\StepMethod.cs(55,0): at BehaveN.StepMethod.Invoke(Object[] parameters)
    c:\code\behaven\BehaveN\Specifications.cs(142,0): at BehaveN.Specifications.<>c__DisplayClass4.<ExecuteStep>b__3()
    c:\code\behaven\BehaveN\Specifications.cs(170,0): at BehaveN.Specifications.ExecuteAndReport(StepAction invoker, String description, StepType stepType, IConvertibleObject convertibleObject)
    c:\code\behaven\BehaveN\Specifications.cs(141,0): at BehaveN.Specifications.ExecuteStep(StepType stepType, String description, Delegate stepDelegate, Object[] args)
    c:\code\behaven\BehaveN\StepInfo.cs(108,0): at BehaveN.DelegateStepInfo.Execute(Specifications specifications)
    c:\code\behaven\BehaveN\Specifications.cs(368,0): at BehaveN.Specifications.Verify()
    C:\Users\cliff\Documents\Projects\Tests\Integration\GMailServiceTests.cs(76,0): at Tests.Integration.CalculatorScenarios2.AddingTwoNumbers()

  Given a new calculator
  When adding the numbers 1 and 2
! Then the result should be2 2
  Expected: 2
  But was:  3

TestCase 'StockTracker.Tests.Integration.CalculatorScenarios2.AddingTwoNumbers2' failed:
  Expected: 2
  But was:  3
    C:\Users\cliff\Documents\Projects\Tests\Integration\CalculateTests.cs(108,0): at Calculator.Tests.Integration.CalculatorScenarios2.TheResultShouldBe2(Int32 result)
    c:\code\behaven\BehaveN\StepMethod.cs(124,0): at BehaveN.StepMethod.<>c__DisplayClass1`1.<GetAction1>b__0()
    c:\code\behaven\BehaveN\StepMethod.cs(55,0): at BehaveN.StepMethod.Invoke(Object[] parameters)
    c:\code\behaven\BehaveN\Specifications.cs(142,0): at BehaveN.Specifications.<>c__DisplayClass4.<ExecuteStep>b__3()
    c:\code\behaven\BehaveN\Specifications.cs(170,0): at BehaveN.Specifications.ExecuteAndReport(StepAction invoker, String description, StepType stepType, IConvertibleObject convertibleObject)
    c:\code\behaven\BehaveN\Specifications.cs(141,0): at BehaveN.Specifications.ExecuteStep(StepType stepType, String description, Delegate stepDelegate, Object[] args)
    c:\code\behaven\BehaveN\StepInfo.cs(108,0): at BehaveN.DelegateStepInfo.Execute(Specifications specifications)
    c:\code\behaven\BehaveN\Specifications.cs(368,0): at BehaveN.Specifications.Verify()
    C:\Users\cliff\Documents\Projects\Tests\Integration\GMailServiceTests.cs(84,0): at Calculator.Tests.Integration.CalculatorScenarios2.AddingTwoNumbers2()


0 passed, 2 failed, 0 skipped, took 1.18 seconds (NUnit 2.5).


It would be nice not to see the duplication of the error message and also not to see the stack trace from BehaveN.

thanks for the great work.

cliff

--
thanks

cliff

[Jason Diamond]

Hi Cliff,

Thanks for the feedback.

I didn't add an overload to Verify because then I would also have to
add overloads for the VerifyText, VerifyFile, and
VerifyEmbeddedResource methods. All of these call Verify internally.

In my actual work where I'm using BehaveN, I tend to use VerifyText or
VerifyEmbeddedResource more than feeding the steps one at a time with
Given, When, and Then, and then invoking Verify.

As far as the output goes, I agree that the duplication of the error
message is a little bothersome. As is the stack trace for the BehaveN
code. I'll look to see if it's possible to do something to filter out
the part of the stack trace you don't care about, but I'm not sure it
will be possible.

I just realized something regarding the new property. If turned on,
this means that Verify will never throw an exception. This will make
it look like everything passed. I'll have to do something at the end
of Verify to throw an exception if any exceptions were caught while
executing the steps.

--
Jason

[Jason Diamond]

I've got the output down to this:

Given a new calculator
When adding the numbers 1 and 2
! Then the result should be 1

TestCase 'CalculatorScenarios2.AddingTwoNumbers' failed:
BehaveN.VerificationException :

Expected: 1
But was: 3

C:\code\behaven\BehaveN.Tests\CalcTests.cs(54,0): at
CalculatorScenarios2.the_result_should_be(Int32 result)

Given a new calculator
When adding the numbers 1 and 2
! Then the result should be2 2

TestCase 'CalculatorScenarios2.AddingTwoNumbers2' failed:
BehaveN.VerificationException :

Expected: 2
But was: 3

C:\code\behaven\BehaveN.Tests\CalcTests.cs(59,0): at
CalculatorScenarios2.TheResultShouldBe2(Int32 result)

0 passed, 2 failed, 0 skipped, took 0.81 seconds (NUnit 2.5).

--
Jason

[cliff vaughn]

Very nice!!  I haven't gotten the code yet, but how does it look if you put both Thens in the same test?

[Jason Diamond]

I haven't finished this yet, but here's the current output for two
failing thens in the same test:

------ Test started: Assembly: BehaveN.Tests.dll ------

Given a new calculator
When adding the numbers 1 and 2

! Then the result should be 4
! And the result should be 5

TestCase 'BehaveN.Tests.CalculatorScenarios2.AddingTwoNumbers'
failed: BehaveN.VerificationException :
Expected: 4
But was: 3
C:\code\behaven\BehaveN.Tests\CalculatorTests.cs(72,0): at
BehaveN.Tests.CalculatorScenarios2.the_result_should_be(Int32 result)

0 passed, 1 failed, 0 skipped, took 0.81 seconds (NUnit 2.5).

Notice how it's only outputting the first exception. You can still see
that both steps failed because of the ! symbols in front of them.

It might be possible to combine all of the exception messages and
stack traces into one exception. I might try doing that tomorrow.

--
Jason