"Invoke-Pester -EnableExit" not behaving as expected

[Jamie Thomson]

Hi folks,

We're using pester in our CI build and we (obviously) want the build to fail when any tests fail. We know we have to use -EnableExit to do this Hence, we have this:

$FailedTestTally = Invoke-Pester -EnableExit -Path $pesterFolder -OutputXml $pesterFolder\TestResult.xml
Write-Host "Failed Powershell unit test tally: $FailedTestTally"

Here's the output from running this:

Passed: 29 Failed: 1

Failed Powershell unit test tally:

As you can see no value is going into $FailedTestTally. My colleague Lloyd Holman tracked this down to function Exit-WithCode in TestResults.ps1. We changed the definition of that function to:

function Exit-WithCode ($FailedCount) {
    return $FailedCount
    #$host.SetShouldExit($FailedCount)
}

and then we got the output we expected:

Passed: 29 Failed: 1

Failed Powershell unit test tally: 1

Note that Lloyd had a Twitter conversation with Matt Wrock about this on 6th August: https://twitter.com/lloydholman/status/496952110601486336 although he never got a round to raising an issue as he said there that he would.

For now we're going to use our modified version of Exit-WithCode, let us know if we should issue a pull request.

regards

Jamie

[Matt Wrock]

Hi Jamie,

The issue I see with your change is that a failure will not emit a positive exit code which many build systems expect. If your concern is capturing the number of failures, you should be able to do that with:

Invoke-Pester -EnableExit -Path $pesterFolder -OutputXml $pesterFolder\TestResult.xml

$FailedTestTally = $LASTEXITCODE

Write-Host "Failed Powershell unit test tally: $FailedTestTally"

Hope that helps.

[Jamie Thomson]

Hi Matt,

Thanks for the (very) swift reply.

Unfortunately we've already tried what you suggest and we're still not getting back the failed error count.

However, I have found out some new info. Currently we are calling Invoke-Pester from inside a psake task. If I call Invoke-Pester directly (without our mod to Exit-WithCode) then I get the expected result:

PS>$FailedTestTally = Invoke-Pester .\Pester -EnableExit
Executing all tests in '...
...
Tests completed in 3.63s
Passed: 29 Failed: 1
PS>$FailedTestTally
1

I can only conclude that psake is affecting this somehow. I shall continue to investigate.

regards

Jamie

[Dave Wyatt]

I'm a bit confused.  The purpose of -EnableExit is to set an exit code from PowerShell.exe, not to create a return value from Invoke-Pester.  The syntax you're using ($FailedTestTally = Invoke-Pester -EnableExit) should never work, whether psake is involved or not.  This is for CI solutions that base their decisions on the return code of a process.

There are two other options for getting non-console feedback from Pester.  For solutions that can make use of NUnit XML files, there's the -OutputXml parameter.  However, if you're using psake, you may find that -PassThru is the most convenient option.  When you set the -PassThru switch, you do get a return value from Invoke-Pester.  It's a custom object which contains all of the same information you could get from the NUnit XML file, including the number of failed tests:

$result = Invoke-Pester

$FailedTestTally = $result.FailedCount

[Dave Wyatt]

Oops, forgot to include -PassThru in that example:

$result = Invoke-Pester -PassThru

$FailedTestTally = $result.FailedCount

[Jamie Thomson]

Awesome, that works a treat. Thanks Dave. Two massive doses of help in two days, I owe you a large quantity of beer, sir!