pass/fail output from Taurus build task

[Darren Smith]

Hi,
I have an Azure pipeline running the Taurus 1.14.1 Taurus tool runner build task. I want to fail the test run using the average response time parameter, but I also want to output the xml for the run so that I can format this and display the results on our Azure build dashboard. I added some yaml configuration to the build task to try to achieve this, but I'm not having any joy... I can get the build to run successfully and output the xml using this Taurus Configuration YAML... 

reporting:

- module: final-stats

  summary: true

  percentiles: true

  failed-labels: true   

  test-duration: true

  dump-xml: C:/Temp/Jmeter/taurus-output/results.xml

However, When I introduce the passfail module, the Taurus tool runner build task fails and doesn't produce any xml output. The console output for the task recognises that the test should have failed based on the criteria but the task doesn't output anything. I have tried several variations of the following Taurus Configuration YAML, but absolutely no joy. Any help would be hugely appreciated, thanks

reporting:

- module: passfail

  criteria:

  - avg-rt of TC01_LoginPage>5ms for 1s, continue as failed

dump-xml: C:/Temp/Jmeter/taurus-output/results.xml

[Alla]

Hello Darren,

Please update to the newest Taurus version. We do not support the old ones. And if after that the error persists, please let me know.

BR,
Alla

[Darren Smith]

Thank you Alla for your advice. 

I am now installing Taurus v1.15.0 using the Azure build task Taurus Tool Installer, but unfortunately I'm still seeing the same issue. I want to fail a test if it exceeds a given value for avg-rt, and I want it to output the results to an xml file so I can then use this for displaying in our Azure dashboard. It works fine if I do a basic summary report, but if I try to do the passfail report the Run Taurus build task fails and doesn't output any results.

I have tried several variations of the following Taurus Configuration YAML on the build task...
 reporting:

- module: passfail

  criteria:

  - avg-rt of TC01_LoginPage>5ms for 1s, continue as failed

  dump-xml: C:/Temp/Jmeter/taurus-output/results.xml

In the console log output for the build I get the message... 

WARNING: Failed: avg-rt>5ms for 1 sec

It doesn't give me anymore than that, and strangely the output above this says the test has passed...

| TC01_LoginPage | OK | 100.00% | 0.834 | |  

The task then fails and doesn't output anything.

Any advice would be much appreciated

Thanks

[Alla]

Hello Darren,

We do not support dumping results in passfail module. We only do that for final stats, because this is what we store in the dump files.

Also, you can find passfail criteria warnings and errors in bzt.log in your artifacts directory.

BR,
Alla

[Darren Smith]

Thanks again for your help Alla.
Ok so that's fine but what would be useful is if the junit results file from the passfail module contained the test name and the avg_rt value. The output I get at the moment is as follows, which has the name attribute set to the criteria rather than the name of the failing test, and it doesn't include the actual  avg_rt value...

<testsuites>

  <testsuite name="bzt_pass_fail" package_name="bzt">

    <testcase classname="bzt-181495706495" name="avg-rt of TC01_LoginPage&gt;5ms for 1s">

      <error message="Failed: avg-rt&gt;5ms for 1 sec" type="pass/fail criteria triggered"/>

    </testcase>

  </testsuite>

</testsuites>

Is there any way you can configure the passfail module to output the test name and the avg_rt value in the junit result xml file?? This is what I have for the  Taurus Configuration YAML  at the moment...

reporting:

- module: passfail

  criteria:

  - avg-rt of TC01_LoginPage>5ms for 1s, continue as failed

- module: junit-xml

  filename: C:/Temp/Jmeter/taurus-output/results.xml

  test-duration: true

  data-source: pass-fail

If it is not possible to configure the passfail module to include the test name and avg_rt value, then do you have any suggestions as to how we could achieve this??

Thanks,
Darren 

[Darren Smith]

Hi Alla,
so to be clear, I would like to be able to configure the passfail module to be able to output something like this. Is this possible, and if so how? If not then is there an alternative module I can use??

<testsuites>

  <testsuite name="bzt_pass_fail" package_name="bzt">

    <testcase classname="bzt-181495706495" name=" TC01_LoginPage " avg_rt="10ms">

      <error message="Failed: avg-rt&gt;5ms for 1 sec" type="pass/fail criteria triggered"/>

    </testcase>

  </testsuite>

</testsuites>

Thanks,

Darren

[Alla]

Hi Darren,

I am afraid it is not possible to do and it is not reasonable due to features of Taurus architecture. We neither store nor process the data the way it is suitable to dump to xml files.

I suggest you try to extract data from JUnitTester.csv.

BR,
Alla

[Darren Smith]

OK, thank you

[Daniel Oprean]

What is passfail useful for if we cannot see results in the report? Are we meant to process bzt.log and make custom reports?