postman/newman executor unable to fail the request even if the assertions failed/not met

[paolocan...@gmail.com]

we are trying to use postman collection for simple performance test.

We added text assertions in postman collection as similar to below

pm.test("Body matches string", function (){

    pm.expect(pm.response.text()).to.include("FAILTHETEST");

});

We ran the build and taurus didn't fail it.

NewmanExecutor.ldjson captured that the "status":"FAILED" due to the assertion

but bzt log didn't reflect these failures and still passes the test

[Dimitri Pribysh]

Hi,

Thanks for reporting. We also caught this issue this week. We're investigating it and are going to fix it in the next release.

  Dimitri

--
You received this message because you are subscribed to the Google Groups "codename-taurus" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codename-taur...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/codename-taurus/18bfd5ef-0665-40ab-bbde-25c32b3b36b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dmitri Pribysh]

And it's fixed in master branch.

You can grab the latest snapshot from http://gettaurus.org/docs/DeveloperGuide/#Python-Egg-Snapshots to use the fixed version of Taurus.

Thanks for reporting this one.

[paolocan...@gmail.com]

Thank you will check it out

[paolocan...@gmail.com]

Hey Dmitri, 

The issue loooks to still be there.

The postman assertions are not recognized by bzt.

The results still shows none of the transactions failed

On Thursday, September 6, 2018 at 8:25:54 AM UTC-4, Dmitri Pribysh wrote:

[Dmitri Pribysh]

Hey,

Are you sure you're running Taurus snapshot? Can you share the results and/or postman collection?

  Dimitri

To view this discussion on the web visit https://groups.google.com/d/msgid/codename-taurus/aa3153f3-234b-4712-9969-380a0654184a%40googlegroups.com.

[paolocan...@gmail.com]

Hey Dimitri,

Try if you can view here?

https://drive.google.com/open?id=1_lFFtG6mHfuIpq1PS8ox3DNJlee3jK7Z

I have uploaded the logs. (postman collection included)

NewmanExecutor.ldjson showed that it failed due to assertions

csv and xml reports. Also, the console showed no errors.

Thanks Dimitri

[Dmitri Pribysh]

Thanks for providing all the artifacts.

When running Taurus with your config (on the latest taurus@master), i can see bzt recognizing that postman tests failed.

14:46:37 INFO: Request label stats:
┌─────────┬────────┬───────┬────────┬─────────────────────┐
│ label   │ status │  succ │ avg_rt │ error               │
├─────────┼────────┼───────┼────────┼─────────────────────┤
│ FakeAPI │  FAIL  │ 0.00% │  0.070 │ Body matches string │
└─────────┴────────┴───────┴────────┴─────────────────────┘

The return code was 0, because by default Taurus is running in load testing mode, and for load tests a single failing sample is usually not a sufficient reason to have the whole test marked as failed.

If you want Taurus to mark the whole test as failed (and have non-zero exit code) if at least one test has failed, you can write a pass/fail criterion for that. Something like `fail of FakeAPI>0% for 1s, stop as failed` should suffice.

Just for the record, Taurus can also be launched in functional mode with `-func` CLI argument. In this mode Taurus will have non-zero return code if there would be at least 1 failed sample. But i think you would be better off with adding another pass/fail criteria.

To view this discussion on the web visit https://groups.google.com/d/msgid/codename-taurus/ee5474f7-4c04-42e5-bdee-359b71512388%40googlegroups.com.

[paolocan...@gmail.com]

Thanks Dmitri,

So yes, now I've installed the one from http://gettaurus.org/docs/DeveloperGuide/#Python-Egg-Snapshots

bzt-1.13.0.26.tar.gz, to be exact. However, mine still shows status OK.

And you mentioned you ran yours from taurus@master, is that the same from the link?

Appreciate your patience

17:24:08 INFO: Request label stats:

+---------+--------+---------+--------+-------+

| label   | status |    succ | avg_rt | error |

+---------+--------+---------+--------+-------+

| FakeAPI |   OK   | 100.00% |  0.096 |       |

+---------+--------+---------+--------+-------+

[Dimitri Pribysh]

Yep, it should be the same.

Just for validation, does Taurus print the following line when you launch it?

> INFO: Taurus CLI Tool v1.13.0.26

To view this discussion on the web visit https://groups.google.com/d/msgid/codename-taurus/dea85c91-2645-4c63-9329-de0c0cb1d96d%40googlegroups.com.

[paolocan...@gmail.com]

Hey Dimitri,

Looks like its something on my machine, used a different machine and got the failed notification

16:10:19 INFO: Request label stats:

┌─────────┬────────┬───────┬────────┬─────────────────────┐

│ label   │ status │  succ │ avg_rt │ error               │

├─────────┼────────┼───────┼────────┼─────────────────────┤

│ FakeAPI │  FAIL  │ 0.00% │  0.053 │ Body matches string │

└─────────┴────────┴───────┴────────┴─────────────────────┘

THANK YOU Dimitri !

really big help

Regards,

Paolo