Taurus do not shows requests errors correctly with Locust 0.13.2

[savva.ge...@gmail.com]

I am usig this Negative Locust scenario that will return 400 errors:

from locust import TaskSet, task, HttpLocust


class Scenario(TaskSet):

    @task
    def success_task(self):
        self.client.get("/users/1")

    @task
    def failure_test(self):
        self.client.get("/users/fail")

class LocustRunner(HttpLocust):
    task_set = Scenario
    host = "http://jsonplaceholder.typicode.com"
    min_wait = 500
    max_wait = 1000

And I am using such blazemeter config:

execution:
- executor: locust
  concurrency: 15
  ramp-up: 5s
  hold-for:  10s
  scenario: load-tests

scenarios:
  load-tests:
    timeout: 30
    think-time: 1s
    default-address: http://jsonplaceholder.typicode.com
    keepalive: true
    script: ./locust_scenario.py

reporting:

- module: console
- module: passfail
  criteria:
    - avg-rt >4500ms for 10s, continue as failed
    - avg-rt >5000ms for 5s, stop as failed
    - p99>8000ms for 10s, stop as failed
    - failures >0% for 500ms, stop as failed
    - hits < 5 for 30s, stop as failedActual result:

Actual result

locustio==0.13.2
bzt==1.14.0

Log returns errors in incorrect way with Locust 0.13.2:

14:50:51 INFO: Request label stats:

┌──────────┬────────┬─────────┬────────┬─────────────────────────────────────────────────────────────────────────────┐

│ label    │ status │    succ │ avg_rt │ error                                                                       │

├──────────┼────────┼─────────┼────────┼─────────────────────────────────────────────────────────────────────────────┤

│ /users/1 │   OK   │ 100.00% │  0.061 │                                                                             │

│ [empty]  │  FAIL  │   0.00% │  0.000 │ __on_request_failure() got an unexpected keyword argument 'response_length' │

└──────────┴────────┴─────────┴────────┴─────────────────────────────────────────────────────────────────────────────┘

Expected result

Log throws error correctly only with Locust version: 0.12.2 

locustio==0.12.2
bzt==1.14.0

14:55:45 INFO: Request label stats:

┌─────────────┬────────┬─────────┬────────┬───────────────────────────────────────────────────────────────────┐

│ label       │ status │    succ │ avg_rt │ error                                                             │

├─────────────┼────────┼─────────┼────────┼───────────────────────────────────────────────────────────────────┤

│ /users/1    │   OK   │ 100.00% │  0.062 │                                                                   │

│ /users/fail │  FAIL  │   0.00% │  0.052 │ Not Found for url: http://jsonplaceholder.typicode.com/users/fail │

└─────────────┴────────┴─────────┴────────┴───────────────────────────────────────────────────────────────────┘

[grey....@gmail.com]

Hello. 

Thanks for your report, looks like locust 0.13.2 is incompatible with your script. It's unavoidable with taurus 1.14.0

We're working on fix release, if execution of this locust script is critical for you at the moment please use taurus 1.13.9 instead.

---

Taras 

[savva.ge...@gmail.com]

The bug is specific only for Locust >= 0.13.0 ... 
Thanks. I will be looking for fixes

вторник, 10 декабря 2019 г., 15:58:00 UTC+2 пользователь grey....@gmail.com написал:

[Nate]

Hi, do you have a time frame for when this bug will be fixed? I tried downgrading to Taurus 1.13.9 but I still get the error as well.

Thanks

[grey....@gmail.com]

hello.

As I see it's all about incompatibility of locust 0.13+ and your script (am I right?). And can be fixed in locust or your script only.

What about taurus the problem is in requirements as 1.14.0 requires locustio>=0.13 explicitly. so you can choose one of the follow:
1. use bzt 1.13.9 and install old locust manually (e.g. 'pip install locustio=0.12.2')
2. use bzt 1.14.0 and fix your script (revise locust doc for compatibility explanation)

We gonna remove locust from requirements in 1.14.1 (February), but if you won't fix your script #1 will be necessary.

---

Taras

[Nate]

Hi, I am using bzt 1.14 and locust 0.13.5. 

This is the part of my script that is falling over. I'm explicitly calling a failure for when certain content is not detected on the page (I can't use status codes for this test).

In this test case, when the response does not contain the word "Summary" in a certain area of the page, the response should be marked as failed (for testing purposes). However, if this failure is triggered, I receive the error below:

This means that I can't actually see what path is failing or why.

When you say "fix your script", can you please advise what needs to be changed to make my script compatible? I can't find anything on the Taurus Locust page that looks like it would help.

Thanks

[Alla]

Hello Nate,

Could you please share your script? And could you please provide your expected and actual results?

BR,
Alla

[smo...@atlassian.com]

Hello! Could you please give me some advice? I have a similar issue with ` __on_request_failure() got an unexpected keyword argument 'response_length'` as on the top of this topic.
I have locustio ==  0.13.5  and bzt == 1.14.11. Do I need to downgrade locust or wait for fix from your side?

[grey....@gmail.com]

please share artifacts dir

---

Taras

[smo...@atlassian.com]

Hello! I've found the issue that I have. But I bumped into another one - https://groups.google.com/forum/#!topic/codename-taurus/KfSKDjyz-E4 (incorrect kpi jtl order for locust executor)
I have created PR  https://github.com/Blazemeter/taurus/pull/1311/ . Could you please take a look?
I figure out that column order set in this file bzt/resources/locustio-taurus-wrapper.py


On Thursday, May 21, 2020 at 7:52:33 AM UTC+3, grey....@gmail.com wrote:

please share artifacts dir

[grey....@gmail.com]

Please don't open tons of topics for one question even if you feel it extra urgent.

---

Taras

[ukloo...@gmail.com]

I had run into the exact same issue as Nate using:

locustio==0.13.5
bzt==1.14.2

I believe the issue is that Locust 0.13.5's response.failure() method is doing the following (source):

events.request_failure.fire(
    request_type=self.locust_request_meta["method"],
    name=self.locust_request_meta["name"],
    response_time=self.locust_request_meta["response_time"],
    response_length=self.locust_request_meta["content_size"],
    exception=exc,
)

which fires all event handlers like this (source):

def fire(self, reverse=False, **kwargs):
    if reverse:
        self._handlers.reverse()
    for handler in self._handlers:
        handler(**kwargs)

However, the handler implemented in Taurus 1.14.2 does not accept kwargs (source):

def __on_request_failure(self, request_type, name, response_time, exception):
    self.num_requests -= 1
    self.writer.writerow(self.__getrec(request_type, name, response_time, 0, exception))
    self.fhd.flush()
    self.__check_limits()

Knowing this, we can write our own implementation of failure() that checks the arguments of each event handler, and calling the Taurus handler differently. The following is my implementation:

def failure(response, exc):
    """
    Report the response as a failure.

    exc can be either a python exception, or a string in which case it will
    be wrapped inside a CatchResponseError.

    This method should be used rather than response.failure() as it accounts
    for the incompatible __on_request_failure event handler implemented in
    the Taurus Locust wrapper by checking the signature of each handler, and
    calling the function appropriately

    Example::
        with self.client.get("/", catch_response=True) as response:
            if response.content == b"":
                failure(response, "No data")
    """
    if isinstance(exc, six.string_types):
        exc = CatchResponseError(exc)

    request_type = response.locust_request_meta["method"]
    name = response.locust_request_meta["name"]
    response_time = response.locust_request_meta["response_time"]
    response_length = response.locust_request_meta["content_size"]

    print('request_failure event handlers: {}'.format(
        events.request_failure._handlers))

    for handler in events.request_failure._handlers:
        argspec = inspect.getfullargspec(handler)
        if argspec.varkw is None and argspec.args==['self', 'request_type', 'name', 'response_time', 'exception']:
            # Special case for incompatible Taurus handler
            handler(request_type, name, response_time, exc)
        else:
            handler(
                request_type=request_type,
                name=name,
                response_time=response_time,
                exception=exc,
                response_length=response_length,
            )
    response._is_reported = True

This method can then be called instead of response.failre(result).

Obviously an issue with this fix is that it will break when either Locust or Taurus is updated, but this does resolve the issue for me on the versions above.

[derevyag...@gmail.com]

Hi,

We are already working on this issue.

I will write again when the problem is resolved.

[derevyag...@gmail.com]

Error fixed, you can find required changes in latest version of taurus

[rgust...@illuminateed.net]

Will this be released soon?  I only see 1.14.2 from two months ago available.

[derevyag...@gmail.com]

You can use an unstable version while required changes are not in the release:

Linux: bzt-1.14.2.10691-py2.py3-none-any.whl

Windows: TaurusInstaller_1.14.2.10691_x64.exe

Check this page to find the latest unstable version if needed