Problem wit distributed testing and reporting on large file uploads using stack taurus+jmeter+blazemeter reporting

[Ilya Evlampiev]

Hi!

I'm doing a distributed testing for docker cloud environment. The testing requires large (1.5Gb) files upload as a test step with Http Request.

I have set up the taurus/blazemeter image as a jmeter server (master), while justb4/jmeter (a bit customized to use jmeter 4.0 inside) is used as agent (slave). So, I'm able to start distributed tests and see alongside with the console output of taurus?blazemeter, jmeter.log for slave and blazemeter online report a docker stats value. it shows me the incoming and outgoing traffic per both containers, ie master and slave.

So, I'have succedd with 20Mb, 500Mb files, the slavese successfully report to master jmeter, taurus on master jmeter container successfully reports to blazemeter. But when the file size is 1.5 Gb I'dont see file upload http request in blazemeter report at all (even in errors tab)

I got for taurus something like this:

08:00:38 INFO: Current: 1 vu    0 succ    1 fail    127.229 avg rt    /    Cumulative: 127.259 avg rt, 100% failures
08:01:33 WARNING: Failed to send data, will retry in 30.0 sec...
08:02:23 ERROR: Fatal error sending data: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/bzt/modules/blazemeter.py", line 141, in _impl
    method(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/bzt/modules/blazemeter.py", line 495, in __send_data
    self._session.send_kpi_data(serialized, do_check)
  File "/usr/local/lib/python2.7/dist-packages/bzt/bza.py", line 664, in send_kpi_data
    response = self._request(url, data, headers=hdr)
  File "/usr/local/lib/python2.7/dist-packages/bzt/bza.py", line 89, in _request
    response = self.http_request(method=log_method, url=url, data=data, headers=headers, timeout=self.timeout)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='data.blazemeter.com', port=443): Max retries exceeded with url: /submit.php?session_id=r-ext5b459c1d1380f&signature=ci1leHQ1YjQ1OWMxZDEzODBmOjE1MzE4OTM0MDU6YjllMmQ2MjI1ZjEyZWQ5ZDA1YTc2OWUyNjNiMDEzYmZjMjkzMjEzNTgwOTRiMmViMzY1YjliZWRmZTM5OTRkOTMyZGMxM2Q1MWMxNGIwMjE3Nzk2YWI5Y2ZiMjdiMjY2YjBlZWFhOGIwNzllN2U2OWE3OTMzNzM4YzFkMTA3YWM=&test_id=6180437&user_id=456489&pq=0&target=labels_bulk&update=1 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fa67fad7250>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))

08:02:23 WARNING: Will skip failed data and continue running
08:02:43 WARNING: Failed to send data, will retry in 30.0 sec...
08:03:15 INFO: Succeeded with retry
08:10:23 INFO: Current: 1 vu    0 succ    1 fail    127.300 avg rt    /    Cumulative: 127.260 avg rt, 100% failures

it seems the importent reporting data was skipped there: 08:02:23 WARNING: Will skip failed data and continue running

So another fact I've noticed is that the traffic comes from jmeter slave container is twice the traffic accepted to jmeter master container (i.e taurus container) and after the file is uploaded the first time has the order of 1.5 Gb. So it looks like all the upload data in http POST body is taken to samples metadata and 1) is firstly uploaded from slave to thge target 2) is secondly uploaded form the slave to the master as sampling metadata 3) master accepts it but doesn't try to resend all the data to blasemeter, so network output of taurus container is much smaller. I believe this is due to jmeter.save.saveservice.samplerData: "false" in my test.yml set up

modules:
  blazemeter:
    token: *validtockenwasthere*
    #send-interval: 60s
    #timeout: 60s
    #artifact-upload-size-limit: 5
    browser-open: none
  jmeter:
    properties:
       server.rmi.ssl.disable: "true"
       #mode: DiskStore
       jmeter.save.saveservice.samplerData: "false"

So I'm currently wondering two problems:

1) despite of taurus even doesn't try to send all post body's request data why the statistics on upload requests don't come to blazemeter reporting. how ca I enable the detailed debugging what happens with it?

2) can I disable the post body data from sampling metadata. i.e. I don't like jmeter slave to send all the post data to jmeter master, just response stats only. I even tried the -Jjmeter.save.saveservice.samplerData=false option while starting jmeter slave, it doesn't help.

[Ilya Evlampiev]

And yes, one more problem. I tried mode: DiskStore too, but then no statistic comes to taurus at al, so I don't see any requests done in tayrys console and in blazemeter report. 

[Ilya Evlampiev]

So the explanation of strange behavior for DiskStore mode is based on docker stats and docker ps. It can be seen that net i/o is much lesser that 1.5Gb, while block i/o (i.e write and reat to disk) is 1.5Gb and more. So t looks like no actual http request are sent to target at all, but http request data is just dumped as sample metadata to disk

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT    MEM %               NET I/O             BLOCK I/O           PIDS
08ad0ee997e9        ecstatic_hoover     1.13%               234.6MiB / 23.3GiB   0.98%               271kB / 8MB         0B / 81.9kB         21
083407e42564        gracious_beaver     70.76%              15.83GiB / 16GiB     98.96%              215kB / 16.6kB      6.87GB / 1.96GB     30

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
08ad0ee997e9        blazemeter/taurus   "sh -c 'bzt -l /tmp/…"   14 minutes ago      Up 14 minutes                                ecstatic_hoover
083407e42564        justb4/jmeter:4.0   "/entrypoint.sh -s -…"   16 minutes ago      Up 16 minutes       0.0.0.0:1099->1099/tcp   gracious_beaver

[Andrey Pokhilko]

Is your question about Taurus or about JMeter? We can't help much with JMeter distributed test specifics...

--

Andrey Pokhilko
Open Source Initiatives Leader

11.07.2018 16:12, Ilya Evlampiev пишет:

CAUTION: This email originated from outside of CA. Do not click links or open attachments unless you recognize the sender and know the content is safe.

--
You received this message because you are subscribed to the Google Groups "codename-taurus" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codename-taur...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/codename-taurus/74f0914b-28e8-4d1d-8e09-ec95339ea918%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ilya Evlampiev]

ok, it looks like distributed jmeter worked it's part normally (while overpayloading the metadata of samples with large post body data), so why the other part, taurus blazemeter reporting doesn't upload the stats correctly?

[Ilya Evlampiev]

And the other:

what actually happens on 08:02:23 WARNING: Will skip failed data and continue running

and how can I force to process all the data and load it to blazemeter report?

On Wednesday, July 11, 2018 at 4:15:08 PM UTC+3, Andrey Pokhilko wrote:

[Andrey Pokhilko]

"Temporary failure in name resolution" means your network on Taurus host is overloaded so much, that even DNS resolve operation does not work.

To send results to BlazeMeter you obviously need working network connection and working DNS resolve.

--

Andrey Pokhilko
Open Source Initiatives Leader

11.07.2018 16:56, Ilya Evlampiev пишет:

To view this discussion on the web visit https://groups.google.com/d/msgid/codename-taurus/c1e1c1b2-6e2f-477d-baaf-28faeee56773%40googlegroups.com.

[Ilya Evlampiev]

ok, that is understood. the network overload is due to jmeter samples with "overpayload" with request body sent to taurus, that's why the network is overloaded; however, is there a way to make this report sending to blazemeter delayed, so let taurus try to send a bit later when the network will be fine. which parameter can be tuned to force taurus to try with reconnecting and to send the data that was skipped? amm I right that according to log, taurus received the detailed data from jmeter, but fauled to send ti to blazemeter report itself and thus lost it from the report? how cat it be fixed via  manipulating the properties file?

[Andrey Pokhilko]

You can switch JMeter into "stripped" sample sending mode, if it's not default.

There is currently no way to postpone result sending, sorry.

--

Andrey Pokhilko
Open Source Initiatives Leader

11.07.2018 17:16, Ilya Evlampiev пишет:

To view this discussion on the web visit https://groups.google.com/d/msgid/codename-taurus/ecd42209-21bb-4cf2-9e17-dc80de95d103%40googlegroups.com.

[Ilya Evlampiev]

and what about using IP instead of url for blazemeter? would it help?

[Andrey Pokhilko]

Most likely no. Because if network is busy, nothing will work through.

--

Andrey Pokhilko
Open Source Initiatives Leader

11.07.2018 19:13, Ilya Evlampiev пишет:

To view this discussion on the web visit https://groups.google.com/d/msgid/codename-taurus/45d2e2ce-141e-42e5-805d-b867abf423f7%40googlegroups.com.

[Ilya Evlampiev]

Andrey, one more question: when we configure 

execution:
- write-xml-jtl: full
  scenario: simple_script
  

do we expect the write-xml-full setting being distributed to remote jmeters or not? how ca we set up distributed settings for remote jmeters in test.yml in taurus/

[Andrey Pokhilko]

Hi,

That setting is applied to local JMeter. I don't know if JMeter propagates that to remote hosts.

--

Andrey Pokhilko
Open Source Initiatives Leader

12.07.2018 13:22, Ilya Evlampiev пишет:

To view this discussion on the web visit https://groups.google.com/d/msgid/codename-taurus/f0359aeb-c6ed-4d76-8e9a-64eb7284a3bd%40googlegroups.com.