Taurus runs out of memory

[klaus.s...@gmail.com]

Hi. 

I'm facing the problem that taurus runs out of memory. The problem occurs after about 15mins. The taskmanager shows that taurus consumes more and more memory. Running the test standalone with jmeter works fine. n the bzt.log I found following entry.

[2019-09-19 08:28:15,678 ERROR root] MemoryError: Unable to allocate array with shape (6144,) and data type int64

  File "C:\Program Files\Taurus\pkgs\bzt\cli.py", line 259, in perform

    self.engine.run()

  File "C:\Program Files\Taurus\pkgs\bzt\engine\engine.py", line 248, in run

    reraise(exc_info, exc_value)

  File "C:\Program Files\Taurus\pkgs\bzt\six\py3.py", line 86, in reraise

    raise exc

  File "C:\Program Files\Taurus\pkgs\bzt\engine\engine.py", line 227, in run

    self._wait()

  File "C:\Program Files\Taurus\pkgs\bzt\engine\engine.py", line 269, in _wait

    while not self._check_modules_list():

  File "C:\Program Files\Taurus\pkgs\bzt\engine\engine.py", line 256, in _check_modules_list

    finished = bool(module.check())

  File "C:\Program Files\Taurus\pkgs\bzt\modules\aggregator.py", line 840, in check

    for point in self.datapoints():

  File "C:\Program Files\Taurus\pkgs\bzt\modules\aggregator.py", line 598, in datapoints

    datapoint[DataPoint.CUMULATIVE] = copy.deepcopy(self.cumulative)  # FIXME: this line eats RAM like hell!

  File "copy.py", line 150, in deepcopy

  File "copy.py", line 240, in _deepcopy_dict

  File "copy.py", line 161, in deepcopy

  File "C:\Program Files\Taurus\pkgs\bzt\modules\aggregator.py", line 217, in __deepcopy__

    mycopy = KPISet(self.perc_levels, self[KPISet.RESP_TIMES].high)

  File "C:\Program Files\Taurus\pkgs\bzt\modules\aggregator.py", line 212, in __init__

    self[KPISet.RESP_TIMES] = RespTimesCounter(1, hist_max_rt, 3, perc_levels)

  File "C:\Program Files\Taurus\pkgs\bzt\modules\aggregator.py", line 103, in __init__

    self.histogram = HdrHistogram(low, high, sign_figures)

  File "C:\Program Files\Taurus\pkgs\hdrpy\__init__.py", line 269, in __init__

    self.counts = numpy.zeros(self.counts_len, dtype=numpy.int64)

I'm running bzt-1.13.8.

Does anyone have an idea?

br

Klaus

[grey....@gmail.com]

sorry, Klaus, I haven't got anything from you, could you share required files again?

---

Taras

[klaus.s...@gmail.com]

Hi Taras.

Did you get the files?

br

Klaus

[grey....@gmail.com]

not yet. attach it to conference looks better idea.

---

Taras 

[klaus.s...@gmail.com]

Hi Taras.

Sorry for the delay. I'm not able to attach files to the group. Just missing the add a file link. :-( I shared a google drive folder with you. You can access the folder here. https://drive.google.com/drive/folders/1_aECwyJGNwzKVNeJprY7K7_mubVp2Wsj

I'm not quite sure if you can reproduce the problem. Jmeter uses a custom component which I can't share. This component is responsible for request delivery and response processing. Maybe you can mock this component. If you are facing problems, let me know.

br

Klaus

[grey....@gmail.com]

It's good idea to check your script without proprietary component (by yourself) and report us if problem is still here. 

More than that you should check pure jmeter without taurus at all to verify if it's a java problem (it's very possibly) or not.

---

Taras

[klaus.s...@gmail.com]

The second check has already been done. Without taurus (jmeter and component) everything runs fine. I'll try to reproduce it without our component.

[klaus.s...@gmail.com]

Hi. Unfortunately I'm not able to reproduce the problem without the proprietary component. I tried a lot and was able to reproduce memory consumtion, but the memory has been released from time to time. Back to the test with the original component I tried to find teh problem in taurus python scripts. 

a) I put some gc.collect() in the aggregator in the 'datapoints' and '_calculate_datapoints' -> my test runs now much longer (without only 10mins) but still consumes memory. Both jmeter instances are pretty stable at about 1.8GB in sum. Taurus Status instance increasing every referesh up to 5.8GB till crash.

15:53:17 ERROR: MemoryError:

  File "C:\Program Files\Taurus\pkgs\bzt\cli.py", line 259, in perform

    self.engine.run()

  File "C:\Program Files\Taurus\pkgs\bzt\engine\engine.py", line 248, in run

    reraise(exc_info, exc_value)

  File "C:\Program Files\Taurus\pkgs\bzt\six\py3.py", line 86, in reraise

    raise exc

  File "C:\Program Files\Taurus\pkgs\bzt\engine\engine.py", line 227, in run

    self._wait()

  File "C:\Program Files\Taurus\pkgs\bzt\engine\engine.py", line 269, in _wait

    while not self._check_modules_list():

  File "C:\Program Files\Taurus\pkgs\bzt\engine\engine.py", line 256, in _check_modules_list

    finished = bool(module.check())

  File "C:\Program Files\Taurus\pkgs\bzt\modules\aggregator.py", line 842, in check

    for point in self.datapoints():

  File "C:\Program Files\Taurus\pkgs\bzt\modules\aggregator.py", line 596, in datapoints

    for datapoint in self._calculate_datapoints(final_pass):

  File "C:\Program Files\Taurus\pkgs\bzt\modules\aggregator.py", line 884, in _calculate_datapoints

    self._process_underlings(final_pass)

  File "C:\Program Files\Taurus\pkgs\bzt\modules\aggregator.py", line 861, in _process_underlings

    for point in underling.datapoints(final_pass):

  File "C:\Program Files\Taurus\pkgs\bzt\modules\aggregator.py", line 596, in datapoints

    for datapoint in self._calculate_datapoints(final_pass):

  File "C:\Program Files\Taurus\pkgs\bzt\modules\jmeter.py", line 877, in _calculate_datapoints

    self.errors_reader.read_file()

  File "C:\Program Files\Taurus\pkgs\bzt\modules\jmeter.py", line 1194, in read_file

    read = self.file.get_bytes(size=1024 * 1024, decode=False)  # "Huge input lookup" error without capping :)

  File "C:\Program Files\Taurus\pkgs\bzt\utils.py", line 682, in get_bytes

    _bytes = self.fds.read(size)

b) I tried to skip the aggregation and commented some lines in '_calculate_datapoints'. taurus runs now in a kind of blind mode, cause I didn't see any transactions, hits, users. -> taurus consumes still memory. and stuck after an hour cause it can't lstart netstat.exe. The jmeter instances run till their given end (2 hours) and the test results are complete, so it is an advantage. An png is available in my google drive folder.

c) As mentioned earlier and due to the outcome of b) ->  the test runs perfect without taurus 

Is there a way to run taurus without the taurus status gui? Maybe just with the jmeter logging (jmeter in non gui mode) on the console. 

Do you have any idea where the problems root cause is? 

I understand that if you can't reproduce the problem it's very hard to you to fix this. I know that our component has something to deal with the problem, but I think that's just a problem how taurus deals with the jmeter instances. I promise you I'll give you all the support I can do, cause running those tests just by starting taurus and taking a look from time to time on the status gui is pretty cool and much smarter than the alternatives I have now.

br

Klaus 

[grey....@gmail.com]

Hi. 

I guess you mean our dashboard under 'taurus status gui'. It's console reporting and you can disable this by that.

---

Taras

[grey....@gmail.com]

Additionally.

As I see in log your case is connected with big load and some level of memory usage is essential. 

I'm not sure if we can (and should) fix that even if you provide us reproducible case. But if you get  such case please share it.

--- 

[klaus.s...@gmail.com]

Hi.

ok. thanks for the information. startet a test without console reporter. think that will work for us. I'll give you an update later when the test is finished.