Plugin Random CSV Data Set (0.7) (on 5.4.1) and StackOverflowError

[Mariusz Pęciak]

Hi .*,

I've come across StackOverflowError for jmeter-plugins-random-csv-data-set-0.7.jar on jmeter 5.4.1 with the following bzt config

data-sources: # list of external data sources
- path: test/resources/categories.csv # this is a full form
delimiter: ';'
quoted: false
encoding: "utf-8"
loop: true
variable-names: category-id
random-order: true
sharing-mode: "Current thread group"

2021-08-03 13:10:29,477 ERROR o.a.j.JMeter: Uncaught exception in thread Thread[10.18.9.4- 1-279,5,RMI Runtime]
java.lang.StackOverflowError: null
    at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:46) ~[?:1.8.0_302]
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_302]
    at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:1.8.0_302]
    at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:159) ~[?:1.8.0_302]
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) ~[?:1.8.0_302]
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[?:1.8.0_302]
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[?:1.8.0_302]
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) ~[?:1.8.0_302]
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) ~[?:1.8.0_302]
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) ~[?:1.8.0_302]
    at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[?:1.8.0_302]
    at java.io.BufferedReader.fill(BufferedReader.java:161) ~[?:1.8.0_302]
    at java.io.BufferedReader.read(BufferedReader.java:182) ~[?:1.8.0_302]
    at com.blazemeter.csv.RandomBufferedReader.read(RandomBufferedReader.java:35) ~[jmeter-plugins-random-csv-data-set-0.7.jar:?]
    at org.apache.jmeter.save.CSVSaveService.csvReadFile(CSVSaveService.java:998) ~[ApacheJMeter_core.jar:5.4.1]
    at com.blazemeter.csv.RandomCSVReader.readLineWithSeek(RandomCSVReader.java:158) ~[jmeter-plugins-random-csv-data-set-0.7.jar:?]
    at com.blazemeter.jmeter.RandomCSVDataSetConfig.readRandom(RandomCSVDataSetConfig.java:112) ~[jmeter-plugins-random-csv-data-set-0.7.jar:?]
    at com.blazemeter.jmeter.RandomCSVDataSetConfig.iterationStart(RandomCSVDataSetConfig.java:68) ~[jmeter-plugins-random-csv-data-set-0.7.jar:?]
    at org.apache.jmeter.control.GenericController.fireIterationStart(GenericController.java:399) ~[ApacheJMeter_core.jar:5.4.1]
    at org.apache.jmeter.control.GenericController.fireIterEvents(GenericController.java:391) ~[ApacheJMeter_core.jar:5.4.1]
    at org.apache.jmeter.control.GenericController.next(GenericController.java:160) ~[ApacheJMeter_core.jar:5.4.1]
    at com.blazemeter.jmeter.control.VirtualUserController.next(VirtualUserController.java:41) ~[jmeter-plugins-casutg-2.9.jar:?]
    at com.blazemeter.jmeter.control.VirtualUserController.nextIsNull(VirtualUserController.java:86) ~[jmeter-plugins-casutg-2.9.jar:?]
    at org.apache.jmeter.control.GenericController.next(GenericController.java:170) ~[ApacheJMeter_core.jar:5.4.1]
    at com.blazemeter.jmeter.control.VirtualUserController.next(VirtualUserController.java:41) ~[jmeter-plugins-casutg-2.9.jar:?]
    at org.apache.jmeter.control.GenericController.nextIsAController(GenericController.java:225) ~[ApacheJMeter_core.jar:5.4.1]

I don't think I got any problems with CSV itself, just the looping is a problem here (and it's just 7 rows).
I have tried to downgrade the version to 0.6 but then got bumped into a class loader exception... (I presume it doesn't go well if JMeter doesn't match the plugin but it may need fixing anyway).

What I was hoping to do is just looping through CSV for all virtual users (VU) all the time until time passed. Initially, when I was using random-order: false I was quickly getting EOF instead of retrieving any of the CSV rows (that looks like another problem with for core jmeter plugin implementation too - unless doing something wrong).

Anyway, the load is generated in a way I wanted however JMeter workers are just flooding with StackOverflowError.
Please find the full bzt config.

Thank for help!
Mariusz

[DT]

Although I cannot reproduce your issue I can suggest a couple of workarounds:

 1. Consider using HTTP Simple Table Server, its READ endpoint has RANDOM mode, seems to be something you can use, moreover it's suitable for Distributed Testing

 2. If your file is not too big (7 lines is a very small file) you can read the random line from it using the following __groovy( ) function

    ${__groovy(def file = new File(' test/resources/categories.csv '); def currentCategoryId = (file.readLines().get(org.apache.commons.lang3.RandomUtils.nextInt(0\, file.readLines().size()))); vars.put('currentCategoryId'\, currentCategoryId); return currentCategoryId,)}

Example Taurus YAML:

scenarios:
  search-categories:
    default-address: ${host}
    variables:
      currentPage: 0
      totalPages: 1
    requests:
      - while: ${__jexl3(${currentPage}<${totalPages})}
        do:
          - url: "${base_path}/categories/${__groovy(def file = new File('test.csv'); def currentCategoryId = (file.readLines().get(org.apache.commons.lang3.RandomUtils.nextInt(0\\, file.readLines().size()))); vars.put('currentCategoryId'\\, currentCategoryId); return currentCategoryId,)}"
            label: "pulling a category: ${currentCategoryId}"
            body:
              currentPage: "${currentPage}"
              pageSize: 10
              fields: FULL
            method: GET
            debug: true
            extract-jsonpath:
              internalCurrentPage:
                jsonpath: $.pagination.currentPage
                from-variable: internalCurrentPage
                default: 10000
              internalTotalPages:
                jsonpath: $.pagination.totalPages
                from-variable: internalTotalPages
                default: 1
          - set-variables:
              currentPage: ${__intSum(1,${internalCurrentPage})}
              totalPages: ${internalTotalPages}

[Mariusz Pęciak]

Hi DT,

Thank you for the replay, the first option really is appealing. For the second although I know it would be possible I hate long inlines like that.

Btw do you know where I could report this issue to BlazeMeter/JMeter plugins?

Cheers,
Mariusz

[Mariusz Pęciak]

I've just have changed the name of the scenario (search-categories -> browse-categories)  and StackOverFlowError has mutated into the one below.

I think it's not CSV reading which is the problem but iteration over to the next virtual user (VU).

Each VU goes through paging till the end (sometimes that is 20-30 pages for a single user to go through)

Running it as follow
bzt execute.yml -o modules.jmeter.path=/opt/apache-jmeter-5.4.1/bin/jmeter -o settings.artifacts-dir=artifacts -o settings.env.host=https://something.co.uk/ -o execution.0.scenario=browse-categories -o execution.0.write-xml-jtl=full -o execution.0.concurrency=50 -o execution.0.ramp-up=300s -o execution.0.hold-for=60m -o execution.0.throughput= -o execution.0.iterations= -o execution.0.distributed.0=10..... -o execution.0.distributed.1=10....

2021-08-13 12:13:19,336 ERROR o.a.j.JMeter: Uncaught exception in thread Thread[10.18.5.31-browse-categories-ThreadStarter 1-158,5,RMI Runtime]
java.lang.StackOverflowError: null
    at java.lang.String.valueOf(String.java:2994) ~[?:1.8.0_302]
    at java.lang.StringBuilder.append(StringBuilder.java:131) ~[?:1.8.0_302]
    at org.apache.jmeter.engine.util.CompoundVariable.execute(CompoundVariable.java:146) ~[ApacheJMeter_core.jar:5.4.1]
    at org.apache.jmeter.engine.util.CompoundVariable.execute(CompoundVariable.java:113) ~[ApacheJMeter_core.jar:5.4.1]
    at org.apache.jmeter.functions.Jexl3Function.execute(Jexl3Function.java:72) ~[ApacheJMeter_functions.jar:5.4.1]
    at org.apache.jmeter.engine.util.CompoundVariable.execute(CompoundVariable.java:138) ~[ApacheJMeter_core.jar:5.4.1]
    at org.apache.jmeter.engine.util.CompoundVariable.execute(CompoundVariable.java:113) ~[ApacheJMeter_core.jar:5.4.1]
    at org.apache.jmeter.testelement.property.FunctionProperty.getStringValue(FunctionProperty.java:100) ~[ApacheJMeter_core.jar:5.4.1]
    at org.apache.jmeter.control.WhileController.getCondition(WhileController.java:142) ~[ApacheJMeter_core.jar:5.4.1]
    at org.apache.jmeter.control.WhileController.endOfLoop(WhileController.java:62) ~[ApacheJMeter_core.jar:5.4.1]
    at org.apache.jmeter.control.WhileController.next(WhileController.java:112) ~[ApacheJMeter_core.jar:5.4.1]
    at org.apache.jmeter.control.GenericController.nextIsAController(GenericController.java:222) ~[ApacheJMeter_core.jar:5.4.1]
    at org.apache.jmeter.control.GenericController.next(GenericController.java:175) ~[ApacheJMeter_core.jar:5.4.1]

    at com.blazemeter.jmeter.control.VirtualUserController.next(VirtualUserController.java:41) ~[jmeter-plugins-casutg-2.9.jar:?]
    at com.blazemeter.jmeter.control.VirtualUserController.nextIsNull(VirtualUserController.java:86) ~[jmeter-plugins-casutg-2.9.jar:?]
    at org.apache.jmeter.control.GenericController.next(GenericController.java:170) ~[ApacheJMeter_core.jar:5.4.1]
    at com.blazemeter.jmeter.control.VirtualUserController.next(VirtualUserController.java:41) ~[jmeter-plugins-casutg-2.9.jar:?]
    at org.apache.jmeter.control.GenericController.nextIsAController(GenericController.java:225) ~[ApacheJMeter_core.jar:5.4.1]

    at org.apache.jmeter.control.GenericController.next(GenericController.java:175) ~[ApacheJMeter_core.jar:5.4.1]

Cheers,

Mariusz