OutOfMemoryError when processing large robot xml files

[Mg]

Hi,

I currently have two sets of tests whose outputs I'd like to merge: generatedTests.xml, and output.xml. generatedTests.xml is very large (550.248 mb), because it's programmatically generated from a big pool of test data through python. Output.xml is hardcoded and small: I'd like to merge them after test execution.

I understand that I can delete the xml files after results are generated, but I'd like to merge them instead of having two separate logs. To merge them, I tried: 

java -cp %THECLASSPATH% org.robotframework.RobotFramework rebot --outputdir ../test-results/ ../test-results/generatedTests.xml ../test-results/output.xml

but I get: 

Exception in thread "MainThread" java.lang.OutOfMemoryError: Java heap space.

I had to allocate 6g of memory to coax rebot --flattenkeywords to start, and even then it crashed quietly after a half hour. Is this just a consequence of running an abnormally high volume of tests? Is there anything that can be done to reduce ouput xml size? Could it be because I'm running through the standalone jar?

Cheers,

Marco

[Pekka Klärck]

Hi,

You could try post-processing results with Rebot on Python, not on Jython. Python in general is fastee and more memory efficienent than Jython, and when processing results you can use Python even if needed to run tests on Jython.

Another easy solution is getting more memory. Using --flattenkeywords like you've already tried may help as well.

If your output.xml size grows really large, even the above tricks may not be enough. In this case it's beat to try making the file smaller. You should especially avoid for loops, Wait Untill Keyword Succeeds, and deeply nested keyword structures by moving such logic into libraries when possible.

Sent from my mobile.

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.
To post to this group, send email to robotframe...@googlegroups.com.
Visit this group at https://groups.google.com/group/robotframework-users.
For more options, visit https://groups.google.com/d/optout.

[adam.wyp...@gmail.com]

You can try as well: --splitlog flag 

http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#splitting-logs

W dniu wtorek, 17 kwietnia 2018 22:22:03 UTC+1 użytkownik Pekka Klärck napisał:

Hi,

You could try post-processing results with Rebot on Python, not on Jython. Python in general is fastee and more memory efficienent than Jython, and when processing results you can use Python even if needed to run tests on Jython.

Another easy solution is getting more memory. Using --flattenkeywords like you've already tried may help as well.

If your output.xml size grows really large, even the above tricks may not be enough. In this case it's beat to try making the file smaller. You should especially avoid for loops, Wait Untill Keyword Succeeds, and deeply nested keyword structures by moving such logic into libraries when possible.

Sent from my mobile.

17.4.2018 21.46 "Mg" <marcogr...@gmail.com> kirjoitti:

Hi,

I currently have two sets of tests whose outputs I'd like to merge: generatedTests.xml, and output.xml. generatedTests.xml is very large (550.248 mb), because it's programmatically generated from a big pool of test data through python. Output.xml is hardcoded and small: I'd like to merge them after test execution.

I understand that I can delete the xml files after results are generated, but I'd like to merge them instead of having two separate logs. To merge them, I tried: 

java -cp %THECLASSPATH% org.robotframework.RobotFramework rebot --outputdir ../test-results/ ../test-results/generatedTests.xml ../test-results/output.xml 

but I get: 

Exception in thread "MainThread" java.lang.OutOfMemoryError: Java heap space.

I had to allocate 6g of memory to coax rebot --flattenkeywords to start, and even then it crashed quietly after a half hour. Is this just a consequence of running an abnormally high volume of tests? Is there anything that can be done to reduce ouput xml size? Could it be because I'm running through the standalone jar?

Cheers,

Marco

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-users+unsub...@googlegroups.com.

[Pekka Klärck]

2018-04-18 8:46 GMT+03:00 <adam.wyp...@gmail.com>:
> You can try as well: --splitlog flag
>
> http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#splitting-logs

--splitlog helps if log.html size starts to got too big for browsers
to handle. Very useful, for example, when running larger test suites
on CI. It doesn't help if output.xml size is so large it cannot be
processed, though.

Cheers,
.peke
--
Agile Tester/Developer/Consultant :: http://eliga.fi
Lead Developer of Robot Framework :: http://robotframework.org

[adam.wyp...@gmail.com]

Thanks for an update, Pekka. If command provided by Marco is performing only merge of few XMLs two one single XML than it doesn't make sens, but if is making report.html and log.html I think is worth to try with --splitlog. As far as I checked robotframework code, it doesn't keep already built and stored already files inside memory (only links are used).  

Anyway I don't see in the command line provided by Marco any kind of JVM memory settings:

java -cp %THECLASSPATH% org.robotframework.RobotFramework rebot --outputdir ../test-results/ ../test-results/generatedTests.xml ../test-results/output.xml

I think it should be something like this in case Java 7 and higher releases:

java -server -Xms1g -Xmx6g -XX:+UseG1GC -cp %THECLASSPATH% org.robotframework.RobotFramework rebot --outputdir ../test-results/ ../test-results/generatedTests.xml ../test-results/output.xml

Best Regards,

-Adam.

[Mg]

 

I think it should be something like this in case Java 7 and higher releases:

java -server -Xms1g -Xmx6g -XX:+UseG1GC -cp %THECLASSPATH% org.robotframework.RobotFramework rebot --outputdir ../test-results/ ../test-results/generatedTests.xml ../test-results/output.xml

 

When I was adding more memory, my commands were:

java -Xms1g -Xmx8g -cp ---> (the remainder of the commands)

I just gave your suggestion a shot. It tried its hardest, but still ran out of memory. It's specifically failing on one arraycopy operation; when I ran rebot on command line it defaulted to the version installed through jython, and it dropped a handy stacktrace.

Traceback (most recent call last):

  File "C:\jython2.7.0\Lib\runpy.py", line 161, in _run_module_as_main

    return _run_code(code, main_globals, None,

  File "C:\jython2.7.0\Lib\runpy.py", line 72, in _run_code

    exec code in run_globals

  File "C:\jython2.7.0\Lib\site-packages\robot\rebot.py", line 411, in <module>

    rebot_cli(sys.argv[1:])

  File "C:\jython2.7.0\Lib\site-packages\robot\rebot.py", line 378, in rebot_cli

    return Rebot().execute_cli(arguments, exit=exit)

  File "C:\jython2.7.0\Lib\site-packages\robot\utils\application.py", line 46, in execute_cli

    rc = self._execute(arguments, options)

  File "C:\jython2.7.0\Lib\site-packages\robot\utils\application.py", line 90, in _execute

    error, details = get_error_details(exclude_robot_traces=False)

  File "C:\jython2.7.0\Lib\site-packages\robot\utils\error.py", line 47, in get_error_details

    details = ErrorDetails(exclude_robot_traces=exclude_robot_traces)

  File "C:\jython2.7.0\Lib\site-packages\robot\utils\error.py", line 60, in ErrorDetails

    raise exc_value

        at java.util.Arrays.copyOf(Arrays.java:3332)

        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)

        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)

        at java.lang.StringBuilder.append(StringBuilder.java:136)

        at java.lang.StringBuilder.append(StringBuilder.java:76)

        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

java.lang.OutOfMemoryError: java.lang.OutOfMemoryError: Java heap space

I'm not sure if it's a jython quirk that's loading the whole file into a buffer or something, but when I ran 

C:\Python27\Lib\site-packages\robot\rebot.py --outputdir -----> (the remainder of the commands)

It dropped another very similar memory error. 

Traceback (most recent call last):
  File "C:\Python27\Lib\site-packages\robot\rebot.py", line 411, in <module>
    rebot_cli(sys.argv[1:])
  File "C:\Python27\Lib\site-packages\robot\rebot.py", line 378, in rebot_cli
    return Rebot().execute_cli(arguments, exit=exit)
  File "C:\Python27\Lib\site-packages\robot\utils\application.py", line 46, in execute_cli
    rc = self._execute(arguments, options)
  File "C:\Python27\Lib\site-packages\robot\utils\application.py", line 90, in _execute
    error, details = get_error_details(exclude_robot_traces=False)
  File "C:\Python27\Lib\site-packages\robot\utils\error.py", line 47, in get_error_details
    details = ErrorDetails(exclude_robot_traces=exclude_robot_traces)
  File "C:\Python27\Lib\site-packages\robot\utils\error.py", line 60, in ErrorDetails
    raise exc_value
MemoryError

I'm going to try to slim down the output.xml file by trimming/cleaning up some keywords, because I have a lot of nested keyword calls that idle in for loops while waiting for responses. It takes a long time to go through all the data (~6h on the weaker build machine), so I'll run it during the night and post again tomorrow. 

[Jayson Panganiban]

I've encountered this before. Since I have large test results from different log path and wanted to export it to an excel file, I created my own solution.

You may look at my script and try it out.

https://github.com/jsonpanganiban/robotframework-result-extractor

--

You received this message because you are subscribed to the Google Groups "robotframework-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.