How to merge old/new statuses using rebot --merge and keep the only one?

[Alex Farinhate]

I'm using rebot tool to merge outputs after re-execution of failed tests.

robot --output original.xml /path/to/dir/.
robot --rerunfailed original.xml --output rerun.xml /path/to/dir/.
rebot -o machine1.xml -l machine1.html --merge original.xml rerun.xml

The same action is provided on few test machines. Test suite is identical and is executed against each VM. All VMs are considered as identical, however, they are not stable and I receive different results on each machine. I want to merge all results from all machines and retrieve the maximum number of Passed tests, to understand whether the test is REALLY failing or it's just unstable environment and test is OK by itself.

Other words, if test passed at least on 1 machine, but failed on other 3 machines, it needs to be considered as Passed in the final report.

However I receive False for such case.

Is it possible to change the behavior somehow?

Example from the final report:

Status: FAIL (critical)
Message:    Re-executed test has been merged.
New status: FAIL
New message: Re-executed test has been merged.
New status: FAIL
New message: IndexError: Given index 0 is out of the range 0--1.
Old status: FAIL
Old message: IndexError: Given index 0 is out of the range 0--1.
Old status: PASS
Old message: Re-executed test has been merged.
New status: PASS
New message: 
Old status: PASS
Old message: Re-executed test has been merged.
New status: PASS
New message: 
Old status: PASS
Old message: 

[Pekka Klärck]

Hi,

When merging tests, the last test always has the precedence and
there's no built-in way to alter that. You could try creating a
pre-rebot modifier that changes test status to PASS if it's message
starets with `Re-executed test has been merged.` and contains any line
with `Old status: PASS`. Alternatively you could create a custom tool
to merge results. That's more work but gives you more flexibility as
well.

Cheers,
.peke

> --
> You received this message because you are subscribed to the Google Groups "robotframework-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/robotframework-users/f5082577-ba39-4b22-bf1d-cb42a7fe5c80%40googlegroups.com.



--
Agile Tester/Developer/Consultant :: http://eliga.fi
Lead Developer of Robot Framework :: http://robotframework.org

[Alex Farinhate]

Thank you! Your suggestion helped me to achieve the result I was looking for.

Maybe it will be helpful for somebody else.

So, I created TestStatusChecker.py which helps to merge reports with the following logic: test status will be failed ONLY if it was failing in ALL reports.

from robot.api import SuiteVisitor

class TestStatusChecker(SuiteVisitor):

    def __init__(self, *args):

        pass

    def visit_test(self, test):

        if 'PASS' in test.message and 'Re-executed test has been merged' in test.message:

            test.status = 'PASS'

            test.message = 'Test passed because it passed at least once.'

CLI command for merging the results:

rebot -l final_log.html --prerebotmodifier TestStatusChecker.py --merge 1.xml 2.xml 3.xml 4.xml