RF and Jenkins - anyone aggregating RF results from different jobs

[Lars Nordin]

I have a multi-configuration test workflow that currently produces an RF result for each test job of a single configuration. I want to create a job that uses rebot to merge all the results into a single report and hence a single RF trend report in Jenkins too.

 

Anyone currently doing this?

I know that I need to use the Jenkins join plug-in but I am trying to work out a way to save the output.xml from each. Use a parameterized job to pass in URLs of the previous results?

[Lars Nordin]

To answer my own question, and hopefully help anyone else trying to do the same thing: I was able to patch together a job flow to do this.

 

I used the matrix (or multi-configuration) job to run my tests of the different configurations and produced an RF report for each. In the matrix job, I used the “Trigger parameterized build” feature to start a downstream job even if the matrix job fails. The downstream job used wget to fetch the output.xml from each test configuration (in URL use lastBuild as build #)  and then rebot to build a summary report.

 

The join plug-in is not useful since the downstream “joining” job is only triggered if all tests (of each configuration) pass.

[Lars Nordin]

Updating my last answer with a more Jenkins style data hand off between the test and reporting jobs.

 

Instead of having the downstream job use wget to fetch the output.xml from the upstream jobs, do this:

- configure the upsteam multi-cfg job: check “Archive the artifacts” and save output.xml (each axis/job creates uniquely named output.xml)

- configure downstream reporting job:   add build step, choose “Copy artifacts from another project”, and specify the name of the upstream multi-config job and “Upstream build that triggered this job”. For artifacts to copy, list a file glob for your different output.xml files. The different output.xml files will reside in directories by multi-configuration job name.

 

Sent: Wednesday, November 09, 2011 3:18 PM

To answer my own question, and hopefully help anyone else trying to do the same thing: I was able to patch together a job flow to do this.

 

I used the matrix (or multi-configuration) job to run my tests of the different configurations and produced an RF report for each. In the matrix job, I used the “Trigger parameterized build” feature to start a downstream job even if the matrix job fails. The downstream job used wget to fetch the output.xml from each test configuration (in URL use lastBuild as build #)  and then rebot to build a summary report.

 

The join plug-in is not useful since the downstream “joining” job is only triggered if all tests (of each configuration) pass.

Sent: Tuesday, October 11, 2011 2:06 PM

[Lakshmi narayana]

I have this requirement.

Jenkins Scenario: 10 Jobs with each job having 5 Tests each (few jobs has TestNg, and has Extent test results and few jobs has robot framework test results)

Prepare dash board with all 10 required jobs 

• Pass Rate (PR)
• In 1 week if each job will be triggered once daily, each one will have run 25 Tests (5 Tests/job x 5  days)
• So, 10 Jobs x 25 Tests/job = 250 Tests triggered in 1 week
• Pass Rate (PR) = Total # of Passed/Total  # of Tests
• example: 235/250 (94%)
• Note that, we are reporting Tests not Builds!
• Average Run Time (ART)
• This refers to the average time required to run the full test suite.
• RTx = Average run time of each job [where x: 1,2,3,..10] computed as run time on each day divided by number of days [ (T1+T2+T3+T4+T5)/5 ]
• ART = RT1 + RT2 + RT3 + ... + RT10
• Total number of Tests triggered
• Total number of unique tests run. In this example, this will be 10 Jobs x 5 Tests/Job so 50 unique tests.
• Trend
• Pass Rate
• To monitor the state of our test scripts on a weekly basis. Mainly for stability or early detection of issues.
• Average Run Time
• To monitor if we can support the under 12 hours regression
• Test Suite Size (Total number of Tests triggered)
• To monitor if we are adding more tests to the automation suite

 

Something like Build stats plugin, Global stat plugin - but not intuitive - to get JobLevel test case pass information to be viewed with.

 

Could you please help me here?

[Jayson Sparrow]

You can create a python script that walks into folders and parses all output.xml or report.html

We had similar issues and I ended up creating a script that gets only necessary information such as tc name, status, error details and tags from multiple test folder results

[sandeep s]

We had a similar problem , we wrote the listener interface for all of it , RF , testng and started drumping the data a database.

Now since its a interface and a run time one , there are so many advantages we can pull data set for...

Its just a matter of an UI to represent the data once you have the data.... Its easy , robust and long time statistical advantage...

./Sandeep

On Wed, Jan 4, 2017 at 11:23 AM, Jayson Sparrow <ramo...@gmail.com> wrote:

You can create a python script that walks into folders and parses all output.xml or report.html

We had similar issues and I ended up creating a script that gets only necessary information such as tc name, status, error details and tags from multiple test folder results

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-users+unsub...@googlegroups.com.
To post to this group, send email to robotframework-users@googlegroups.com.
Visit this group at https://groups.google.com/group/robotframework-users.
For more options, visit https://groups.google.com/d/optout.