Compare boolean & numerical test results over builds in a table

[Daniel Mercier]

Dear Jenkins developer community,

 

I am looking for a plugin to compare test results between builds.

 

My test results are not limited to a Boolean (Success/Failure) but may include numerical values (beyond just execution time).

 

Ideally, I am looking for a Jenkins plugin to archive for each build, a file in a standard data format (CSV or XML or json or upgraded Junit report) that holds information about group name/name/value and display a table to visually compare the values.

 

I found a few plugin with parts of the answer

• test analyzer plugin
  
• Plot plugin
  
• Measurement plots plugin
  

 

And a few performance plugins linked to commercial products that seems to combine some of the features.

• Performance signature plugin
  

 

I am considering either contributing to an existing if one is close enough and the owner is willing or building a new one.

 

Let me know,

 

Thank you,

 

Daniel Mercier 

[JordanGS]

Off the top of my head, i wouldn't even bother writing a new plugin or updating one. Write a script in any language to collect all the information from each individual build/test and then aggregate that into a JUnit XML file.

Take a look at Tap plugin, it's similar to JUnit but simpler. and the artifact diff plugin. Not sure if they will help you but i remember seeing these two when i was searching a few months ago.

[Daniel Mercier]

Thank you for the quick answer. 

'Measurement plots plugin' is doing a similar thing by requesting the data to be placed in the test standard output of the JUnit file, then process it inside Jenkins.The data is not condensed in a table but the graph displays are pretty good with build line for each result.

'Artifact diff plugin' has a good concept to compare artifacts for all builds but the output is a diff or a delta. PS: I did not get it to work. I am missing the postbuild 'Publish ACI reports' mentioned in the help. 

I am still undecided. We need the condensed information for our QA team.

Daniel Mercier