Splitting/Merging/Clustering JUnit test results?

[Simon Richter]

Hi,

I have a matrix job that in each leaf runs the test suite twice, once
normally and once with memory checking enabled, each of which generates
an XML file that can be parsed with the JUnit parser.

Now I'd like to present the results in the overview in a useful way:

1. separate normal/memcheck runs from the same build
2. merge results from different cells of the matrix
3. if tests fail only in some configurations, try to cluster by label

Problems:

1. I have the JUnit plugin parse both generated XML files, but only the
normal test run (<Site><Testing>...</></>)shows up in the result
summary, not the memory checker (<Site><DynamicAnalysis>...</></>).

2. Test results are only available inside the matrix configurations, no
summary is available

3. I haven't looked at yet, since I need result merging first.

Simon

[jeremy mordkoff]

I have a similar issue, except my test results some from multiple jobs

my basic build pipeline runs unit tests then invokes the systemtest pipeline. These run 10 to 30 times a day.

I also have a nightly pipeline that runs a lot of additional tests, like coverage, interop, stress, load, upgrade. It takes many hours so it only runs once a day against whatever basic build is most recent.

I would like to be able to aggregate the test results of the three pipelines together into some master report. 

I was thinking of creating an external web service that would accept (pointers to) junit xml files and aggregate the results for me. It would also generate links back into jenkins in case I needed to do some investigating.

Back to your problem...if the xml path inside your tests is the same regardless of whether the test run includes memchecker, then one set of results might be overwriting the other. Just postprocess the junit files and tweak the paths. I do this now to remove any host or workspace specifics from them so I can compare one build to the next.

[jeremy mordkoff]

Hi, 

I am heading down this path.  My plan is build a web app that can accept test results in various formats, and can aggregate them by build and identify them by type. 

It will track sources and link back to them

It will have the ability to normalize the xml paths, e.g. /net/server1/workspace and /net/server2/workspace could both be normalized to /net/server/workspace

It will have the ability to augment the paths, e.g. building on the example above, it could convert the paths to /net/server/functional/workspace and /net/server/coverage/workspace

It will compare builds and identify tests that are flaky and tests that are failing continually

It will link to JIRA 

It will link to testlink

It will identify builds that have only been through CI testing from those that have also had additional testing done

It will identify builds that have been through QA or have been released from internal builds. 

I have a framework that we have been using for years that does track builds, but it does not track test results yet. I do plan to opensource this, if anyone is interested in helping or has additional ideas, please ping me directly jeremy dot mordkoff at riftio dot com 

On Friday, February 14, 2020 at 8:29:52 AM UTC-5, Simon Richter wrote: