Visualizing tests separately from different stages of a pipeline

[ST]

Hi!

I am migrating our java/maven build pipeline to a Jenkinsfile, so both the unit tests (maven-surefire-plugin) and integration tests (maven-failsafe-plugin) will be run inside the same pipeline, but in different stages.

How can I separate the test results for unit tests and integration tests in Jenkins? I am collecting test results like this:

for unit tests:

step([$class: 'JUnitResultArchiver', testResults: '**/target/surefire-reports/*.xml'])

for integration tests:

step([$class: 'JUnitResultArchiver', testResults: '**/target/failsafe-reports/*.xml'])

And they end up as a mixed list under "Test Result" for every build.

Since they were separate jobs before, they were separated in Jenkins as well. Would be nice to have them as separated list also in the Jenkins 2.x pipeline, but how?

Best regards,

 stefan.

[James Dumay]

Hi Stefan,

There is no way to do this with Pipeline today. However, when designing the new test reporting screen in Blue Ocean I did have someone ask if we could group them by stage. 

The problem I saw here is that most peoples pipelines look like this:

Build-->Test-->Integration Test-->Deploy

If any tests fail in "Test" then you would not see any tests displayed for "Integration Tests" as they were not executed - rendering the grouping reasonably useless except for viewing successful tests. (Viewing successful tests is an edge case we've opted not to handle for the moment due to UI scaling concerns).

Whats your use case for grouping tests by stage?

[ST]

James,

Interesting statement "successful tests is an edge case" - I've never thought about it this way, but if all tests pass I dont need to know anything else, so true I guess :-)

Our use case is (with daily releases):

* unit tests

* itests that set up their own data (most of our itests)

* itests relying on specific test data (i.e. they have to be run against a specific test database)

* itests that test typical API consumption (to ensure we haven't broken existing API)

So when:

(A) a test fails I'd like to know into which category it falls to better assess any consequences and being able to make a first guess about what the root cause could be

(B) many tests fail it's useful to know whether all of them fall into same test category.

Putting tests into a specific java package (*.itest.api.* etc.) would probably help to solve (A) but not (B).

Best regards,

 stefan.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/c48d03ee-fe43-4538-a875-46406f65f601%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Arnaud bourree]

Hi

I'm interested in test grouped by stage: we'd like to run the same set of test in different context: tests have same name and each can work in one context and not in other. Grouped by stage we can see easily which context has issue

Regards

Arnaud Bourrée

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CABwQARv-Fnh7j4YucDnzWc-uDC16wekW29E4E5FD9o6kPSsGtQ%40mail.gmail.com.

[James Dumay]

Are you describing a situation here that's closer to what the Matrix job type caters to? Eg Operating System X Browser Version or something?

If you don't mind drawing a picture, that could help me get across what you're use case entails :)

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/CgTuhpm5sZY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAESUf_BH5Yvpx-VoMWeHe%3DpT%2BiorKfqucjZty%2BOeYdiOhgtvmg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

James Dumay

Product Manager

CloudBees, Inc.

Skype: jdumaycb Twitter: i386

[Arnaud bourree]

Yes, like Matrix job but not only. Separate tests by stage between unit tests and integration tests looks important for me.

Arnaud.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAErbOvSxgVrj_DqNCVxyBoHJV-WVszkM8DmeDbjUv%3DXfW9b7DQ%40mail.gmail.com.

[Joshua Harlow]

I to would like something like this,

It seems to be a common thing to do and a real weakness of the pipeline style,

If people are supposed to setup complex pipelines in a pipeline file then its pretty important that there be a way to ship out multiple result files,

Even if they are successful, discarding information, especially in jenkins seems bad.

Because in my view the 'Build-->Test-->Integration Test-->Deploy' is a real simple example and I personally plan on doing a much bigger pipeline like,

Build project Y--> Unit Test Y--> Local functional test Y
                                                                             |
                                                                             ---> Integration Test Y + Z -> Perf test Y + Z -> Request deploy to stage ---> (wait) -> Integration Test -> Perf test --> Deploy to prod ---> (wait) -->  ....
                                                                             |
Build project Z-->Unit Test Z--> Local functional test Z

And in that there will be more than Y and Z (all done in parallel with a join), at each stage having test results,

Now maybe pipelines aren't meant to approach this complexity? Though if they are not, then I'm not quite sure what they are really useful for (because I can do the same thing in ansible/equivalent),

-Josh

[James Dumay]

Joshua, I don't think there is anything stopping you from recording the test results in the Pipeline you have described. Jenkins will capture them wherever you ask it to with the JUnitArchiver however it won't record what stage it captured them from (there is an outstanding Improvement in JIRA for this).

I hope I didn't miss anything

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/8b2d8171-2f75-44d2-908d-f701fa131744%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Joshua Harlow]

Do I need to just give each capture a unique filename?

I was trying this earlier and it appears giving them the same name only caused the first to show up on the UI :(

Perhaps though I was doing something wrong...

[Joshua Harlow]

It also gets interesting that if I have 2 runs (pre-patch unit testing and post-patch unittest) but both have tests with the same name in them,

Unsure exactly how that will render :-P

I'll see what I can find out, other option I guess is to just publish a bunch of html (and not use the results renderer).

-Josh

[James Dumay]

I believe what is happening is that when you achieve for the second time with tests of the same name, they replace the other tests. This is described in https://issues.jenkins-ci.org/browse/JENKINS-27395 

One workaround is if you modify the .xml files so that the test names are different. There's a Pipeline script function in the linked ticket that can help with that https://issues.jenkins-ci.org/browse/JENKINS-27395?focusedCommentId=256459&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-256459 

This is not ideal though - we should really fix this.