How to aggregate the artifacts of a build matrix

28 views
Skip to first unread message

Alexey Savchkov

unread,
Jan 6, 2023, 8:24:02 AM1/6/23
to go-cd
I'm stuck with a problem which I can't think of a good solution for so would like to know your thoughts on the subject.
One stage of my pipeline is a test matrix consisting of 6 jobs. Each job produces an Allure test report. Now I need to collect these individual test reports and build an aggregate one from them. Normally I would do it in some kind of a post-build job running at the end of each stage or the entire pipeline. But as far as I see there are no such post-build jobs in GoCD. I also can't use a separate stage for this since it won't start if the test stage fails due to failed tests. The only possibility I see is to start a dedicated job (say, "Report") in the test stage together with the test jobs, make it continuously poll the states of the test jobs and when all of them are completed fetch their report artifacts and build the total report. Wasting of runners when they are just waiting for the test jobs to finish but still acceptable.
Am I not missing anything very basic as to how to go about such cases in GoCD? What would you propose? An external storage (Nexus) is available if this matters.

Thank you very much in advance.

Ashwanth Kumar

unread,
Jan 6, 2023, 6:23:25 PM1/6/23
to go...@googlegroups.com
Under ideal conditions, I would assume a test failure means the report would either be incomplete or not useful isn't it? 

AFAIK there isn't a post-build job for a stage, you can do it at a job level though using "Run If" Attribute of the tasks. Here are a few ways to get what you need, it might range from ugly to "Oh, that's interesting". If you do decide to choose these methods please drop a line so I know if it was useful :-)

Option 1
Assuming you can tell from the reports if a test has failed or not, you just have a dummy task after the tests that runs on runif="any" and do the aggregation on the next stage. If you know from the reports something has failed you can always fail the aggregate stage with some useful information. 
Cons: The failure will not immediately tell you what failed but again if the folks who need to view the error have access to GoCD logs they can always view the error and take necessary actions.

Option 2:
Very similar to Option 1, but if you can't tell from the reports if a test has failed then you can emit an artifact that says if a test has failed or not and then do the same in the aggregate stage to parse these artifacts and then decide to fail the stage.
Cons: If the previous option wasn't ugly enough, this definitely feels that way for me.

Option 3:
Have a separate stage after the tests that does the aggregation, instead of doing all the gymnastics with artifacts or run if attributes, you can write a notification plugin that listens to only failure events of this stage and then manually trigger the next stage.
Cons: Effort to writing to a plugin and maintaining it. Let me know if you are comfortable contributing this to the gocd-slack-notifier plugin. Happy to collaborate on this if required too.

Thanks,


--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/d70ab788-59d1-45d3-a088-77b97b3fe309n%40googlegroups.com.


--

Ashwanth Kumar / ashwanthkumar.in

Alexey Savchkov

unread,
Jan 7, 2023, 12:14:46 AM1/7/23
to go-cd
Thank you for the detailed response.


> a test failure means the report would either be incomplete or not useful isn't it?
No, by a test failure I mean the negative result of a test ("Test passed" / "Test failed"). From the programmatical point of view there is no failure so the resulting test reports as such are intact and just show that some tests have failed (not passed).

As far as I can see adding a dummy task with "runIf = any" doesn't prevent the stage from going into the failed state and terminating the pipeline. The intended next stage is not started, see below.

post-stage-run.png
post-stage-log.png
A manual trigger for generating reports is not an option for me, I'm affraid. The reports are an essential part of our daily work and are expected to be available without manually processing them for every build.

I still think that the only production-level solution to avoid such workarounds would be to have optional "post" jobs for each stage and for the entire pipeline. These jobs would be then run after the completeion of all previous taks (jobs in a stage or stages in a pipeline) and additionally controlled by a similar "run If" attribute for flexibility. One side not with respect to this is that now when a build is cancelled then even "runIf = any" jobs are not run. A "post" job with "runIf = any" should run even if a build is cancelled as there may be cleanup steps in it which must be always performed.
Also found this discussion showing that something like this is in demand.

суббота, 7 января 2023 г. в 06:23:25 UTC+7, ashwant...@gmail.com:

Sriram Narayanan

unread,
Jan 7, 2023, 6:38:09 AM1/7/23
to go...@googlegroups.com
On Sat, 7 Jan 2023 at 10:44 AM, 'Alexey Savchkov' via go-cd <go...@googlegroups.com> wrote:
Thank you for the detailed response.

> a test failure means the report would either be incomplete or not useful isn't it?
No, by a test failure I mean the negative result of a test ("Test passed" / "Test failed"). From the programmatical point of view there is no failure so the resulting test reports as such are intact and just show that some tests have failed (not passed).

As far as I can see adding a dummy task with "runIf = any" doesn't prevent the stage from going into the failed state and terminating the pipeline. The intended next stage is not started, see below.

post-stage-run.png
post-stage-log.png
A manual trigger for generating reports is not an option for me, I'm affraid. The reports are an essential part of our daily work and are expected to be available without manually processing them for every build.

I still think that the only production-level solution to avoid such workarounds would be to have optional "post" jobs for each stage and for the entire pipeline. These jobs would be then run after the completeion of all previous taks (jobs in a stage or stages in a pipeline) and additionally controlled by a similar "run If" attribute for flexibility. One side not with respect to this is that now when a build is cancelled then even "runIf = any" jobs are not run. A "post" job with "runIf = any" should run even if a build is cancelled as there may be cleanup steps in it which must be always performed.
Also found this discussion showing that something like this is in demand.


Until such a feature is implemented, you could consider the following:
- capture the exit code and save as an artifact
- exit with zero ( to indicate a successful run )
- have a job in the next stage gather all the test reports from the earlier stage and merge them
- have another job fetch all the exit code artifacts from the jobs in the earlier stages and then exit 1 if any of those exit codes were zero.

This way,
- all the tests run
- the test artifacts are available in the next stage
- the overall pipeline fails while making test execution reports available as an aggregate.


Reply all
Reply to author
Forward
0 new messages