[JIRA] (JENKINS-61130) "ignoring quality gate result for one build" feature seems not to work

[fitzner@ifao.net (JIRA)]

R. F. created an issue

Jenkins / JENKINS-61130 "ignoring quality gate result for one build" feature seems not to work Issue Type: Bug Assignee: Ulli Hafner Components: warnings-ng-plugin Created: 2020-02-18 15:26 Priority: Minor Reporter: R. F. Hello, I am using the plugin version 7.2.2 and Jenkins reports "Resetting reference build, ignoring quality gate result for one build" after pressing the "Reset quality gate" button of the previous build, but it still prints "-> Some quality gates have been missed: overall result is FAILED". I have implemented at first my Jenkins pipeline without referenceJobName and then I set it to a fixed branch name "master", but it did not change anything. Log example: 00:03:12.892 [Static Analysis] Attaching ResultAction with ID 'analysis' to run 'XYZ/aaaaaa/master #7'. 00:03:12.917 [Static Analysis] Resetting reference build, ignoring quality gate result for one build 00:03:12.917 [Static Analysis] Using reference build 'XYZ/aaaaaa/master #2' to compute new, fixed, and outstanding issues 00:03:12.917 [Static Analysis] Issues delta (vs. reference build): outstanding: 61, new: 52, fixed: 1 00:03:12.917 [Static Analysis] Evaluating quality gates 00:03:12.917 [Static Analysis] -> PASSED - New (errors only): 0 - Quality QualityGate: 1 00:03:12.917 [Static Analysis] -> PASSED - New (severity high only): 0 - Quality QualityGate: 1 00:03:12.917 [Static Analysis] -> FAILED - New (severity normal only): 51 - Quality QualityGate: 4 00:03:12.917 [Static Analysis] -> PASSED - New (severity low only): 1 - Quality QualityGate: 10 00:03:12.917 [Static Analysis] -> Some quality gates have been missed: overall result is FAILED 00:03:12.917 [Static Analysis] Enabling health report (Healthy=1, Unhealthy=12, Minimum Severity=LOW) 00:03:12.965 [Static Analysis] Created analysis result for 113 issues (found 52 new issues, fixed 1 issues) With build no. 3 I started pressing the "Reset quality gate" button. No effect. Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

[fitzner@ifao.net (JIRA)]

R. F. updated an issue

Jenkins / JENKINS-61130 "ignoring quality gate result for one build" feature seems not to work

Change By: R. F.

Hello, I am using the plugin version 7.2.2 and Jenkins reports "Resetting reference build, ignoring quality gate result for one build" after pressing the "Reset quality gate" button of the previous build, but it still prints "-> Some quality gates have been missed: overall result is FAILED". I have implemented at first my Jenkins pipeline without referenceJobName and then I set it to a fixed branch name "master", but it did not change anything. Log example:

{noformat}

00:03:12.892  [Static Analysis] Attaching ResultAction with ID 'analysis' to run 'XYZ/aaaaaa/master #7'. 00:03:12.917  [Static Analysis] Resetting reference build, ignoring quality gate result for one build 00:03:12.917  [Static Analysis] Using reference build 'XYZ/aaaaaa/master #2' to compute new, fixed, and outstanding issues 00:03:12.917  [Static Analysis] Issues delta (vs. reference build): outstanding: 61, new: 52, fixed: 1 00:03:12.917  [Static Analysis] Evaluating quality gates 00:03:12.917  [Static Analysis] -> PASSED - New (errors only): 0 - Quality QualityGate: 1 00:03:12.917  [Static Analysis] -> PASSED - New (severity high only): 0 - Quality QualityGate: 1 00:03:12.917  [Static Analysis] -> FAILED - New (severity normal only): 51 - Quality QualityGate: 4 00:03:12.917  [Static Analysis] -> PASSED - New (severity low only): 1 - Quality QualityGate: 10 00:03:12.917  [Static Analysis] -> Some quality gates have been missed: overall result is FAILED 00:03:12.917  [Static Analysis] Enabling health report (Healthy=1, Unhealthy=12, Minimum Severity=LOW) 00:03:12.965  [Static Analysis] Created analysis result for 113 issues (found 52 new issues, fixed 1 issues)

{noformat} With build no. 3 I started pressing the "Reset quality gate" button. No But it has no effect.

Add Comment

[fitzner@ifao.net (JIRA)]

With build no. 3 I started pressing the "Reset quality gate" button. But it has no effect. Any ideas or suggestions? Thank you.

Add Comment

[fitzner@ifao.net (JIRA)]

R. F. updated an issue

Jenkins / JENKINS-61130 "ignoring quality gate result for one build" feature seems not to work Change By: R. F.

Hello, I am using the plugin version 7.2.2 and Jenkins (LTS v2.190.3) reports

[ullrich.hafner@gmail.com (JIRA)]

Ulli Hafner commented on JENKINS-61130

Re: "ignoring quality gate result for one build" feature seems not to work The button changes only the selection of the reference in the following way: as reference any previous build will be chosen that is not failing (regardless of the quality gate). So I am assuming your builds #3-#6 are failing? Or let me ask the other way around: what do you want to achieve?

Add Comment

[fitzner@ifao.net (JIRA)]

R. F. updated an issue

Jenkins / JENKINS-61130

"ignoring quality gate result for one build" feature seems not to work

Change By: R. F. Attachment: 2020-02-19 09_28_56-Window.png

Add Comment

[fitzner@ifao.net (JIRA)]

R. F. commented on JENKINS-61130

Re: "ignoring quality gate result for one build" feature seems not to work With build #2 the repository suppressed JavaDoc warnings/errors via Gradle options. With #3 I enabled it and off course now the "new" issues increased dramatically. See How can I make build #3 as a new baseline?

Add Comment

[fitzner@ifao.net (JIRA)]

R. F. edited a comment on JENKINS-61130

Re: "ignoring quality gate result for one build" feature seems not to work

With build #2 the repository suppressed JavaDoc warnings/errors via Gradle options. With #3 I enabled disabled suppressing it and off course now the "new" issues increased dramatically. See !2020-02-19 09_28_56-Window.png|thumbnail!

How can I make build #3 as a new baseline?

Add Comment

[fitzner@ifao.net (JIRA)]

R. F. edited a comment on JENKINS-61130

Re: "ignoring quality gate result for one build" feature seems not to work

With build #2 the repository suppressed JavaDoc warnings/errors via Gradle options. With #3 I disabled suppressing it and off course now the "new" issues increased dramatically.

See !2020-02-19 09_28_56-Window.png|thumbnail! How can I make build #3 as a new baseline?

PS. Yes build #3-#7 are failing because the quality gates insists that the "overall result is FAILED".

Add Comment

[fitzner@ifao.net (JIRA)]

R. F. commented on JENKINS-61130

Re: "ignoring quality gate result for one build" feature seems not to work

The official Pipeline Syntax documentation is: Ignore Quality Gate (Default for ignoreQualityGate: false): If checked, then the last finished build of the reference job will be used as baseline no matter if the quality gate of that build has been passed or not. " ... Jenkins will mark all builds as unstable until the issues have been resolved." Ignore Failed Builds (Default for ignoreFailedBuilds: true): If checked, then only successful or unstable builds will be considered as baseline. I have noticed that: 1. It seems like that against the documentation "unstable builds" are treated as failed! Even if QualityGate is configured with unstable: true neither the "Reset quality gate" button is displayed nor after re-running the job the previous UNSTABLE build is concidered as new baseline. 00:02:39.415 [Static Analysis] Attaching ResultAction with ID 'analysis' to run 'XYZ/aaaaaa/develop #12'. 00:02:39.436 [Static Analysis] Using reference build 'XYZ/aaaaaa/develop #10' to compute new, fixed, and outstanding issues 00:02:39.436 [Static Analysis] Issues delta (vs. reference build): outstanding: 61, new: 52, fixed: 1 00:02:39.436 [Static Analysis] Evaluating quality gates 00:02:39.436 [Static Analysis] -> PASSED - New (errors only): 0 - Quality QualityGate: 1 00:02:39.436 [Static Analysis] -> PASSED - New (severity high only): 0 - Quality QualityGate: 1 00:02:39.436 [Static Analysis] -> WARNING - New (severity normal only): 51 - Quality QualityGate: 4 00:02:39.436 [Static Analysis] -> PASSED - New (severity low only): 1 - Quality QualityGate: 10 00:02:39.436 [Static Analysis] -> Some quality gates have been missed: overall result is WARNING 00:02:39.436 [Static Analysis] Enabling health report (Healthy=1, Unhealthy=12, Minimum Severity=LOW) 00:02:39.483 [Static Analysis] Created analysis result for 113 issues (found 52 new issues, fixed 1 issues) This explains why adding "ignoreQualityGate: true" to my pipeline did not solve my problem because the parameter ignoreFailedBuilds was still true (default). 2. Since "unstable" and "failed" builds are treated equally the plugin now would force me to set "ignoreFailedBuilds: false". If I would do this then defining a QualityGate does not realy make sense anymore because the baseline is always the previous build of the last one. Even if I would define a QualityGate and set it to unstable I cannot differenciate between UNSTABLE (publish SNAPSHOTS) and FAILED (abort and do not publish RELEASE) anymore because the plugin checks the build status and not its quality gate status. Or am I missing something?

Add Comment

[fitzner@ifao.net (JIRA)]

R. F. edited a comment on JENKINS-61130

Re: "ignoring quality gate result for one build" feature seems not to work

The official Pipeline Syntax documentation is:

# 1. Ignore Quality Gate (Default for ignoreQualityGate: false):

If checked, then the last finished build of the reference job will be used as baseline no matter if the quality gate of that build has been passed or not. " ... Jenkins will mark all builds as unstable until the issues have been resolved."

# 2. Ignore Failed Builds (Default for ignoreFailedBuilds: true):

If checked, then only successful or unstable builds will be considered as baseline.

I have noticed that: 1. It seems like that against the documentation "unstable builds" are treated as failed! Even if QualityGate is configured with unstable: true neither the "Reset quality gate" button is displayed nor after re-running the job the previous UNSTABLE build is concidered as new baseline.

{noformat}

00:02:39.415  [Static Analysis] Attaching ResultAction with ID 'analysis' to run 'XYZ/aaaaaa/develop #12'. 00:02:39.436  [Static Analysis] Using reference build 'XYZ/aaaaaa/develop #10' to compute new, fixed, and outstanding issues 00:02:39.436  [Static Analysis] Issues delta (vs. reference build): outstanding: 61, new: 52, fixed: 1 00:02:39.436  [Static Analysis] Evaluating quality gates 00:02:39.436  [Static Analysis] -> PASSED - New (errors only): 0 - Quality QualityGate: 1 00:02:39.436  [Static Analysis] -> PASSED - New (severity high only): 0 - Quality QualityGate: 1 00:02:39.436  [Static Analysis] -> WARNING - New (severity normal only): 51 - Quality QualityGate: 4 00:02:39.436  [Static Analysis] -> PASSED - New (severity low only): 1 - Quality QualityGate: 10 00:02:39.436  [Static Analysis] -> Some quality gates have been missed: overall result is WARNING 00:02:39.436  [Static Analysis] Enabling health report (Healthy=1, Unhealthy=12, Minimum Severity=LOW) 00:02:39.483  [Static Analysis] Created analysis result for 113 issues (found 52 new issues, fixed 1 issues)

{noformat}

This explains why adding "ignoreQualityGate: true" to my pipeline did not solve my problem because the parameter ignoreFailedBuilds was still true (default). 2. Since "unstable" and "failed" builds are treated equally the plugin now would force me to set "ignoreFailedBuilds: false". If I would do this then defining a QualityGate does not realy make sense anymore because the baseline is always the previous build of the last one. Even if I would define a QualityGate and set it to unstable I cannot differenciate between UNSTABLE (publish SNAPSHOTS) and FAILED (abort and do not publish RELEASE) anymore because the plugin checks the build status and not its quality gate status. Or am I missing something?

Add Comment

[ullrich.hafner@gmail.com (JIRA)]

Ulli Hafner commented on JENKINS-61130

Re: "ignoring quality gate result for one build" feature seems not to work

This is exactly what I mean: the reference build selection for quality gates that fail a build is not really working, see JENKINS-61140. The options currently available make only sense if you are using a quality gate that sets a build to unstable.

Add Comment

[fitzner@ifao.net (JIRA)]

R. F. commented on JENKINS-61130

Re: "ignoring quality gate result for one build" feature seems not to work

A little correction on my side concerning my previous comment: " ... neither the "Reset quality gate" button is displayed ... " this is wrong. That buttons is displayed but only on the very latest build.

Add Comment

[fitzner@ifao.net (JIRA)]

R. F. edited a comment on JENKINS-61130

Re: "ignoring quality gate result for one build" feature seems not to work

A little correction on my side concerning my previous comment: " ... neither the "Reset quality gate" button is displayed ... " this is wrong. That buttons The button is displayed but only on the very latest build.

Add Comment

[oliverst@online.de (JIRA)]

Oliver Stöneberg commented on JENKINS-61130

Re: "ignoring quality gate result for one build" feature seems not to work

I hope I am not going off-topic with this, but it seems like a variation of the issue reported here. I have a repo with quality gates and the reference is "master". If I create a PR request it will fail if there are additional findings. I expected Reset quality gate to allow these additional findings and set a new reference. That does seems to be the case. When I start the build again it will again FAIL. The configuration looks like this: qualityGates: [[threshold: 1, type: 'NEW', unstable: false]]

Add Comment

This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)

[ullrich.hafner@gmail.com (JIRA)]

Ulli Hafner commented on JENKINS-61130

Re: "ignoring quality gate result for one build" feature seems not to work

If you specify another branch as reference, then the reference build is always from that specified branch. In this case the option has no effect at all. Would you expect that the reference build is then chosen from the PR branch?

Add Comment

[oliverst@online.de (JIRA)]

Oliver Stöneberg commented on JENKINS-61130

Re: "ignoring quality gate result for one build" feature seems not to work

As the button is shown I would expect it to do something useful. If it will never have any effect it should not be offered at all. It would also mean, that quality gates cannot be used with a branching/pull request model at all. If you would use the branch as reference the first build of the branch would create that reference. But the first push on the branch might already contain additional changes compared to the code you branched from. So even if you have automatic builds on that branch the reference might be different than the master. So after merging that branch to the master it would cause a build failure if there are additional issues and you need to reset the quality gate and run the build again. It might work if you are flagging the build as UNSTABLE but in that case issues would be introduced in master builds temporarily.

Add Comment

[ullrich.hafner@gmail.com (JIRA)]

Ulli Hafner commented on JENKINS-61130

Re: "ignoring quality gate result for one build" feature seems not to work

As the button is shown I would expect it to do something useful. If it will never have any effect it should not be offered at all.

Yes, when the button has been added, we simply forgot this use case...

It would also mean, that quality gates cannot be used with a branching/pull request model at all. If you would use the branch as reference the first build of the branch would create that reference. But the first push on the branch might already contain additional changes compared to the code you branched from. So even if you have automatic builds on that branch the reference might be different than the master. So after merging that branch to the master it would cause a build failure if there are additional issues and you need to reset the quality gate and run the build again. It might work if you are flagging the build as UNSTABLE but in that case issues would be introduced in master builds temporarily.

Selecting the right baseline build is not trivial indeed. One of my students is currently developing an algorithm for that problem (see also /JENKINS-33274) in his master thesis.

Add Comment

[ullrich.hafner@gmail.com (JIRA)]

Ulli Hafner edited a comment on JENKINS-61130

Re: "ignoring quality gate result for one build" feature seems not to work

{quote}

As the button is shown I would expect it to do something useful. If it will never have any effect it should not be offered at all.

{quote}

Yes, when the button has been added, we simply forgot this use case...

Can you please create a followup Jira bug, so that we can see what to do in this case? {quote}

It would also mean, that quality gates cannot be used with a branching/pull request model at all. If you would use the branch as reference the first build of the branch would create that reference. But the first push on the branch might already contain additional changes compared to the code you branched from. So even if you have automatic builds on that branch the reference might be different than the master. So after merging that branch to the master it would cause a build failure if there are additional issues and you need to reset the quality gate and run the build again. It might work if you are flagging the build as UNSTABLE but in that case issues would be introduced in master builds temporarily.

{quote}

Selecting the right baseline build is not trivial indeed. One of my students is currently developing an algorithm for that problem (see also /JENKINS-33274) in his master thesis.

Add Comment