[JIRA] (JENKINS-57258) Lost builds historical data after updating to Jenkins v2.175

[nico.tgee@gmail.com (JIRA)]

Nicolas Echegut created an issue

Jenkins / JENKINS-57258 Lost builds historical data after updating to Jenkins v2.175 Issue Type: Bug Assignee: Unassigned Attachments: branches.PNG, build_history.PNG, failed_build_console_log.txt Components: core, pipeline Created: 2019-04-30 14:14 Environment: Jenkins v2.175 / Unix master / Java 64bit 1.8.0_192 Priority: Blocker Reporter: Nicolas Echegut I'm using Multibranch Pipeline to automatically build changes in my Enterprise GitHub Organization repositories. After updating Jenkins to 2.175 and restarting, ALL the successful builds of ALL the branches of ALL my GitHub org repositories (that's around 75 repositories and approximately 2,000 builds) have been retriggered and instantly failed. All the builds failed with the same error, see attached console log. Now, all the projects and branches and builds are marked as Failed, and some build information such as the build date appear to be lost (see screenshots). In the example log, we can see the build was successfully completed a week ago, but restarted and failed today after restarting Jenkins.  I can consistenly reproduce this issue by re-building a project and restarting Jenkins. The issue does not happen if I downgrade to Jenkins 2.174. Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[dbeck@cloudbees.com (JIRA)]

Daniel Beck commented on JENKINS-57258

Re: Lost builds historical data after updating to Jenkins v2.175 Did you also update any plugins?

Add Comment

[nico.tgee@gmail.com (JIRA)]

Nicolas Echegut commented on JENKINS-57258

Re: Lost builds historical data after updating to Jenkins v2.175

I did prior to updating Jenkins, but restarted the instance in between. The only plugin updates were related to Blue Ocean.

Add Comment

[jeremy@mathevet.net (JIRA)]

Jeremy Mathevet commented on JENKINS-57258

Re: Lost builds historical data after updating to Jenkins v2.175

Similar issue here, only not on all repos and all branches. I can't find a common pattern unfortunately.

Add Comment

[bmathus+ossjira@cloudbees.com (JIRA)]

Baptiste Mathus commented on JENKINS-57258

Re: Lost builds historical data after updating to Jenkins v2.175

That would (loosely?) hint about the serialization format of the workflow-support:3.x line, but if you're saying you had updates all plugins before, that doesn't look like it. To be sure: can you confirm you were already running with all latest plugins, specifically workflow-support was 3.2? cc Devin Nusbaum

Add Comment

[bmathus+ossjira@cloudbees.com (JIRA)]

Baptiste Mathus edited a comment on JENKINS-57258

Re: Lost builds historical data after updating to Jenkins v2.175

That would (loosely?) hint about the serialization format change of the workflow-support:3.x line, but if you're saying you had updates all plugins before, that doesn't look like it.

To be sure: can you confirm you were already running with all latest plugins, specifically workflow-support was 3.2?

cc [~dnusbaum]

Add Comment

[o.v.nenashev@gmail.com (JIRA)]

Oleg Nenashev commented on JENKINS-57258

Re: Lost builds historical data after updating to Jenkins v2.175

After going through https://github.com/jenkinsci/jenkins/compare/jenkins-2.174...jenkins-2.175 , I do not see anything strictly related to this issue. I could assume a weird side effect of https://github.com/jenkinsci/jenkins/pull/3980 , but there is no data to confirm it.   Baptiste Mathus Devin Nusbaum .dat files and format changes should be related to Running builds only, I am not sure these files are used for completed builds. Any chance we have issues with build status persistency in  build.xml    Nicolas Echegut If you have build.xml files and System logs after the upgade to 2.175, it would help a lot

Add Comment

[dnusbaum@cloudbees.com (JIRA)]

Devin Nusbaum commented on JENKINS-57258

Re: Lost builds historical data after updating to Jenkins v2.175

Any chance we have issues with build status persistency in build.xml We have had issues with this kind of thing in the past (JENKINS-50199), but I am not aware of any related issues in recent plugin versions, although it does remind me of the issues some people are seeing where Pipeline Jobs seem to hold onto flyweight executors after they have apparently completed (discussed in JENKINS-45571). Nicolas Echegut If you have build.xml files from before the upgrade and can tell us whether you updated the Pipeline Job, Pipeline Groovy, or Pipeline Supporting APIs plugins and if so, what versions they were previously and what versions they are now, that would help.

Add Comment

[nico.tgee@gmail.com (JIRA)]

Nicolas Echegut updated an issue

Jenkins / JENKINS-57258

Lost builds historical data after updating to Jenkins v2.175

Change By: Nicolas Echegut Attachment: build_before.xml

Add Comment

[nico.tgee@gmail.com (JIRA)]

Nicolas Echegut updated an issue

Jenkins / JENKINS-57258 Lost builds historical data after updating to Jenkins v2.175

Change By: Nicolas Echegut Attachment: build_after.xml

Add Comment

[nico.tgee@gmail.com (JIRA)]

Nicolas Echegut commented on JENKINS-57258

Re: Lost builds historical data after updating to Jenkins v2.175 I grabbed the build.xml from a build before and after restart.  I updated workflow-support from 3.2 to 3.3 right before upgrading Jenkins, but restarted Jenkins in-between. I was already on the latest version of workflow-job-plugin and workflow-cps-plugin.

Add Comment

[dnusbaum@cloudbees.com (JIRA)]

Devin Nusbaum commented on JENKINS-57258

Re: Lost builds historical data after updating to Jenkins v2.175

Based on those files, the pre-update run and flow execution have a result, the run is marked as completed and the flow execution is marked as done, which all looks good. The only things that seem off is that the flow execution does not have persistedClean = true, but that by itself shouldn't be a problem, and that the CloverHtmlBuildAction and checkout information were nulled out as part of the update. This does remind me of JENKINS-57254, which was also reported against 2.175 recently, although apparently that was caused by the SCoverage plugin which doesn't appear in your build.xml files. Do you see any errors in your Jenkins system logs around the time of the upgrade?

Add Comment

[nico.tgee@gmail.com (JIRA)]

Nicolas Echegut commented on JENKINS-57258

Re: Lost builds historical data after updating to Jenkins v2.175

I don't see anything useful in the log and error files from the day of the upgrade. I'm not using the SCoverage plugin.

Add Comment

[dbeck@cloudbees.com (JIRA)]

Daniel Beck commented on JENKINS-57258

Re: Lost builds historical data after updating to Jenkins v2.175

Does this problem still occur with Jenkins 2.176? I assume instances were restored from backup when this happened, so what happens when you skip 2.175?

Add Comment

[nico.tgee@gmail.com (JIRA)]

Nicolas Echegut commented on JENKINS-57258

Re: Lost builds historical data after updating to Jenkins v2.175

Unfortunately this Jenkins instance did not have any backup so nothing was recovered. I skipped 2.175 and 2.176 and upgraded to 2.177. The good news is the issue with builds automatically restarting and failing after a Jenkins restart is not occurring anymore. The bad news is we probably will never know what happened now  It may have been related to JENKINS-57254. Anyway, I will go ahead and close this Jira since the issue cannot be reproduced anymore. Thanks for looking into this.

Add Comment

[nico.tgee@gmail.com (JIRA)]

Nicolas Echegut closed an issue as Cannot Reproduce

Jenkins / JENKINS-57258

Lost builds historical data after updating to Jenkins v2.175

Change By: Nicolas Echegut Status: Open Closed Resolution: Cannot Reproduce

Add Comment