[JIRA] (JENKINS-17265) Builds disappearing from history

[1990.alex@gmail.com (JIRA)]

Alex Vesely created JENKINS-17265 Builds disappearing from history Issue Type: Bug Assignee: Unassigned Components: core Created: 19/Mar/13 8:54 AM Description: Revisiting https://issues.jenkins-ci.org/browse/JENKINS-15156 once again. The issue still persists. Builds are disappearing from history, even though they are present on the disk. They reappear after Jenkins restart. Environment: Jenkins 1.506 Project: Jenkins Labels: history Priority: Critical Reporter: Alex Vesely

This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira

[harpreetnain@gmail.com (JIRA)]

Harpreet Nain commented on JENKINS-17265 Builds disappearing from history We have the same issue on version 1.494, 1.499 and 1.502. We dont have any master/slave configuration in place. The job history seems to come back after we do a "Reload Configuration from Disk"

[matt@bunkerhollow.com (JIRA)]

Matt Williamson commented on JENKINS-17265 Builds disappearing from history Same issue on 1.506, job history for some jobs disappears. Selecting "Reload Configuration from Disk" brings the history back.

[florian.kirchmeir@unycom.com (JIRA)]

kflorian commented on JENKINS-17265 Builds disappearing from history Confirming this on Jenkins 1.505, running on Linux with Linux slaves. Build results seem to disappear from some jobs, while they show up as usual on others. Seems to happen even more frequently in 1.505 than in the version(s) before! Also confirming that "Reload Configuration from Disk" brings back the history.

[josesa@java.net (JIRA)]

Jose Sa commented on JENKINS-17265 Builds disappearing from history Please notice that "Reload from disk" may be unsafe if you have actively running jobs as indicated in JENKINS-3265

[wbauer@java.net (JIRA)]

wbauer commented on JENKINS-17265 Builds disappearing from history Same issue for me, had it with the 1.4xxx builds and now still with 1.505 - really annoying!

[oldelvet@java.net (JIRA)]

Richard Mortimer commented on JENKINS-17265 Builds disappearing from history We need to try and identify what may/may not be causing the problems. Quite a few fixes went into 1.505 and earlier but there are obviously still a few gremlins in there. Here's a few questions that those of you seeing the problem can try to answer: are there any relevant messages in any of the Jenkins logs. Any exceptions, notices about failure to load are useful. what job types do problem occur on? maven, freestyle, other. are you running Jenkins direct from the war file or within tomcat/similar environment. what OS are your masters/slaves running? Answers to those will help to point towards the right person to fix/investigate further.

[harpreetnain@gmail.com (JIRA)]

Harpreet Nain commented on JENKINS-17265 Builds disappearing from history Richard, Here are the answers as per our environment: 1. There is nothing that we can see coming through in the logs 2. We are using freestyle jobs 3. The problem occurs on both i.e running direct from war file and from tomcat. 4. We dont have a master / slave setup. We have 2 instances one on linux and and one on windows. It happens on both.

[nickolay.martinov@hp.com (JIRA)]

Nickolay Martinov commented on JENKINS-17265 Builds disappearing from history 1. Havent spotted exceptions. The only thing reporting error is Copy artifact plugin that cant find a build 2. Freestyle jobs. Before 1.505 in matrix jobs builds disappeared veeery often . Matrix jobs dont seem to disappear now or eat least not so often 3. Running using winstone embedded in jenkins 4. HP-UX 11.21 Itanium Java 7 master. HP-UX 11.31 Itanium Java 7 and RHEL 5.4/5.6/5.8 Java 7 slaves.

[wbauer@java.net (JIRA)]

wbauer commented on JENKINS-17265 Builds disappearing from history 1. Nothing visible in the logs. 2. We use freestyle jobs across the board. 3. Running from war file. 4. master jenkins 1.505 on Mac OS X 10.8.2 / Java 1.7 latest, slaves Win XP/7/w2k8s + RedHat 6.2 with Java 1.6 + Mac OS X 10.6/7/8 with Java 1.6

[matt@bunkerhollow.com (JIRA)]

Matt Williamson commented on JENKINS-17265 Builds disappearing from history FYI - This issue hasn't occurred again for us since "Reload from Disk" and updating all plugins about a week ago. That's progress, this use to happen all the time. 1. Can't find anything in the logs. 2. All freestyle projects. 3. Running from war file I assume. 4. Amazon Linux (amzn-ami-pv-2012.03.3.x86_64-ebs (ami-aecd60c7)), installed Jenkins through yum.

[1990.alex@gmail.com (JIRA)]

Alex Vesely commented on JENKINS-17265 Builds disappearing from history I'm not sure, but at one time I was under the impression that this issue is seen more often with newly-created jobs - i.e., jobs created during the current Jenkins session. I'm using the Jenkins CLI to create chains of jobs (A runs B, B runs C, etc). After several runs of the chain, some or all builds disappear. 1. Haven't seen anything. Where should I look? 2. "Freestyle -> matrix -> matrix -> freestyle" chain 3. From the .war 4. SLES 10 for the master, SLES 10, Win2003, Win2008 slaves.

[oldelvet@java.net (JIRA)]

Richard Mortimer commented on JENKINS-17265 Builds disappearing from history @Alex Vesely you can generally find the logs at http://<myjenkins.url>/log/all (replace <myjenkins.uk> with the base URL of your instance).

[oldelvet@java.net (JIRA)]

Richard Mortimer commented on JENKINS-17265 Builds disappearing from history All, thanks for you quick feedback. It certainly looks like freestyle jobs are affected and maybe other types of jobs too. Certainly there no obvious common denominator in what has been reported. Ok. So I guess its time to look for a few more possibilities. does it happen on any specific revision control system? SVN, GIT etc. what plugins are loaded in your jenkins and are there any common build steps added to failing jobs. are there any jobs on your system that do not seem to go missing?

[josesa@java.net (JIRA)]

Jose Sa commented on JENKINS-17265 Builds disappearing from history I was trying to create some script to crosscheck locally available builds with the ones reported on web page using REST api appending to job "/api/json?tree=builds[number]" and noticed something curious, not all builds are reported in that list when comparing with the ones actually visible in build history, only the most recent ones on variable ammount. Not sure if there is any limitation/restriction to the api, but maybe this is a symptom of this problem because if it already thinks it has less builds that it actually has, on some cleanup procedure it might only keep the ones reported and discard the rest.

[harpreetnain@gmail.com (JIRA)]

Harpreet Nain commented on JENKINS-17265 Builds disappearing from history Richard, We are using Git. Builds disappear randomly. I have not seen any pattern. On one occasion a build that had been run about 200 times without issues, lost all the build history which obviously reappeared when we did a reload. I am attaching screenshots of the plugins installed and their versions

[harpreetnain@gmail.com (JIRA)]

Harpreet Nain updated JENKINS-17265 Builds disappearing from history HN Change By: Harpreet Nain (28/Mar/13 5:35 AM) Attachment: plugin3.bmp Attachment: plugin1.bmp Attachment: plugin2.bmp

[nickolay.martinov@hp.com (JIRA)]

Nickolay Martinov commented on JENKINS-17265 Builds disappearing from history Using SVN, standard plugin. Using only matrix and freestyle jobs. Previously both were disappearing like crazy. Recently matrix jobs seems to become more stable. Freestyle jobs still have problems. Perhaps makes to note that jobs are not standalone: both matrix and freestyle jobs trigger each other (build other builds) heavily. Installed plugins (env injector is disabled): Jenkins Mailer Plugin 1.4 External Monitor Job Type Plugin 1.1 LDAP Plugin 1.2 pam-auth 1.0 Javadoc Plugin 1.1 Token Macro Plugin This plug-in adds reusable macro expansion capability for other plug-ins to use. 1.6 Maven 2 Project Plugin 1.505 Jenkins Subversion Plug-in 1.45 Ant Plugin 1.2 Jenkins CVS Plug-in Integrates Jenkins with CVS version control system using a modified version of the Netbeans cvsclient. 2.8 Dump Info BuildWrapper Plugin The Dump Info BuildWrapper Plugin allows jobs to automatically dump some important Hudson-specific information into the job log. 1.1 Hudson platformlabeler plugin Assigns labels on nodes automatically based on the computer operating system. 1.1 Tracking SVN Plugin 1.1 Priority Sorter This plugin allows for the build queue to be sorted based on pre-assigned priorities for each job. 1.3 Jenkins Translation Assistance plugin 1.10 Static Analysis Utilities This plug-in provides utilities for the static code analysis plug-ins. If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 1.49 Task Scanner Plugin This plug-in scans for open tasks in a specified set of files in the project modules and visualizes the results. If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 4.35 Disk Usage Plugin This plugin counts disk usage. 0.18 SSH Slaves plugin 0.22 FindBugs Plug-in This plug-in collects the FindBugs analysis results of the project modules and visualizes the found warnings. If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 4.48 Checkstyle Plug-in This plug-in collects the Checkstyle analysis results of the project modules and visualizes the found warnings. If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 3.35 Git Client Plugin 1.0.3 Git Plugin This plugin integrates GIT with Jenkins. 1.2.0 View Job Filters Create smart views with exactly the jobs you want. Your smart views can automatically include or exclude jobs by using things like the SCM path or type, the job type, build statuses or trends or triggers, relevance to the logged-in user, email recipients, Maven configuration, job parameterization, and user permissions. Mix and match filters to narrow down to exactly what you want. 1.22 Warnings Plug-in This plug-in collects the compiler warnings of the project modules and visualizes the results. If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 4.23 PMD Plug-in This plug-in collects the PMD analysis results of the project modules and visualizes the found warnings. If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 3.34 Copy Artifact Plugin Adds a build step to copy artifacts from another project. 1.25 Downstream build view This plug-in allows you to view the full status all the downstream builds so that we can graphically see that everything for this build has been completed successfully. 1.8 Hudson Blame Subversion Plug-in 1.200 Groovy Postbuild Groovy postbuild task. 1.8 groovyaxis This plugin is a sample to explain how to write a Jenkins plugin. 0.3 Duplicate Code Scanner Plug-in This plug-in collects the duplicate code analysis results of the project modules and visualizes the found warnings. Supported duplicate code analysis tools: • PMD's Copy Paste Detector (CPD) • Simian If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 2.34 Static Analysis Collector Plug-in This plug-in is an add-on for the plug-ins Checkstyle, Dry, FindBugs, PMD, Tasks, and Warnings: the plug-in collects the different analysis results and shows the results in a combined trend graph. Additionally, the plug-in provides health reporting and build stability based on these combined results. If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 1.35 Slave Prerequisites plugin Run a script on slave node before starting job on it to check some prerequisites. 1.0 Matrix Tie Parent plugin 1.1 Jenkins Maven Invoker plugin 1.2 M2 Repository Cleanup Plugin This plugin allows you to configure a maven2 job to clean some or all of the artifacts from the repository before it runs. 1.0 Jenkins Multijob plugin This plugin is a MultiJob plugin. 1.8 Jenkins Maven Release Plug-in Plug-in A plug-in that enables you to perform releases using the maven-release-plugin from Hudson. 0.9.1 Config File Provider Plugin Ability to provide configuration files (e.g. settings.xml for maven, XML, groovy, custom files,...) loaded trough the UI which will be copy to the job workspace. 2.4 Run Condition Plugin 0.10 conditional-buildstep A buildstep wrapping any number of other buildsteps, controlling there execution based on a defined condition (e.g. BuildParameter). 1.2.1 Copy data to workspace plugin This plugin copies data to workspace directory before each build. After build data will be deleted. 1.0 Tool Environment plugin Lets you use "tools" in unusual ways, such as from shell scripts. 1.0 Copy To Slave Plugin This plugin allows copying files located somewhere on the master node into the jobs' workspaces, whether their builds take place on the master node or on slave nodes. 1.4 Async Http Client This plugin provides a shared dependency on the async-http-client library so that other plugins can co-operate when using this library. 1.7.8 Maven Deployment Linker 1.5 Build Pipeline Plugin This plugin renders upstream and downstream connected jobs that typically form a build pipeline. In addition, it offers the ability to define manual triggers for jobs that require intervention prior to execution, e.g. an approval process outside of Jenkins. 1.3.3 Downstream-Ext This plugin provides extended options for triggering downstream projects. 1.8 Matrix Reloaded Plugin The Matrix Reloaded plugin. Used for rebuilding a subset of a matrix' configurations, reusing the rest. 1.1.0 Environment Injector Plugin This plugin makes it possible to set an environment for the builds. 1.83 Jenkins Maven Repository Server Plugin 0.11 Hudson Groovy builder This plugin executes Groovy code. 1.13 All changes plugin This plugin shows all changes (also from dependent projects, sub-projects, ...) for a project 1.3 Pending Changes plugin This plugin adds a view of all changes in the scm after the last build 0.3.0 Changes since last successfull build Plugin Generate changelog for a range of jenkins builds. 0.5

[nickolay.martinov@hp.com (JIRA)]

Nickolay Martinov commented on JENKINS-17265 Builds disappearing from history Jose has made interesting note. I recall that sometimes we can see build listed in UI but Copy Artifacts Plugin seems to fail to find the build.

[ivan.veselkov@gmail.com (JIRA)]

Ivan Veselkov commented on JENKINS-17265 Builds disappearing from history We have a matrix job with two configurations (os = hpux or linux ). There is a keep-forever build which appears in parent history but it is absent in histories of configurations. From parent build one can still access both configurations.

[ivan.veselkov@gmail.com (JIRA)]

Ivan Veselkov edited a comment on JENKINS-17265 Builds disappearing from history We have a matrix job with two configurations (os = hpux or linux ). There is a keep-forever build which appears in parent history but it is absent in histories of configurations. From parent build one can still access both configurations. The copy plugin cannot copy artifacts from this build: "Unable to find a build for artifact copy from".

[ivan.veselkov@gmail.com (JIRA)]

Ivan Veselkov edited a comment on JENKINS-17265 Builds disappearing from history We have a matrix job with two configurations (os = hpux or linux ). There is a keep-forever build which appears in parent history but it is absent in histories of configurations. From parent build one can still access both configurations. The copy plugin almost always cannot copy artifacts from this build: "Unable to find a build for artifact copy from" but at least once it succeeded for os=hpux.

[fmerrow@qualcomm.com (JIRA)]

Frank Merrow commented on JENKINS-17265 Builds disappearing from history Just chiming in with "me too" on 1.502 . . . it is new installation and no matter what I do it only seems to show jobs from "today". (Have not been around at midnight to look . . . but in morning onward I only see jobs that ran today). FWIW: If I look in the jobs directory, the correct number of history directories are there . . . but web view just doesn't show them. (???) Windows7, no slaves, nothing odd about the installation that isn't in 10 others I have that are working on Jenkins from 1.488 onward. Also, I have a second 1.502 installation that was brought on-line around the same time . . . everything shows there. I cannot say the plugin mix on the two systems is identical, but it is very similar.

[josesa@java.net (JIRA)]

Jose Sa commented on JENKINS-17265 Builds disappearing from history Both "/api/json?tree=builds[number]" and "/rssAll" job urls seem to be returning only latest builds (not all as expected), like I wrote above this could be a symptom for this case. Luckily "/buildHistory/all" does still provide all visible builds, so I was able to create created a bash script to help automatically to check for this kind of problems and provide a text report. # Assuming some host specific variables jenkins_url=https://localhost:8080 jobs_dir=/work/jenkins/jobs # This should work for any unix environment using bash for job_dir in ${jobs_dir}/*; do job=$(basename ${job_dir}) echo "Checking $job..." >&2 job_url=${jenkins_url}/job/${job} wget -q --no-check-certificate -O all ${job_url}/buildHistory/all gawk -F'#' '/ #/{print $2}' all > online_builds find ${jobs_dir}/${job}/builds/* * -type l ! -name "last*" -prune | sed "s,${jobs_dir}/${job}/builds/,,g" > local_builds missing=$(gawk ' BEGIN {first_file = ""} { if (first_file == "") first_file = FILENAME if (first_file == FILENAME) {build_list[$1]=1} else {delete build_list[$1]} } END {for (build in build_list) print build} ' local_builds online_builds) [ ! -z "${missing}" ] && echo Job $job missing builds: $missing done | tee missing_builds.txt I found out that after 3 days online, 85 (out of 1406) jobs are showing missing builds varying from a few to all. I have FreeStyle and MultiJob types and they both show this problem. We are using mixed ClearCase and SVN (both showing same problems) Server is Solaris 10 Sparc, having slaves on HP-UX, Windows (XP, 2k3, 7), RHEL (5, 6), Solaris. Running Jenkins from war file, currently with 1.505 (but noticing this problem since 1.485)

[josesa@java.net (JIRA)]

Jose Sa edited a comment on JENKINS-17265 Builds disappearing from history Both "/api/json?tree=builds[number]" and "/rssAll" job urls seem to be returning only latest builds (not all as expected), like I wrote above this could be a symptom for this case.

Luckily "/buildHistory/all" does still provide all visible builds, so I was able to create created a bash script to help automatically to check for this kind of problems and provide a text report.

# Assuming some host specific variables jenkins_url=https://localhost:8080 jobs_dir=/work/jenkins/jobs # This should work for any unix environment using bash for job_dir in ${jobs_dir}/*; do job=$(basename ${job_dir}) echo "Checking $job..."

>&2 job_url=${jenkins_url}/job/${job} wget -q --no-check-certificate -O all ${job_url}/buildHistory/all gawk -F'#' '/ #/{print $2}' all > online_builds find ${jobs_dir}/${job}/builds/* -type l ! -name "last*" -prune | sed "s,${jobs_dir}/${job}/builds/,,g" > local_builds missing=$(gawk ' BEGIN {first_file = ""} {

if (first_file == "") first_file = FILENAME if (first_file == FILENAME) {build_list[$1]=1} else {delete build_list[$1]} } END {for (build in build_list) print build} ' local_builds online_builds) [ ! -z "${missing}" ] && echo Job $job missing builds: $missing done | tee missing_builds.txt

I found out that after 3 days online, 85 (out of 1406) jobs are showing missing builds varying from a few to all. I have FreeStyle and MultiJob types and they both show this problem. We are using mixed ClearCase and SVN (both showing same problems) Server is Solaris 10 Sparc, having slaves on HP-UX, Windows (XP, 2k3, 7), RHEL (5, 6), Solaris. Running Jenkins from war file, currently with 1.505 (but noticing this problem since 1.485)

This message is automatically generated by JIRA.

[nickolay.martinov@hp.com (JIRA)]

Nickolay Martinov commented on JENKINS-17265 Builds disappearing from history I confirm that script is perfect reproducer: Checking Infrastructure_update_sys... Checking Infrastructure_update_valgrind... Checking NOM_6_build... Job NOM_6_build missing builds: 887 618 Checking NOM_6_kit... Job NOM_6_kit missing builds: 204 Checking NOM_6_kit_smoke... Job NOM_6_kit_smoke missing builds: 139 Checking NOM_6_test_f... Checking NOM_6_test_f_m... Job NOM_6_test_f_m missing builds: 88 Checking NOM_6_test_l... Checking NOM_6_test_l_m... Job NOM_6_test_l_m missing builds: 204 Checking NOM_6_test_sdk... Job NOM_6_test_sdk missing builds: 85 Checking NOM_6_test_sdk_m... Checking NOM_V600_build... Checking NOM_V600_kit... Checking NOM_V600_kit_smoke...

[josesa@java.net (JIRA)]

Jose Sa commented on JENKINS-17265 Builds disappearing from history Got this exception on some jobs when doing the wget for <job>/buildHistory/all a few minutes after jenkins was restarted. Mar 29, 2013 9:26:04 PM hudson.ExpressionFactory2$JexlExpression evaluate WARNING: Caught exception evaluating: it.renderList in /hudson/job/PM_OSS5_EBSS_LV2_FUNCTIONAL_GA12/buildHistory/all. Reason: java.lan g.reflect.InvocationTargetException java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314) at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185) at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsIterator(ExpressionSupport.java:94) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:89) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53) at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658) at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:384) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:487) at org.kohsuke.stapler.Stapler.service(Stapler.java:164) at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:248) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.NullPointerException at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:214) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:349) at hudson.model.Run.onLoad(Run.java:319) at hudson.model.RunMap.retrieve(RunMap.java:226) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:619) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:358) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:214) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:349) at hudson.model.Run.onLoad(Run.java:319) at hudson.model.RunMap.retrieve(RunMap.java:226) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:619) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:358) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:214) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:349) at hudson.model.Run.onLoad(Run.java:319) at hudson.model.RunMap.retrieve(RunMap.java:226) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:619) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:358) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:214) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:349) at hudson.model.Run.onLoad(Run.java:319) at hudson.model.RunMap.retrieve(RunMap.java:226) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:619) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:358) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:214) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:101) at hudson.model.RunMap$1.next(RunMap.java:107) at hudson.model.RunMap$1.next(RunMap.java:96) at hudson.util.Iterators$2$1.next(Iterators.java:204) at hudson.widgets.HistoryWidget.updateFirstTransientBuildKey(HistoryWidget.java:110) at hudson.widgets.HistoryWidget.getRenderList(HistoryWidget.java:139) ... 126 more Which seems related to another issue JENKINS-16845 regarding the specific NPE on AbstractBuild.getPreviousBuild.

[oldelvet@java.net (JIRA)]

Richard Mortimer commented on JENKINS-17265 Builds disappearing from history Good detective work all!

Caused by: java.lang.NullPointerException at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:214) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:349) at hudson.model.Run.onLoad(Run.java:319) at hudson.model.RunMap.retrieve(RunMap.java:226) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657)

I think this relates to the following in AbstractBuild R pb = getParent().builds.search(number-1, Direction.DESC); This could be builds not being initialised as per the suggested JENKINS-16845 or it could be getParent() returning null. getParent() returns the transient project value from Run.java protected transient final JobT project; @Jose Sa can you confirm what version you saw this error on.

[josesa@java.net (JIRA)]

Jose Sa commented on JENKINS-17265 Builds disappearing from history I'm with core 1.505 now. Tried more recent versions like 1,505, 1.506 and 1.508, but had to revert back due worse problems caused by NPE not being able to load jobs like the previous issue, but now extended to Promote plugin JENKINS-17381 along with weird changed in SSH Slaves plugin credentials definitions that got all messed up.

[josesa@java.net (JIRA)]

Jose Sa edited a comment on JENKINS-17265 Builds disappearing from history I'm with core 1.505 now. Tried more recent versions like 1.506 and 1.508, but had to revert back due worse problems caused by NPE not being able to load jobs like the previous issue, but now extended to Promote plugin JENKINS-17381 along with weird changed in SSH Slaves plugin credentials definitions that got all messed up.

[josesa@java.net (JIRA)]

Jose Sa updated JENKINS-17265 Builds disappearing from history After some analysis of my previous script I realized it contained a lot of false positives possibly due to years of accumulated history of broken builds, java crashes, wrong copying of jobs between servers, etc, that lead to the following most common cases: numbered links pointing to removed build directories (solution: remove links) build directories without build.xml file (solution: remove directories) build directories without numbered links (solution: create links) build directories not in date format (solution: delete directories) Since this is a common cause many I created a simple script to detect those common problems with optional cleanup (passing argument '-c') - Unable to embed resource: check_invalid_builds.sh of type application/x-sh Also the same script for checking the missing builds as given as excerpt above - Unable to embed resource: check_missing_builds.sh of type application/x-sh Tested in RHEL6 and Solaris 10 (with gnu utils) Change By: Jose Sa (01/Apr/13 4:24 AM) Attachment: check_invalid_builds.sh Attachment: check_missing_builds.sh

[josesa@java.net (JIRA)]

Jose Sa edited a comment on JENKINS-17265 Builds disappearing from history After some analysis of my previous script I realized it contained a lot of false positives possibly due to years of accumulated history of broken builds, java crashes, wrong rsync of jobs between servers, etc, that lead to the following most common problems:

numbered links pointing to removed build directories (solution: remove links) build directories without build.xml file (solution: remove directories) build directories without numbered links (solution: create links) build directories not in date format (solution: delete directories)

Since these happen often I created some scripts to: Detect those common problems with optional cleanup (passing argument '-c') - check_invalid_builds.sh Checking the missing builds as given as excerpt above - check_missing_builds.sh NOTE: to effectively apply the cleanups first run the script without '-c' to run in preview mode and review the suggestions, when ready to apply them shutdown Jenkins first so that there are no files pending in memory to be written to file system.

Tested in RHEL6 and Solaris 10 (with gnu utils)

This message is automatically generated by JIRA.

[jglick@cloudbees.com (JIRA)]

Jesse Glick resolved JENKINS-17265 as Duplicate Builds disappearing from history Seems to be the same as JENKINS-16845. Change By: Jesse Glick (03/Apr/13 7:13 PM) Status: Open Resolved Resolution: Duplicate

[westhusing@gmail.com (JIRA)]

Adam Westhusing commented on JENKINS-17265 Builds disappearing from history The shell scripts don't work when the jobs have spaces in the names. What modifications need to be made to the scripts to adapt it to be able to work when there are spaces in the job names?

[westhusing@gmail.com (JIRA)]

Adam Westhusing edited a comment on JENKINS-17265 Builds disappearing from history

The shell scripts don't work when the jobs have spaces in the names. What modifications need to be made to the scripts to adapt it to be able to work when there are spaces in the job names?

Additionally, running this script in Cygwin on Windows is generating output that indicates EVERY build has a missing build link.

[josesa@java.net (JIRA)]

Jose Sa updated JENKINS-17265 Builds disappearing from history Change By: Jose Sa (03/Apr/13 10:29 PM) Attachment: check_invalid_builds.sh Attachment: check_missing_builds.sh

[josesa@java.net (JIRA)]

Jose Sa commented on JENKINS-17265 Builds disappearing from history Allowing jobs with spaces in names does complicate things, making it harder (but not impossible) to handle via script. I prefer not to allow any spaces using "Create Job Advanced" plugin to enforce some common rules. To allow spaces, basically any path that contains the job name should be enclosed with double quotes either on variable assignment as well in usage. Since command "find" doesn't like to have wildcard enclosed in double quotes, that can be split into first a change directory and then the find command with wildcard. I uploaded new versions of scripts but it didn't seem to have replaced the old ones when uploaded, so refer to the most recent ones. Jenkins on Windows also seems to have a different behavior since it doesn't seem to use any symbolic links (not visible on cygwin anyway with 1.484), so the the link related checks probably are not applicable on Windows.

[westhusing@gmail.com (JIRA)]

Adam Westhusing commented on JENKINS-17265 Builds disappearing from history Thanks for updating the script, but I suppose the behavior is different on Windows. Jenkins has never created symbolic links on Windows until upgrading from 1.506 to 1.509. In the logs, Jenkins is complaining all over the place that it can't find directories but the build directories are actually there and nothing appears to be wrong when bringing up the specific build being complained about in the Jenkins logs.

[josesa@java.net (JIRA)]

Jose Sa commented on JENKINS-17265 Builds disappearing from history You can remove from the script the link sections in Windows environment. But probably in some past upgrade it may have been possible that this same behaviour was accidentally induced in Unix environments which could explain why so many situations similar to that. Still after running the cleanup and running for a few days I don't the problem happening so often now. Actually i saw it happening right in the the first day, but somehow it "recovered" without doing a reload from disk or restart.

[reece.g.johnston@boeing.com (JIRA)]

Reece Johnston commented on JENKINS-17265 Builds disappearing from history I just saw the same issue on version 1.511. It occurred after renaming a job and running that job with a scheduled build. Oddly, it did not occur with the first few manual builds.

[martin.danjou14@gmail.com (JIRA)]

Martin d'Anjou commented on JENKINS-17265 Builds disappearing from history Seeing this problem coming back on LTS 1.509.2. I have no idea how to debug...

[eder_stefan@gmx.at (JIRA)]

Stefan Eder commented on JENKINS-17265 Builds disappearing from history Same problem with 1.509.1 LTS. We are using two Jenkins instances with the same version right now, and this issue seems to affect only the one where the builds are executed on slaves only. Additionally only builds which where renamed and are triggered automatically by croned upstream projects are having this issue. The only thing I noticed in the logs was the following: Aug 01, 2013 12:44:28 AM hudson.model.Run execute INFO: some-build #15 main build action completed: SUCCESS Aug 01, 2013 12:21:20 AM hudson.model.listeners.RunListener report WARNING: RunListener failed java.lang.NullPointerException This happened right before/during every build which "lost" builds. However, this does not occur on the Jenkins instance which builds on master only. Unfortunately there was no trace attached to the NullPointerException in the log, but I'll try to get one with a higher log output setting. Maybe it's connected to issue JENKINS-18823.

[alex.ouzounis@me.com (JIRA)]

Alex Ouzounis commented on JENKINS-17265 Builds disappearing from history Hi all, I am seeing the same problem occasionally and either restarting or reloading from disk fixes the problem. I have a feeling that its related with: "WARNING: Multiple builds have the same number: " because every time I delete conflicting builds it goes away.

[alex.ouzounis@me.com (JIRA)]

Alex Ouzounis commented on JENKINS-17265 Builds disappearing from history

Problem no longer present in the latest versions ie 1.570

[eguess74@java.net (JIRA)]

eguess74 commented on JENKINS-17265 Builds disappearing from history I'm seeing it happening right now on Jenkins 1.578 I had a build that i triggered using gerrit trigger (query and trigger gerrit patches) It appeared in the GUI, I saw it running in the the build queue, it sent an email about build being fixed, registered the result with gerrit, but no matter how i'm trying to access it the link to the build number leads to 404. Build is present on the file system as well.

[eguess74@java.net (JIRA)]

eguess74 reopened JENKINS-17265 Builds disappearing from history Change By: eguess74 (02/Dec/14 12:20 AM) Resolution: Duplicate Status: Resolved Reopened

[jglick@cloudbees.com (JIRA)]

Jesse Glick resolved JENKINS-17265 as Incomplete Builds disappearing from history eguess74 if you are using the Gerrit Trigger plugin, you are probably seeing JENKINS-23152. Change By: Jesse Glick (02/Dec/14 1:05 AM) Status: Reopened Resolved Resolution: Incomplete

[eguess74@java.net (JIRA)]

eguess74 commented on JENKINS-17265 Builds disappearing from history Jesse, Thanks for pointing me to the right direction!

[seaware@yahoo.com (JIRA)]

Frank Canova commented on JENKINS-17265 Builds disappearing from history I'm seeing this same issue when changing between v1.590 and v1.610. Is there a work-around or way to recreate the build.xml files? The history directories are intact but history has disappeared from the web view.

[daniel@beckweb.net (JIRA)]

Daniel Beck commented on JENKINS-17265 Builds disappearing from history

I'm seeing this same issue when changing between v1.590 and v1.610. Is there a work-around or way to recreate the build.xml files? The history directories are intact but history has disappeared from the web view.

Unrelated issue, almost sure this is the storage layout change from http://jenkins-ci.org/changelog#v1.597 – Jenkins provides information how to downgrade the storage format in its logs, and on the UI of 1.597+. As there are 50+ watchers here, and your issue has nothing to do with this one, please move any follow-up discussion to the users mailing list or IRC.