[JIRA] (JENKINS-60717) Jobs with a wildcard tag refspec sometimes rebuild tags at same commit

[piersoned6@gmail.com (JIRA)]

Eric Pierson created an issue

Jenkins / JENKINS-60717 Jobs with a wildcard tag refspec sometimes rebuild tags at same commit Issue Type: Bug Assignee: nkns165 Components: discard-old-build-plugin Created: 2020-01-09 21:02 Environment: Jenkins 2.176.3 on traditional VMs (not containers) GitHub Plugin 1.29.5 Git Plugin 4.0.0 Priority: Minor Reporter: Eric Pierson Steps to reproduce:   1. Add a webhook to a GitHub repo which sends the following events to Jenkins:  Pushes, Branch or tag creation, Releases.   2. Create two "Pipeline" jobs in Jenkins, both referencing the same GitHub repo.  The jobs execute a Declarative Pipeline Jenkinsfile script.   NOTE: JJB is used to create the jobs (jobs are not copied).   Settings for job 1 (master branch job):     Do not allow concurrent builds     GitHub hook trigger for GITScm polling     Pipeline script from SCM: Git     Repo ID:  origin     Refspec:  +refs/heads/:refs/remotes/origin/     Branch Specifier: refs/heads/master     Additional Behaviors:  Wipe out repository & force clone     Lightweight checkout:  false   Settings for job 2 (tag job):       Do not allow concurrent builds     GitHub hook trigger for GITScm polling     Pipeline script from SCM: Git     Repo ID:  origin     Refspec:  +refs/tags/:refs/remotes/origin/tags/     Branch Specifier: /tags/     Additional Behaviors:  Wipe out repository & force clone     Lightweight checkout:  false   3.  When commits are merged to the master branch of the repo, the webhook fires and Job 1 executes.   4.  When new releases (tags) are published, the webhook fires and Job 2 executes for the most recent (new) tag.      (Things have been running smoothly for me with this type of setup for a couple years.)   5.  Occasionally there is a situation with jobs configured like job 2, specifically: New commits are pushed to master which triggers the webhook to Jenkins. Job 1 polls and detects the new commits to master and runs. Job 2 polls and detects changes even though no tags have been created and previously built tags still reference the same commit. Job 2 then runs and rebuilds the last tag it had already built successfully. The polling log shows last built revision is the same commit and tag, no diff is performed, but changes are found. The changes page for the job shows no changes.    Observations when this occurs: Let's say the tag has been built 4 times.  We will have a polling log and change logs for the first job run that initially built the tag, then jobs 2 and 3 will have no polling log on the filesystem but will have an empty changelog0.xml, then the last build of the tag has a polling log and change log files. When a tag is being re-built, when you look at Git Build Data for the job rebuilding that tag, all prior job runs that built that tag successfully are missing from the history. The problem does not always happen in my environment.  It can happen for a repo one day then not happen the next.  Perhaps I am simply overlooking some activity in the repo that results in polling detecting changes with my wildcard tag specifier. Workspaces appear to be intact as required for the git plugin to perform a wildcard tag poll (post-clone).  There are no indications of workspaces needing to be created as job runs start. The same build node (same workspace) can be used for all job runs and the issue can still occur.     Example polling log and Git Build Data for a re-run:   Started on Dec 30, 2019 7:54:51 AM Started by event from 10.25.59.190 ⇒ https://xxxx/github-webhook/ on Mon Dec 30 07:54:51 EST 2019 Using strategy: Default [poll] Last Built Revision: Revision f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, refs/tags/v2.3.0) using credential 28d104ae-ad72-401a-8dc2-d72cf4b8e913  > /data/git-client/bin/git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repositories  > /data/git-client/bin/git config remote.origin.url https://github.com/repo.git # timeout=10 Fetching upstream changes from https://github.com/repo.git  > /data/git-client/bin/git --version # timeout=10 using GIT_ASKPASS to set credentials xxxx Service Account  > /data/git-client/bin/git fetch --tags --force --progress -- https://github.com/repo.git +refs/tags/:refs/remotes/origin/tags/ # timeout=10 Polling for changes in Seen branch in repository origin/2.2.0 Seen branch in repository origin/2.3.0 Seen branch in repository origin/2.4.0 Seen branch in repository origin/gh-pages Seen branch in repository origin/master Seen branch in repository origin/openshift-migration Seen branch in repository origin/rhel-build Seen branch in repository origin/tags/v0.10.0 Seen branch in repository origin/tags/v0.3 Seen branch in repository origin/tags/v0.4 Seen branch in repository origin/tags/v0.7.0 Seen branch in repository origin/tags/v0.7.1 Seen branch in repository origin/tags/v0.8.0 Seen branch in repository origin/tags/v0.9.0 Seen branch in repository origin/tags/v0.9.1 Seen branch in repository origin/tags/v1.0.0 Seen branch in repository origin/tags/v1.0.0-secure Seen branch in repository origin/tags/v1.1.0 Seen branch in repository origin/tags/v1.2.0 Seen branch in repository origin/tags/v1.3.0 Seen branch in repository origin/tags/v1.3.1 Seen branch in repository origin/tags/v1.4.0 Seen branch in repository origin/tags/v1.5.0 Seen branch in repository origin/tags/v1.6.0 Seen branch in repository origin/tags/v1.6.1 Seen branch in repository origin/tags/v1.7.0 Seen branch in repository origin/tags/v1.7.1 Seen branch in repository origin/tags/v1.7.2 Seen branch in repository origin/tags/v1.8.0 Seen branch in repository origin/tags/v1.9.0 Seen branch in repository origin/tags/v2.0.0 Seen branch in repository origin/tags/v2.1.0 Seen branch in repository origin/tags/v2.2.0 Seen branch in repository origin/tags/v2.3.0 Seen 34 remote branches  > /data/git-client/bin/git show-ref --tags -d # timeout=10 Using strategy: Default [poll] Last Built Revision: Revision f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, refs/tags/v2.3.0) Done. Took 0.53 sec Changes found   ---------------------------------   Git Build Data (for Build #9) Revision: 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc origin/tags/v2.4.0 refs/tags/v2.4.0 Built Branches refs/tags/v2.1.0: Build #4 of Revision 42d67f58dc99a77c91eca90132a065ebca4f5c66 (origin/tags/v2.1.0, refs/tags/v2.1.0) origin/tags/v2.2.0: Build #5 of Revision 2f0b66b9cf8b6f9852a461c31f435196c9270f71 (origin/tags/v2.2.0, refs/tags/v2.2.0) refs/tags/v2.2.0: Build #5 of Revision 2f0b66b9cf8b6f9852a461c31f435196c9270f71 (origin/tags/v2.2.0, refs/tags/v2.2.0) origin/tags/v2.1.0: Build #4 of Revision 42d67f58dc99a77c91eca90132a065ebca4f5c66 (origin/tags/v2.1.0, refs/tags/v2.1.0) origin/tags/v2.0.0: Build #2 of Revision 11f93f5e9eb5fa87745f4db062b6ee48f4975350 (origin/tags/v2.0.0, refs/tags/v2.0.0) refs/tags/v2.4.0: Build #9 of Revision 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc (origin/tags/v2.4.0, refs/tags/v2.4.0) refs/tags/v2.3.0: Build #8 of Revision f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, refs/tags/v2.3.0) origin/tags/v1.0.0-secure: Build #1 of Revision 3e54ce51fbc67ea8f3f59d3a2389fe0aca2ba15d (origin/tags/v1.0.0-secure, refs/tags/v1.0.0-secure) origin/tags/v2.4.0: Build #9 of Revision 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc (origin/tags/v2.4.0, refs/tags/v2.4.0) refs/tags/v1.0.0-secure: Build #1 of Revision 3e54ce51fbc67ea8f3f59d3a2389fe0aca2ba15d (origin/tags/v1.0.0-secure, refs/tags/v1.0.0-secure) refs/tags/v2.0.0: Build #2 of Revision 11f93f5e9eb5fa87745f4db062b6ee48f4975350 (origin/tags/v2.0.0, refs/tags/v2.0.0) origin/tags/v2.3.0: Build #8 of Revision f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, refs/tags/v2.3.0)   NOTE:   Build #6 was the initial build of refs/tags/v2.3.0 Build #7 was the second rerun of refs/tags/v2.3.0 Build #8 was the last rerun of refs/tags/v2.3.0 Build #9 was for refs/tags/v2.4.0       I have not tried to run any scripts to prune history as I see described in JENKINS-19022.   I do have "Discard old builds" enabled with Strategy "Log Rotation" and "Max # of builds to keep" = 10   I am looking closer at "Discard old builds" settings and build.xml content, and what may be happening to the build history when a build completes.  I have one job that has only run a total of 7 times that had the issue on builds #3 and #4, and the example above has only run a total of 12 times and we see the issue on multiple builds there, too.  So while there may be many tags in the repo, the build history is not "large" yet.   I have another Jenkins environment I used for a while where I did not have this issue.  It had the same settings (including "Discard old builds"), but when I was running jobs through this environment on a regular basis, Jenkins and plugins would have been at older versions.     To hone in on git plugin versions just in the change they are relevant, I compared my build.xml:   New Environment <hudson.plugins.git.util.BuildData plugin="g...@4.0.0"> <marked plugin="git-c...@3.0.0"> <revision plugin="git-c...@3.0.0">   Old Environment <hudson.plugins.git.util.BuildData plugin="g...@3.9.3"> <marked plugin="git-c...@2.7.7"> <revision plugin="git-c...@2.7.7">   The issue does not occur for all builds and I am still troubleshooting.   Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)