Jobs with a wildcard tag refspec sometimes rebuild tags at same commit

[Eric Pierson]

I am troubleshooting a scenario where jobs with a wildcard tag refspec (+refs/tags/*:refs/remotes/origin/tags/*) sometimes detect changes and rebuild a successfully built tag referencing the same commit.  

When a rerun occurs, prior job runs for the tag are no longer present in the Git Build History for the job.

I am seeking help to further investigate and trace polling activity to try to determine what changes are being detected that trigger the job to run again, or learn if i should submit this as a bug report for assistance instead.

Thanks, everyone!

---------------

Existing Bugs Found:

This was the most relative bug I could find, yet my situation is different.

https://issues.jenkins-ci.org/browse/JENKINS-17614?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&showAll=true

---------------

My Environment:

Jenkins 2.176.3 on traditional VMs (not containers)

GitHub Plugin 1.29.5

Git Plugin 4.0.0

Steps to reproduce:

1. Add a webhook to a GitHub repo which sends the following events to Jenkins:  Pushes, Branch or tag creation, Releases.

2. Create two “Pipeline” jobs in Jenkins, both referencing the same GitHub repo.  The jobs execute a Declarative Pipeline Jenkinsfile script.

NOTE: JJB is used to create the jobs (jobs are not copied).

 

Settings for job 1 (master branch job):

    Do not allow concurrent builds

    GitHub hook trigger for GITScm polling

    Pipeline script from SCM: Git

    Repo ID:  origin

    Refspec:  +refs/heads/*:refs/remotes/origin/*

    Branch Specifier: refs/heads/master

    Additional Behaviors:  Wipe out repository & force clone

    Lightweight checkout:  false

 

Settings for job 2 (tag job):  

    Do not allow concurrent builds

    GitHub hook trigger for GITScm polling

    Pipeline script from SCM: Git

    Repo ID:  origin

    Refspec:  +refs/tags/*:refs/remotes/origin/tags/*

    Branch Specifier: */tags/*

    Additional Behaviors:  Wipe out repository & force clone

    Lightweight checkout:  false

 

3.  When commits are merged to the master branch of the repo, the webhook fires and Job 1 executes.

4.  When new releases (tags) are published, the webhook fires and Job 2 executes for the most recent (new) tag.

    

(Things have been running smoothly for me with this type of setup for a couple years.)

5.  Occasionally there is a situation with jobs configured like job 2, specifically:

• New commits are pushed to master which triggers the webhook to Jenkins.
• Job 1 polls and detects the new commits to master and runs.
• Job 2 polls and detects changes even though no tags have been created and previously built tags still reference the same commit.
• Job 2 then runs and rebuilds the last tag it had already built successfully.
• The polling log shows last built revision is the same commit and tag, no diff is performed, but changes are found.
• The changes page for the job shows no changes.

 Observations when this occurs:

• Let's say the tag has been built 4 times.  We will have a polling log and change logs for the first job run that initially built the tag, then jobs 2 and 3 will have no polling log on the filesystem but will have an empty changelog0.xml, then the last build of the tag has a polling log and change log files.
  
• When a tag is being re-built, when you look at Git Build Data for the job rebuilding that tag, all prior job runs that built that tag successfully are missing from the history.
• The problem does not always happen in my environment.  It can happen for a repo one day then not happen the next.  Perhaps I am simply overlooking some activity in the repo that results in polling detecting changes with my wildcard tag specifier.
• Workspaces appear to be intact as required for the git plugin to perform a wildcard tag poll (post-clone).  There are no indications of workspaces needing to be created as job runs start.
• The same build node (same workspace) can be used for all job runs and the issue can still occur.

Example polling log and Git Build Data for a re-run:

Started on Dec 30, 2019 7:54:51 AM

Started by event from 10.25.59.190 ⇒ https://xxxx/github-webhook/ on Mon Dec 30 07:54:51 EST 2019

Using strategy: Default

[poll] Last Built Revision: Revision f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, refs/tags/v2.3.0)

using credential 28d104ae-ad72-401a-8dc2-d72cf4b8e913

 > /data/git-client/bin/git rev-parse --is-inside-work-tree # timeout=10

Fetching changes from the remote Git repositories

 > /data/git-client/bin/git config remote.origin.url https://github.com/repo.git # timeout=10

Fetching upstream changes from https://github.com/repo.git

 > /data/git-client/bin/git --version # timeout=10

using GIT_ASKPASS to set credentials xxxx Service Account

 > /data/git-client/bin/git fetch --tags --force --progress -- https://github.com/repo.git +refs/tags/*:refs/remotes/origin/tags/* # timeout=10

Polling for changes in

Seen branch in repository origin/2.2.0

Seen branch in repository origin/2.3.0

Seen branch in repository origin/2.4.0

Seen branch in repository origin/gh-pages

Seen branch in repository origin/master

Seen branch in repository origin/openshift-migration

Seen branch in repository origin/rhel-build

Seen branch in repository origin/tags/v0.10.0

Seen branch in repository origin/tags/v0.3

Seen branch in repository origin/tags/v0.4

Seen branch in repository origin/tags/v0.7.0

Seen branch in repository origin/tags/v0.7.1

Seen branch in repository origin/tags/v0.8.0

Seen branch in repository origin/tags/v0.9.0

Seen branch in repository origin/tags/v0.9.1

Seen branch in repository origin/tags/v1.0.0

Seen branch in repository origin/tags/v1.0.0-secure

Seen branch in repository origin/tags/v1.1.0

Seen branch in repository origin/tags/v1.2.0

Seen branch in repository origin/tags/v1.3.0

Seen branch in repository origin/tags/v1.3.1

Seen branch in repository origin/tags/v1.4.0

Seen branch in repository origin/tags/v1.5.0

Seen branch in repository origin/tags/v1.6.0

Seen branch in repository origin/tags/v1.6.1

Seen branch in repository origin/tags/v1.7.0

Seen branch in repository origin/tags/v1.7.1

Seen branch in repository origin/tags/v1.7.2

Seen branch in repository origin/tags/v1.8.0

Seen branch in repository origin/tags/v1.9.0

Seen branch in repository origin/tags/v2.0.0

Seen branch in repository origin/tags/v2.1.0

Seen branch in repository origin/tags/v2.2.0

Seen branch in repository origin/tags/v2.3.0

Seen 34 remote branches

 > /data/git-client/bin/git show-ref --tags -d # timeout=10

Using strategy: Default

[poll] Last Built Revision: Revision f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, refs/tags/v2.3.0)

Done. Took 0.53 sec

Changes found

---------------------------------

Git Build Data (for Build #9)

Revision: 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc

origin/tags/v2.4.0

refs/tags/v2.4.0

Built Branches

refs/tags/v2.1.0: Build #4 of Revision 42d67f58dc99a77c91eca90132a065ebca4f5c66 (origin/tags/v2.1.0, refs/tags/v2.1.0)

origin/tags/v2.2.0: Build #5 of Revision 2f0b66b9cf8b6f9852a461c31f435196c9270f71 (origin/tags/v2.2.0, refs/tags/v2.2.0)

refs/tags/v2.2.0: Build #5 of Revision 2f0b66b9cf8b6f9852a461c31f435196c9270f71 (origin/tags/v2.2.0, refs/tags/v2.2.0)

origin/tags/v2.1.0: Build #4 of Revision 42d67f58dc99a77c91eca90132a065ebca4f5c66 (origin/tags/v2.1.0, refs/tags/v2.1.0)

origin/tags/v2.0.0: Build #2 of Revision 11f93f5e9eb5fa87745f4db062b6ee48f4975350 (origin/tags/v2.0.0, refs/tags/v2.0.0)

refs/tags/v2.4.0: Build #9 of Revision 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc (origin/tags/v2.4.0, refs/tags/v2.4.0)

refs/tags/v2.3.0: Build #8 of Revision f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, refs/tags/v2.3.0)

origin/tags/v1.0.0-secure: Build #1 of Revision 3e54ce51fbc67ea8f3f59d3a2389fe0aca2ba15d (origin/tags/v1.0.0-secure, refs/tags/v1.0.0-secure)

origin/tags/v2.4.0: Build #9 of Revision 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc (origin/tags/v2.4.0, refs/tags/v2.4.0)

refs/tags/v1.0.0-secure: Build #1 of Revision 3e54ce51fbc67ea8f3f59d3a2389fe0aca2ba15d (origin/tags/v1.0.0-secure, refs/tags/v1.0.0-secure)

refs/tags/v2.0.0: Build #2 of Revision 11f93f5e9eb5fa87745f4db062b6ee48f4975350 (origin/tags/v2.0.0, refs/tags/v2.0.0)

origin/tags/v2.3.0: Build #8 of Revision f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, refs/tags/v2.3.0)

NOTE:  

Build #6 was the initial build of refs/tags/v2.3.0

Build #7 was the second rerun of refs/tags/v2.3.0

Build #8 was the last rerun of refs/tags/v2.3.0

Build #9 was for refs/tags/v2.4.0

[Mark Waite]

On Fri, Jan 3, 2020 at 2:51 PM Eric Pierson <piers...@gmail.com> wrote:

I am troubleshooting a scenario where jobs with a wildcard tag refspec (+refs/tags/*:refs/remotes/origin/tags/*) sometimes detect changes and rebuild a successfully built tag referencing the same commit.  

When a rerun occurs, prior job runs for the tag are no longer present in the Git Build History for the job.

That sounds to me as though you are relying on the JENKINS-19022 memory bloat bug to retain history of more jobs in the BuildData than are actually in the build history.  

If someone executed one of the scripts that is mentioned in JENKINS-19022 to clean the excess BuildData from the history, that seems like it might cause the SHA-1's referenced by some of the tags to be removed from the history.

If build history were being removed based on a specific number of builds, then it also could be that the history entry which included that tag was removed from the list, and thus was no longer visible.

Mark Waite

 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b114b34b-3f60-4dc5-87cd-072bcfb81f8c%40googlegroups.com.

--

Thanks!

Mark Waite

[Eric Pierson]

Thank you for your response, Mark!  

I have not tried to run any scripts to prune history as I see described in JENKINS-19022.  There are a lot of comments in this one and I am still re-reading them to be sure I am not missing something else relevant folks discovered along the way.

I do have "Discard old builds" enabled with Strategy "Log Rotation" and "Max # of builds to keep" = 10.  Forgive me for omitting the fact that I use Discard old builds initially.   

Reading more about  "Discard old builds":

• Build count: discard the oldest build when a certain number of builds already exist  
• Note that Jenkins does not discard items immediately when this configuration is updated, or as soon as any of the configured values are exceeded; these rules are evaluated each time a build of this project completes.  

I am now focusing on the "Discard old builds" settings and build.xml content, and what may be happening to the build history when a build completes.  I have one job that has only run a total of 7 times that had the issue on builds #3 and #4, and the example I initially provided has only run a total of 12 times and we see the issue on multiple builds there, too.  So while there are indeed many tags in the repo, the build history is not "large" yet.

I have another Jenkins environment I used for a while where I did not have this issue.  It had the same settings (including "Discard old builds"), but when I was running jobs through this environment on a regular basis, Jenkins and plugins would have been at older versions.  

To hone in on git plugin versions just in the change they are relevant, I compared my build.xml:

New Environment

<hudson.plugins.git.util.BuildData plugin="g...@4.0.0">

<marked plugin="git-c...@3.0.0">

<revision plugin="git-c...@3.0.0">

Old Environment

<hudson.plugins.git.util.BuildData plugin="g...@3.9.3">

<marked plugin="git-c...@2.7.7">

<revision plugin="git-c...@2.7.7">

 ---------------

Given this additional information, please let me know if you have any new thoughts or if you feel I should file a bug report (for perhaps the logRotate or BuildDiscarder area?) as I continue to research and test.

Thank you again for your help and everything you do for the Jenkins community!

-Eric 

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/_JJM5DZoPvA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtGFkmt76_HEU4h_Aey%2Bai8UiD3hYkwNE%3Da3ZaF70SK-JA%40mail.gmail.com.

[Mark Waite]

Discard old builds is very likely the problem.  If a tag build is removed from the history by "Discard old builds" then future searches for the that tag (or the SHA-1 of the commit associated with the tag) will fail and it will build again.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAKh5vxkMvHNW%2B8JpOUPFj%2BqRHgf%2Bi-6%3DR-R0Ppmam4XjDPXO-A%40mail.gmail.com.

--

Thanks!

Mark Waite

[Beushausen, Christian]

Hi Eric, Mark,

 

Could you keep the list updated? Or have you created a new issue on Jira for this?
We do see this (or a very similar) issue also for at least one job in our environment

 

Thank you and cheers.

 

Mit freundlichen Gruessen/Best regards,

Christian Beushausen
I S&T PD SW SWF
Interior Systems & Technology

E-Mail: Christian....@continental-corporation.com

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtGNKPNDDfBNKwbN22%2BVK18VFSmT6%2BWYD1syv95Ucs98vg%40mail.gmail.com.

[Eric Pierson]

Yes, I intend to write up the issue today for tracking.

I have been troubleshooting more but unfortunately don't have anything valuable to add just yet. I will reply to this thread once the issue has been opened with more details.

Eric

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/AM0PR03MB571566E46420E5508F3069F7CF390%40AM0PR03MB5715.eurprd03.prod.outlook.com.

[Eric Pierson]

Issue entered and will update this thread as things develop.

https://issues.jenkins-ci.org/browse/JENKINS-60717

Thanks,

Eric