[JIRA] [scm-sync-configuration-plugin] (JENKINS-33348) Deleting job under git failed, leaving a poisened commit queue

27 views
Skip to first unread message

douglas.royds@gmx.com (JIRA)

unread,
Mar 6, 2016, 6:08:03 PM3/6/16
to jenkinsc...@googlegroups.com
Douglas Royds created an issue
 
Jenkins / Bug JENKINS-33348
Deleting job under git failed, leaving a poisened commit queue
Issue Type: Bug Bug
Assignee: Frédéric Camblor
Components: scm-sync-configuration-plugin
Created: 06/Mar/16 11:07 PM
Environment: Jenkins 1.625.3 on Debian Wheezy
scm-sync-configuration 0.0.9
git 1.9.1
Priority: Minor Minor
Reporter: Douglas Royds

After renaming a job, we evidently wound up with a poisoned queue. scm-sync-configuration's attempts to delete the old job were failing:

[INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration/jobs' && 'git' 'rm' '-r' '--' 'reset-watchdog-test'
[INFO] Working directory: /var/lib/jenkins/scm-sync-configuration/checkoutConfiguration/jobs
Mar 07, 2016 11:09:27 AM hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy
SEVERE: [deleteHierarchy] Problem during remove : The git command failed.

Status in ~/scm-sync-configuration/checkoutConfiguration was:

$ git status
# On branch master
# Your branch is up-to-date with 'origin/master'.
#
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	new file:   jobs/reset-watchdog-test/config.xml
#

The following actions failed:

  1. Manually unstaged the offending (old) config.xml file and its directory
  2. Deleted the local jobs/reset-watchdog-test directory and config.xml file
  3. Committed and pushed.
  4. Made a trivial edit in an unrelated job, and committed it (via the pop-up in the web-app)
  • scm-sync-configuration reported an error
  • The (old) config.xml file had reappeared, and was staged.

Another approach also failed:

  1. Manually committed the (staged) config.xml file
  2. Made a trivial edit in an unrelated job, and committed it (via the pop-up in the web-app)
  • scm-sync-configuration reported an error

Examining the tomcat log, scm-sync-configuration was trying to run the following command:

[INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration' && 'git' 'add' '--' 'jobs'
[INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration' && 'git' 'add' '--' 'jobs/reset-watchdog-test'
[INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration' && 'git' 'add' '--' 'jobs/reset-watchdog-test/config.xml'
[INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration' && 'git' 'rev-parse' '--show-toplevel'
[INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration' && 'git' 'status' '--porcelain' '.'
[INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration/jobs' && 'git' 'rm' '-r' '--' 'reset-watchdog-test'

When I ran the git rm -r command myself from the command-line, with the config.xml file staged, git failed:

error: the following file has changes staged in the index:
    jobs/reset-watchdog-test/config.xml
(use --cached to keep the file, or -f to force removal)

The workaround that avoided having to restart Jenkins just to un-poison the queue (this is a very busy little box) was to manually commit the rogue config.xml file, so that the git rm -r could work. With the file already staged:

$ git commit jobs/reset-watchdog-test/config.xml -m"Fake commit"
[master d348b30] Fake commit
 1 file changed, 103 insertions(+)
 create mode 100644 jobs/reset-watchdog-test/config.xml

I returned to the web-app and made a trivial commit, and the problem was gone. scm-sync-configuration cheerfully deleted the no-longer-wanted job, and pushed it. The plugin's successful commit immediately followed by fake commit in the git log.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

douglas.royds@gmx.com (JIRA)

unread,
Mar 6, 2016, 6:15:02 PM3/6/16
to jenkinsc...@googlegroups.com
Douglas Royds updated an issue
Change By: Douglas Royds
After renaming a job, we evidently wound up with a poisoned queue. scm-sync-configuration's attempts to delete the old job were failing:

{noformat}

[INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration/jobs' && 'git' 'rm' '-r' '--' 'reset-watchdog-test'
[INFO] Working directory: /var/lib/jenkins/scm-sync-configuration/checkoutConfiguration/jobs
Mar 07, 2016 11:09:27 AM hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy
SEVERE: [deleteHierarchy] Problem during remove : The git command failed.
{noformat}

Status in ~/scm-sync-configuration/checkoutConfiguration was:
{noformat}

$ git status
# On branch master
# Your branch is up-to-date with 'origin/master'.
#
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
# new file:   jobs/reset-watchdog-test/config.xml
#
{noformat}

The following actions failed:

# Manually unstaged the offending (old) config.xml file and its directory
# Deleted the local jobs/reset-watchdog-test directory and config.xml file
# Committed and pushed.
# Made a trivial edit in an unrelated job, and committed it (via the pop-up in the web-app)
* scm-sync-configuration reported an error
* The (old) config.xml file had reappeared, and was staged.

Another approach also failed:

# Manually committed the (staged) config.xml file
# Made a trivial edit in an unrelated job, and committed it (via the pop-up in the web-app)
* scm-sync-configuration reported an error


Examining the tomcat log, scm-sync-configuration was trying to run the following command:

{noformat}

[INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration' && 'git' 'add' '--' 'jobs'
[INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration' && 'git' 'add' '--' 'jobs/reset-watchdog-test'
[INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration' && 'git' 'add' '--' 'jobs/reset-watchdog-test/config.xml'
[INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration' && 'git' 'rev-parse' '--show-toplevel'
[INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration' && 'git' 'status' '--porcelain' '.'
[INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration/jobs' && 'git' 'rm' '-r' '--' 'reset-watchdog-test'
{noformat}


When I ran the git rm -r command myself from the command-line, with the config.xml file staged, git failed:

{noformat}

error: the following file has changes staged in the index:
    jobs/reset-watchdog-test/config.xml
(use --cached to keep the file, or -f to force removal)
{noformat}


The workaround that avoided having to restart Jenkins just to un-poison the queue (this is a very busy little box) was to manually commit the rogue config.xml file, so that the git rm -r could work. With the file already staged:

{noformat}

$ git commit jobs/reset-watchdog-test/config.xml -m"Fake commit"
[master d348b30] Fake commit
 1 file changed, 103 insertions(+)
 create mode 100644 jobs/reset-watchdog-test/config.xml
{noformat}


I returned to the web-app and made a trivial commit, and the problem was gone. scm-sync-configuration cheerfully deleted the no-longer-wanted job, and pushed it. The plugin's successful commit immediately followed by fake commit in the git log.


Annoyingly, I just failed to reproduce this by creating a throw-away job and deleting it.

I originally added this as a comment on 

douglas.royds@gmx.com (JIRA)

unread,
Mar 6, 2016, 6:16:03 PM3/6/16
to jenkinsc...@googlegroups.com
I originally added this as a comment on    https://issues.jenkins-ci.org/browse/JENKINS-25786 in error - that issue was about a rename, in this case we just deleted.

douglas.royds@gmx.com (JIRA)

unread,
Mar 6, 2016, 6:17:02 PM3/6/16
to jenkinsc...@googlegroups.com
I originally added this as a comment on  [JENKINS-25786|  https://issues.jenkins-ci.org/browse/JENKINS-25786 ]  in error - that issue was about a rename, in this case we just deleted.

douglas.royds@gmx.com (JIRA)

unread,
Mar 6, 2016, 6:37:01 PM3/6/16
to jenkinsc...@googlegroups.com
Douglas Royds closed an issue as Cannot Reproduce
 

Closing as I can't reproduce it.

Change By: Douglas Royds
Status: Open Closed
Resolution: Cannot Reproduce

kmitov@axlessoft.com (JIRA)

unread,
Nov 14, 2019, 9:32:06 AM11/14/19
to jenkinsc...@googlegroups.com
Kiril Mitov commented on Bug JENKINS-33348
 
Re: Deleting job under git failed, leaving a poisened commit queue

like to reopen it.

 

I currently have the following log

 

````

[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

Nov 14, 2019 4:19:05 PM SEVERE hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy


[deleteHierarchy] Problem during remove : The git command failed.

````

The last commit in the repo is

````

Author: ....
Date: Tue Oct 29 14:43:06 2019 +0200

`````

And today we are 14 November

In the same time th SCM status says SCM Sync status :  Last operation @ Thu Nov 14 16:18:52 EET 2019

which is incorrect and sometimes there is an error in the SCM status that says

````

SCM Sync status :  <br/>Thu Nov 14 16:30:27 EET 2019 : Delete jobs/is docker Build<br/>Thu Nov 14 16:30:29 EET 2019 : Delete jobs/is docker Build<br/><br/>To remove this message, please <a href='#' onclick='javascript:removeLog();'>click here</a>
````

So basically I can not use the SCM, I can not push to it I can not commit in it. Every change I make in jenkins is not in any way synchronized with SCM.

 

Wha

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

kmitov@axlessoft.com (JIRA)

unread,
Nov 14, 2019, 9:50:03 AM11/14/19
to jenkinsc...@googlegroups.com
Kiril Mitov edited a comment on Bug JENKINS-33348
I would like to reopen it.


 

I currently have the following log
:

 

````

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.
{color:#aaaaaa}Nov 14, 2019 4:19:05 PM{color} {color:#aaaaaa}SEVERE{color} {color:#aaaaaa}hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy{color}

[deleteHierarchy] Problem during remove : The git command failed.

````

The last commit in the repo is

````

Author: ....
Date: Tue Oct 29 14:43:06 2019 +0200

`````

And today we are 14 November

In the same time th SCM status says {color:#808080}SCM Sync status : {color}!https://sf.robopartans.com/jenkins/static/770dab3e/images/16x16/health-80plus.png|width=16,height=16,align=right!{color:#808080} Last operation @ Thu Nov 14 16:18:52 EET 2019{color}


which is incorrect and sometimes there is an error in the SCM status that says

````

{color:#808080}SCM Sync status : {color}!https://sf.robopartans.com/jenkins/static/770dab3e/images/16x16/error.png|width=16,height=16,align=right!{color:#808080} <br/>Thu Nov 14 16:30:27 EET 2019 : Delete jobs/is docker Build<br/>Thu Nov 14 16:30:29 EET 2019 : Delete jobs/is docker Build<br/><br/>To remove this message, please <a href='#' onclick='javascript:removeLog();'>click here</a>{color}

````

So basically I can not use the SCM, I can not push to it I can not commit in it. Every change I make in jenkins is not in any way synchronized with SCM.

 

Wha What else could I provide. This has happened about two weeks ago so I do not have a clear understanding how. But I have deleted a job.

kmitov@axlessoft.com (JIRA)

unread,
Nov 14, 2019, 9:56:03 AM11/14/19
to jenkinsc...@googlegroups.com

Tried installing and uninstalling the SCM Configuration plugin. The following status "SCM Sync status :  <br/>Thu Nov 14 16:43:51 EET 2019 : Delete jobs/is docker Build<br/><br/>To remove this message, please <a href='#' onclick='javascript:removeLog();'>click here</a>" is still shown. Can not sync to scm configuration.

kmitov@axlessoft.com (JIRA)

unread,
Nov 14, 2019, 9:59:02 AM11/14/19
to jenkinsc...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages