[JIRA] [workflow-plugin] (JENKINS-34313) Exceptions by parallel SCM checkout

Exceptions by parallel SCM checkout

Change By:	Jesse Glick
Component/s:	subversion-plugin
Component/s:	workflow-plugin
Labels:	workflow
Assignee:	Jesse Glick

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)

jean@mertz.fm (JIRA)

unread,

Nov 8, 2016, 9:28:05 AM11/8/16

to jenkinsc...@googlegroups.com

Jean Mertz commented on

I believe we are seeing the same problem, using Git.

The below error happens when running tests in parallel, on different nodes, and each node triggering `checkout(scm)` using Jenkins Pipeline workflow.

 
                                                                [test:1]  > git rev-parse --is-inside-work-tree # timeout=10
[test:1] Fetching changes from 2 remote Git repositories
[test:1]  > git config remote.origin.url https://github.com/org/our-repo.git # timeout=10
[test:1] Fetching upstream changes from https://github.com/org/our-repo.git
[test:1]  > git --version # timeout=10
[test:1] using GIT_ASKPASS to set credentials github login credentials for "ci" account
[test:1]  > git fetch --tags --progress https://github.com/org/our-repo.git +refs/heads/*:refs/remotes/origin/*
[test:1]  > git config remote.origin1.url https://github.com/org/our-repo.git # timeout=10
[test:1] ERROR: Error fetching remote repo 'origin1'
[test:1] hudson.plugins.git.GitException: Failed to fetch from https://github.com/org/our-repo.git
[test:1]   at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:797)
[test:1]   at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1051)
[test:1]   at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1082)
[test:1]   at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
[test:1]   at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
[test:1]   at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
[test:1]   at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:52)
[test:1]   at hudson.security.ACL.impersonate(ACL.java:221)
[test:1]   at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:49)
[test:1]   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[test:1]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[test:1]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[test:1]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[test:1]   at java.lang.Thread.run(Thread.java:745)
[test:1] Caused by: hudson.plugins.git.GitException: Command "git config remote.origin1.url https://github.com/org/our-repo.git" returned status code 255:
[test:1] stdout: 
[test:1] stderr: error: could not lock config file .git/config: File exists
[test:1] 
[test:1]   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1752)
[test:1]   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1725)
[test:1]   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1721)
[test:1]   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1391)
[test:1]   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1403)
[test:1]   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1107)
[test:1]   at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
[test:1]   at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
[test:1]   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[test:1]   at java.lang.reflect.Method.invoke(Method.java:498)
[test:1]   at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:887)
[test:1]   at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:862)
[test:1]   at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:821)
[test:1]   at hudson.remoting.UserRequest.perform(UserRequest.java:153)
[test:1]   at hudson.remoting.UserRequest.perform(UserRequest.java:50)
[test:1]   at hudson.remoting.Request$2.run(Request.java:332)
[test:1]   at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
[test:1]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[test:1]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[test:1]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[test:1]   at hudson.remoting.Engine$1$1.run(Engine.java:88)
[test:1]   at java.lang.Thread.run(Thread.java:745)
[test:1]   at ......remote call to kubernetes-f2462ed7839d4e6a833775d67a2b867d-3b5c4c72b81c(Native Method)
[test:1]   at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1433)
[test:1]   at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
[test:1]   at hudson.remoting.Channel.call(Channel.java:797)
[test:1]   at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:252)
[test:1]   at com.sun.proxy.$Proxy108.setRemoteUrl(Unknown Source)
[test:1]   at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:294)
[test:1]   at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:785)
[test:1]   ... 13 more
[test:1] Failed in branch test:1
 
                                                            

jglick@cloudbees.com (JIRA)

unread,

Nov 8, 2016, 4:13:01 PM11/8/16

to jenkinsc...@googlegroups.com

Jesse Glick commented on

Jean Mertz probably an unrelated issue which should be filed separately with detailed steps to reproduce from scratch. Offhand it sounds like you are actually using the same workspace in different branches, but I do not really recognize the error, which is coming from the command-line git tool.

kerrhome (JIRA)

unread,

Jul 17, 2018, 11:59:03 AM7/17/18

to jenkinsc...@googlegroups.com

Shannon Kerr commented on

Just ran into this today when we switched from VM hosted Jenkins to docker container:

 
                                                                org.xml.sax.SAXParseException; Premature end of file.
 at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
 at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
 at org.apache.commons.digester.Digester.parse(Digester.java:1871)
 at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
Also: org.xml.sax.SAXParseException; Premature end of file.
 at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
 at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
 at org.apache.commons.digester.Digester.parse(Digester.java:1871)
 at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
 Caused: hudson.util.IOException2: Failed to parse /var/jenkins_home/jobs/*job*/branches/*branch*/builds/2/changelog1.xml
Caused: hudson.util.IOException2: Failed to parse /var/jenkins_home/jobs/*job*/branches/*branch*/builds/2/changelog1.xml
 at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80)
 at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43)
 at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:1031)
 at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1400(WorkflowRun.java:144)
 at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:1239)
 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:127)
 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
 at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
 at hudson.security.ACL.impersonate(ACL.java:290)
 at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE 
                                                            

Before today, we had never seen this. Been using Jenkins out of a VM (Ubuntu 12) for 4 years.

This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

kerrhome (JIRA)

unread,

Nov 8, 2018, 4:47:04 PM11/8/18

to jenkinsc...@googlegroups.com

Shannon Kerr commented on

We are getting this so frequently now. It has to be about 50% of our builds are failing with:

 
                                                                Failed to parse /var/jenkins_home/jobs/*job*/branches/*branch*/builds/103/changelog1.xml 
                                                            

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

kerrhome (JIRA)

unread,

Dec 6, 2018, 5:16:04 PM12/6/18

to jenkinsc...@googlegroups.com

Shannon Kerr commented on

This is killing us today. So man occurrences. More than 50% of our builds. The recommended "sleep time" work-around did not appear to help when implemented on some branch build testing. We are move it to trunk because it has gotten so bad today, but I'm not hopeful.

dietz1993@gmail.com (JIRA)

unread,

Dec 6, 2018, 9:12:10 PM12/6/18

to jenkinsc...@googlegroups.com

Aaron Dietz commented on

@Shannon Kerr

Have you also tried the lock method from my approach above?
Acquire a singleton lock "execution end" (or a nicer name ) at the end of the execution and release it instandly.
This way no execution can end in parallel.
Next escalation step is to add a minimal delay before you release the lock.

Definitely not pretty, but fixes it for me

kerrhome (JIRA)

unread,

Dec 11, 2018, 9:31:04 AM12/11/18

to jenkinsc...@googlegroups.com

Shannon Kerr commented on

Aaron Dietz I have not yet. Will definitely try this today. The team I support is really suffering with this issue. I stopped updating Jenkins and our plugins over a month ago (as we approach our product release), but this issue seems to be increasing in frequency, which I cannot explain since I've frozen everything.

kerrhome (JIRA)

unread,

Dec 11, 2018, 9:37:02 AM12/11/18

to jenkinsc...@googlegroups.com

Shannon Kerr commented on

Aaron Dietz can you share your implementation of the post checkout lock? I've not used this plugin up to now (lock). Thank you.

kerrhome (JIRA)

unread,

Dec 11, 2018, 10:10:02 AM12/11/18

to jenkinsc...@googlegroups.com

Shannon Kerr edited a comment on

[~mixermachine] can you share your implementation of the post checkout lock? I've not used this plugin up to now (lock). Thank you.

edit: Perhaps your implementation is what you wrote in your original comment above. I'm using scripted pipeline at this point and right now I have:

{code:java}
Random random = new Random()
sleep time: random.nextInt(2000), unit: 'MILLISECONDS'
checkout(.....)
lock ("SVN CO end for ${env.NODE_NAME}") {
echo "Post-svnco lock"
}
sleep time: random.nextInt(2000), unit: 'MILLISECONDS'
{code}
Is that sufficient?

dietz1993@gmail.com (JIRA)

unread,

Dec 11, 2018, 2:55:02 PM12/11/18

to jenkinsc...@googlegroups.com

Aaron Dietz commented on

Shannon Kerr

Just try your luck with the lock at first.

 
                                                                lock ("end_lock") {
    echo "end lock"
}

This is the least time consuming way.

If that does not work:

 
                                                                lock ("end_lock") {
    echo "end lock"
    sleep time: (50 + random.nextInt(400)) // hopefully enough time
}

It might be enough to have one lock for each node (as you did show), but it could also be necessary to synchronize all jobs as some files are written back to the master.

I hope this helps

Greetings from Germany

Aaron

kerrhome (JIRA)

unread,

Dec 11, 2018, 4:31:02 PM12/11/18

to jenkinsc...@googlegroups.com

Shannon Kerr commented on

Aaron Dietz Thank you. I've been testing with this all day today and so far so good. Hopefully your solution works until this ticket gets a proper resolution. Thank you again.

dietz1993@gmail.com (JIRA)

unread,

Dec 13, 2018, 5:39:02 AM12/13/18

to jenkinsc...@googlegroups.com

Aaron Dietz commented on

Shannon Kerr No problem, happy to help

kerrhome (JIRA)

unread,

Dec 14, 2018, 9:57:03 AM12/14/18

to jenkinsc...@googlegroups.com

Shannon Kerr commented on

Aaron Dietz well, I've had two occurrences of the issue since the new lock implementation. Just to be clear about what we're doing, we are running at least 3 svn co's in parallel via a call to our groovy script. These checkouts are happening on 3 different machines (one Linux physical machine for docker builds and 2 Windows VMs), all 3 checkouts in parallel. I assume when others are doing checkouts that they too are running parallel checkouts on different machines, but thought I'd better clarify.

Are there any other changes you (or others) recommend we try to stop these issues?

david.aldrich@emea.nec.com (JIRA)

unread,

Jan 3, 2019, 11:34:02 AM1/3/19

to jenkinsc...@googlegroups.com

David Aldrich commented on

Please show me how to implement a lock in a declarative pipeline script.

dietz1993@gmail.com (JIRA)

unread,

Jan 3, 2019, 12:46:02 PM1/3/19

to jenkinsc...@googlegroups.com

Aaron Dietz commented on

Shannon Kerr sorry for the late answer, I'm currently busy with my bachelor thesis.

Mh I can only guess...

I think preventing paralle CO is not an option ? (then you could simply create a lock for the CO command)

A very wild guess is to check what hard drives your (main?) server uses. Maybe you can upgrade to a SSD setup so the machine spends less time to write the files out? (not optimal fix for a software problem, I know)

The fastest thing you can currently try is to place locks and delays on variouse places.

I might look into this further after I submited my bachelor thesis (time budget is currently packed).

David Aldrich

Hy David

I have used the "normal" jenkins pipeline scripts till now and don't know of a declarative pipeline script (maybe we talk about the same? Groovy scripting language that defines a Jenkins Pipeline)

If we talk about the same, my approach from a few posts up should work for you (hopefully, as this is a temporary fix).

jglick@cloudbees.com (JIRA)

unread,

Jan 11, 2019, 5:04:02 PM1/11/19

to jenkinsc...@googlegroups.com

Jesse Glick commented on

AFAIK workflow-scm-step 2.7 should have resolved this (~~JENKINS-47201~~), unless you are talking about some unrelated issue.

dietz1993@gmail.com (JIRA)

unread,

Jan 13, 2019, 9:33:02 AM1/13/19

to jenkinsc...@googlegroups.com

Aaron Dietz commented on

Jesse Glick that are some good news, thanks for the implementation.

Shannon Kerr fixed for you?

david.aldrich@emea.nec.com (JIRA)

unread,

Jan 14, 2019, 3:24:03 AM1/14/19

to jenkinsc...@googlegroups.com

David Aldrich commented on

Jesse Glick I have seen the problem but I don't see workflow-scm-step 2.7 listed in Manage Plugins (Available or Installed). How can I see if it is installed?

david.aldrich@emea.nec.com (JIRA)

unread,

Jan 14, 2019, 3:26:03 AM1/14/19

to jenkinsc...@googlegroups.com

David Aldrich edited a comment on

[~jglick] I have seen the problem but I don't see workflow-scm-step 2.7 listed in Manage Plugins (Available or Installed). How can I see if it is installed guess you mean Pipeline: SCM Step 2.7 ?

david.aldrich@emea.nec.com (JIRA)

unread,

Jan 14, 2019, 3:34:02 AM1/14/19

to jenkinsc...@googlegroups.com

David Aldrich edited a comment on

[~jglick] I have seen the problem but I don't see workflow-scm-step 2.7 listed in Manage Plugins (Available or Installed). I guess you mean Pipeline: SCM Step 2.7? I updated to that plugin in October 2018, but have seen the problem since.

david.aldrich@emea.nec.com (JIRA)

unread,

Jan 14, 2019, 3:35:02 AM1/14/19

to jenkinsc...@googlegroups.com

David Aldrich edited a comment on

[~jglick] I guess you mean Pipeline: SCM Step 2.7? I updated to that plugin in October 2018, but have seen the problem since then .

jglick@cloudbees.com (JIRA)

unread,

Jan 14, 2019, 1:42:02 PM1/14/19

to jenkinsc...@googlegroups.com

Jesse Glick commented on

the problem

Meaning a ConcurrentModificationException serializing MultiSCMRevisionState#revisionStates, or some other problem?

david.aldrich@emea.nec.com (JIRA)

unread,

Jan 15, 2019, 4:11:02 AM1/15/19

to jenkinsc...@googlegroups.com

David Aldrich commented on

@jglick Sorry, I don't have a record of the exception now, but it did mention org.xml.sax.SAXParseException.

I seem to have fixed it by adding a lock at the end of each parallel stage.

david.aldrich@emea.nec.com (JIRA)

unread,

Jan 21, 2019, 3:42:07 AM1/21/19

to jenkinsc...@googlegroups.com

David Aldrich commented on

@jglick The exception occurred again for us this weekend:

 
                                                                org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/Zodiac_gcc_all_versions/builds/123/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
	at org.apache.commons.digester.Digester.parse(Digester.java:1871)
	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
Caused: hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/Zodiac_gcc_all_versions/builds/123/changelog1.xml
	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80)
	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:835)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1100(WorkflowRun.java:133)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:1100)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:51)
	at hudson.security.ACL.impersonate(ACL.java:290)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:48)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
 
                                                            

We are running Jenkins 2.150.1.

jglick@cloudbees.com (JIRA)

unread,

Jan 22, 2019, 2:52:03 PM1/22/19

to jenkinsc...@googlegroups.com

Jesse Glick updated an issue

Jenkins /

Exceptions by parallel SCM checkout

Studying the fix of ~~JENKINS-47201~~ by Devin Nusbaum I suspect it was incomplete, as this block is not synchronized as a whole, and so two concurrent branches could both pick the same changelog file.

This is probably reported against the Subversion plugin since that is the most widely used SCM plugin to actually treat changelog.xml as an XML file. Despite the name, the format is up to the SCM plugin to interpret, and the Git plugin writes and reads a completely non-XML format. (The Mercurial plugin also uses XML but is probably too little used for people to hit this and report it.)

Change By:	Jesse Glick
Component/s:	workflow-scm-step-plugin
Component/s:	subversion-plugin
Labels:	pipeline