[JIRA] [core] (JENKINS-25930) Race condition creating build links during concurrent builds causes build failures

36 views
Skip to first unread message

angelo.perera@iress.com.au (JIRA)

unread,
Jun 4, 2015, 6:22:02 PM6/4/15
to jenkinsc...@googlegroups.com
Angelo Perera commented on Bug JENKINS-25930
 
Re: Race condition creating build links during concurrent builds causes build failures

We have a very large distributed Jenkins CI undertaking at the moment and our first few tests always fail due to this error.

TraderPlus.UI.Test #4641 completed : FAILURE
TraderPlus.UI.Test #4646 completed : FAILURE
Build TraderPlus.UI.Test #4655 started
Build TraderPlus.UI.Test #4654 started
TraderPlus.UI.Test #4645 completed : FAILURE
Build TraderPlus.UI.Test #4656 started
TraderPlus.UI.Test #4648 completed : FAILURE
Build TraderPlus.UI.Test #4657 started
TraderPlus.UI.Test #4639 completed : FAILURE
Build TraderPlus.UI.Test #4658 started
TraderPlus.UI.Test #4642 completed : FAILURE
Build TraderPlus.UI.Test #4659 started
TraderPlus.UI.Test #4647 completed
Build TraderPlus.UI.Test #4660 started
TraderPlus.UI.Test #4644 completed
Build TraderPlus.UI.Test #4661 started
(Rest of the tests pass)

Failure details:
ln builds\lastSuccessfulBuild C:\Users\Administrator\.jenkins\jobs\TraderPlus.UI.Test\lastSuccessful failed
java.nio.file.FileAlreadyExistsException: C:\Users\Administrator\.jenkins\jobs\TraderPlus.UI.Test\lastSuccessful
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.createSymbolicLink(Unknown Source)
at java.nio.file.Files.createSymbolicLink(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at hudson.Util.createSymlinkJava7(Util.java:1227)
at hudson.Util.createSymlink(Util.java:1143)
at hudson.model.Run.createSymlink(Run.java:1848)
at hudson.model.Run.updateSymlinks(Run.java:1829)
at hudson.model.Run.execute(Run.java:1742)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:374)
[EnvInject] - Loading node environment variables.
Building remotely on au03-devcivm4 (UITest) in workspace D:\Jenkins\workspace\TraderPlus.UI.Test

Running Jenkins ver. 1.616

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

Adam.Beben@dysant.com (JIRA)

unread,
Jul 8, 2015, 6:42:03 AM7/8/15
to jenkinsc...@googlegroups.com

The same issue occures when two builds starts at the same time (parallel builds) on different Jenkins nodes.
Platform: Windows (master and nodes)
Jenkins: ver. 1.619
Builds run in parallel with Build Flow Plugin 0.18

ln builds\lastSuccessfulBuild [path_to_project_workspace]\lastSuccessful failed
java.nio.file.FileAlreadyExistsException: [path_to_project_workspace]\lastSuccessful


at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.createSymbolicLink(Unknown Source)
at java.nio.file.Files.createSymbolicLink(Unknown Source)

at sun.reflect.GeneratedMethodAccessor359.invoke(Unknown Source)


at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at hudson.Util.createSymlinkJava7(Util.java:1227)
at hudson.Util.createSymlink(Util.java:1143)

at hudson.model.Run.createSymlink(Run.java:1845)
at hudson.model.Run.updateSymlinks(Run.java:1826)
at hudson.model.Run.execute(Run.java:1739)


at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)

at hudson.model.Executor.run(Executor.java:381)
ln builds\lastStableBuild [path_to_project_workspace]\lastStable failed
java.nio.file.FileAlreadyExistsException: [path_to_project_workspace]\lastStable


at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.createSymbolicLink(Unknown Source)
at java.nio.file.Files.createSymbolicLink(Unknown Source)

at sun.reflect.GeneratedMethodAccessor359.invoke(Unknown Source)


at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at hudson.Util.createSymlinkJava7(Util.java:1227)
at hudson.Util.createSymlink(Util.java:1143)

at hudson.model.Run.createSymlink(Run.java:1845)
at hudson.model.Run.updateSymlinks(Run.java:1827)
at hudson.model.Run.execute(Run.java:1739)


at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)

at hudson.model.Executor.run(Executor.java:381)

Adam.Beben@dysant.com (JIRA)

unread,
Jul 8, 2015, 6:44:01 AM7/8/15
to jenkinsc...@googlegroups.com
Adam Beben edited a comment on Bug JENKINS-25930
The same issue occures when two builds starts at the same time (parallel builds) on different Jenkins nodes.
Platform: Windows (master and nodes)
Jenkins: ver. 1.619
Builds run in parallel with Build Flow Plugin 0.18
In my case exceptions occurs in one of two builds (first or seconds, it's random) but build continues normally.

Adam.Beben@dysant.com (JIRA)

unread,
Jul 8, 2015, 6:45:01 AM7/8/15
to jenkinsc...@googlegroups.com
Adam Beben edited a comment on Bug JENKINS-25930
The same issue occures when two builds starts at the same time (parallel builds) on different Jenkins nodes.
Platform: Windows (master and nodes)
Jenkins: ver. 1.619
Builds run in parallel with Build Flow Plugin 0.18

In my case exceptions occurs in one of two builds (first or  seconds  second , it's random) but build continues normally.

elgalu3@gmail.com (JIRA)

unread,
Aug 5, 2015, 4:14:02 AM8/5/15
to jenkinsc...@googlegroups.com

I can recreate, any updates on this?

as_kumlali@yahoo.com (JIRA)

unread,
Jan 15, 2016, 5:49:02 AM1/15/16
to jenkinsc...@googlegroups.com

The problem still exists in 1.643 version.

as_kumlali@yahoo.com (JIRA)

unread,
Jan 27, 2016, 12:57:01 PM1/27/16
to jenkinsc...@googlegroups.com

It seems the problem only occurs when the job triggered by MultiJob Plug-in. The job can run parallel if I trigger it manually.

The other thing is the problem does not occur on Windows. I can reproduce it in Linux.

as_kumlali@yahoo.com (JIRA)

unread,
Jan 28, 2016, 3:11:01 AM1/28/16
to jenkinsc...@googlegroups.com

I figured out, in Windows, MultiJob Plug-in does not execute the jobs in parallel. In addition, the parallel job directories (e.g. myjob@1, myjob@2, ...) were not created. This behavior explains why I do not see the problem on Windows.

In Linux, to confirm the problem is related with MultiJob Plug-in, I created a shell script to execute the job in parallel through Jenkins' Remote API:

#!/bin/sh

( curl -X POST http://jenkins.mycompany.com/jenkins/job/myjob/build?delay=0sec --user jenkinsuser:jenkinspassword ) &
( curl -X POST http://jenkins.mycompany.com/jenkins/job/myjob/build?delay=0sec --user jenkinsuser:jenkinspassword ) &
( curl -X POST http://jenkins.mycompany.com/jenkins/job/myjob/build?delay=0sec --user jenkinsuser:jenkinspassword ) &
wait
echo "completed"

The job executed in parallel and the problem did not happen. I confirmed that the parallel job directories were created successfully.

Should we changed the issue's component to multijob-plugin?

dev@baltrinic.com (JIRA)

unread,
Feb 1, 2016, 12:54:03 PM2/1/16
to jenkinsc...@googlegroups.com

Not sure if it is really the mutli-job plugin or just the ability to start the jobs really closely together. We are experiencing this and do no have the multi-job plugin installed. In stead we use the build-flow plugin. Our use case is that we kick of multiple jobs in parallel using the parallel command in the build-flow dsl.

as_kumlali@yahoo.com (JIRA)

unread,
Mar 11, 2016, 6:39:02 AM3/11/16
to jenkinsc...@googlegroups.com

You seem to be right Kenneth. Thanks for the clarification.

mark.mann@krogan.co.uk (JIRA)

unread,
May 16, 2016, 12:09:02 PM5/16/16
to jenkinsc...@googlegroups.com

i recently upgraded to Jenkins 1.651.1 (on windows)
we make a lot of use of the parallel builds function... running 15 jobs in parallel.
proportionately.. only a couple spit out this error... and can end up killing the entire job run.

In the same parallel run, I can end up with different fault reasons among the jobs suggesting that dependent on the timing, a couple were competing with each other. eg: access denied, file already exists....

------ job@1
16:32:42 ln builds\lastStableBuild C:\.jenkins\jobs\XXXXX\lastStable failed
16:32:42 java.nio.file.AccessDeniedException: C:\.jenkins\jobs\XXXXX\lastStable
16:32:42 at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
16:32:42 at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
16:32:42 at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
16:32:42 at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
16:32:42 at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
16:32:42 at java.nio.file.Files.deleteIfExists(Unknown Source)
16:32:42 at sun.reflect.GeneratedMethodAccessor1458.invoke(Unknown Source)
16:32:42 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
16:32:42 at java.lang.reflect.Method.invoke(Unknown Source)
16:32:42 at hudson.Util.createSymlinkJava7(Util.java:1233)
16:32:42 at hudson.Util.createSymlink(Util.java:1151)
16:32:42 at hudson.model.Run.createSymlink(Run.java:1840)
16:32:42 at hudson.model.Run.updateSymlinks(Run.java:1822)
16:32:42 at hudson.model.Run.execute(Run.java:1736)
16:32:42 at com.tikal.jenkins.plugins.multijob.MultiJobBuild.run(MultiJobBuild.java:73)
16:32:42 at hudson.model.ResourceController.execute(ResourceController.java:98)
16:32:42 at hudson.model.Executor.run(Executor.java:410)

------ job@2
16:32:41 ln builds\lastSuccessfulBuild C:\.jenkins\jobs\XXXX\lastSuccessful failed
16:32:41 java.nio.file.FileAlreadyExistsException: C:\.jenkins\jobs\XXXXX\lastSuccessful
16:32:41 at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
16:32:41 at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
16:32:41 at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
16:32:41 at sun.nio.fs.WindowsFileSystemProvider.createSymbolicLink(Unknown Source)
16:32:41 at java.nio.file.Files.createSymbolicLink(Unknown Source)
16:32:41 at sun.reflect.GeneratedMethodAccessor1459.invoke(Unknown Source)
16:32:41 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
16:32:41 at java.lang.reflect.Method.invoke(Unknown Source)
16:32:41 at hudson.Util.createSymlinkJava7(Util.java:1235)
16:32:41 at hudson.Util.createSymlink(Util.java:1151)
16:32:41 at hudson.model.Run.createSymlink(Run.java:1840)
16:32:41 at hudson.model.Run.updateSymlinks(Run.java:1821)
16:32:41 at hudson.model.Run.execute(Run.java:1736)
16:32:41 at com.tikal.jenkins.plugins.multijob.MultiJobBuild.run(MultiJobBuild.java:73)
16:32:41 at hudson.model.ResourceController.execute(ResourceController.java:98)
16:32:41 at hudson.model.Executor.run(Executor.java:410)

cmodien+jenkins@gmail.com (JIRA)

unread,
Feb 20, 2018, 10:04:03 AM2/20/18
to jenkinsc...@googlegroups.com

I was able to work around this by creating a bunch of jobs from a template job and giving each job a unique name (e.g. unit-test-1, unit-test-2).

Here's some groovy code that I put in a job to recreate the jobs. It takes a single parameter NUM_JOBS that it also updates a global jenkins environment variable called NUM_RUN_TESTS_JOBS in case your tests need to know that they are job X of NUM_RUN_TESTS_JOBS. 

 

 

import jenkins.model.Jenkins
import hudson.model.AbstractProject
def num_jobs = build.getEnvironment(listener).get("NUM_JOBS")
//update the global env property 
instance = Jenkins.getInstance()
globalNodeProperties = instance.getGlobalNodeProperties()
envVarsNodePropertyList = globalNodeProperties.getAll(hudson.slaves.EnvironmentVariablesNodeProperty.class)
envVars = envVarsNodePropertyList.get(0).getEnvVars()
envVars.put("NUM_RUN_TESTS_JOBS", num_jobs)
instance.save()
base_name = "unit-tests-"
// delete the jobs
instance.getAllItems(AbstractProject.class).each {it ->
 if(it.fullName.contains(base_name)) {
 println('deleting ' + it.fullName)
 it.delete()
 }
}
//create the jobs
template_name = "unit-test-template"
(1..num_jobs.toInteger()).each {
 copied_name = base_name + it
 println('copying ' + template_name + ' to ' + copied_name)
 try {
 def job = instance.copy(instance.getItem(template_name), copied_name);
 job.save()
 } catch (IllegalArgumentException e) {}
}

 

 

 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages