[JIRA] (JENKINS-50975) Concurrent Groovy Shared Library syncs on different jobs use same workspace root

0 views
Skip to first unread message

kroutley@ea.com (JIRA)

unread,
Apr 24, 2018, 6:01:02 PM4/24/18
to jenkinsc...@googlegroups.com
Kurt Routley created an issue
 
Jenkins / Bug JENKINS-50975
Concurrent Groovy Shared Library syncs on different jobs use same workspace root
Issue Type: Bug Bug
Assignee: Unassigned
Attachments: job1.txt, job2.txt
Components: p4-plugin, workflow-cps-global-lib-plugin
Created: 2018-04-24 22:00
Environment: CloudBees Jenkins Enterprise 2.89.4.2-rolling
Pipeline: Groovy (workflow-cps): 2.47
Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.9
P4 Plugin (p4): 1.7.3-DRE1.25
Operating System: CentOS release 6.7 (Final)
Java version: java version "1.8.0_131" , Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Labels: shared-groovy-libraries sync root
Priority: Major Major
Reporter: Kurt Routley

We're encountering an issue where multiple jobs syncing a groovy shared library concurrently results in the wrong sync root being used for some of the jobs. For example, job1 and job2 are triggered at the same time and use the p4 plugin for syncing the groovy shared library. job1 will have the correct workspace root (e.g.  /var/lib/jenkins/jobs/job1/workspace%40libs/LIBRARYNAME) for the groovy shared library sync, but job2 will sync the groovy shared library to the same workspace root as job1 (e.g.  /var/lib/jenkins/jobs/job1/workspace%40libs/LIBRARYNAME). When job2 loads the groovy shared library, it references what the workspace root should be for job2 (e.g.  /var/lib/jenkins/jobs/job2/workspace%40libs/LIBRARYNAME), which was not sync'd to, and causes job2 to run based off of an old groovy shared library sync.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

kroutley@ea.com (JIRA)

unread,
Apr 24, 2018, 6:03:02 PM4/24/18
to jenkinsc...@googlegroups.com

pallen@perforce.com (JIRA)

unread,
Jul 25, 2018, 6:08:36 AM7/25/18
to jenkinsc...@googlegroups.com
Paul Allen closed an issue as Fixed
 

Resolved in 1.8.12

Change By: Paul Allen
Status: Open Closed
Assignee: Paul Allen
Resolution: Fixed
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

sven.schubert.eso@gmail.com (JIRA)

unread,
Oct 9, 2018, 2:38:02 AM10/9/18
to jenkinsc...@googlegroups.com
Sven Schubert updated an issue
Change By: Sven Schubert
Attachment: error_logs.txt
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

sven.schubert.eso@gmail.com (JIRA)

unread,
Oct 9, 2018, 2:39:02 AM10/9/18
to jenkinsc...@googlegroups.com
Sven Schubert commented on Bug JENKINS-50975
 
Re: Concurrent Groovy Shared Library syncs on different jobs use same workspace root

This bug still exists with version 1.8.14 of the P4 plugin, and Pipeline 2.5:  We have one shared library configured in Jenkins. If the Jenkins tries to start multiple jobs (A, B, C) at the same time (e.g. because an upstream project has been built and multiple jobs depend on this project), only the first job (A) will be able to load the shared library. The other jobs (B, C) get confused about the workspace when trying to load the shared library, e.g. job B tries to find the shared library (pipeline-commons) within the workspace of another job aerror_logs.txtnd not within his own workspace. 

(If the jobs are not started at the same time, everything works fine.)

sven.schubert.eso@gmail.com (JIRA)

unread,
Oct 9, 2018, 2:40:03 AM10/9/18
to jenkinsc...@googlegroups.com
Sven Schubert edited a comment on Bug JENKINS-50975
This bug still exists with version 1.8.14 of the P4 plugin, and Pipeline 2.5:  We have one shared library configured in Jenkins. If the Jenkins tries to start multiple jobs (A, B, C) at the same time (e.g. because an upstream project has been built and multiple jobs depend on this project), only the first job (A) will be able to load the shared library. The other jobs (B, C) get confused about the workspace when trying to load the shared library, e.g. job B tries to find the shared library (pipeline-commons) within the workspace of another job a and not within his own workspace ( [^error_logs.txt] nd not within his own workspace )


(If the jobs are not started at the same time, everything works fine.)

sven.schubert.eso@gmail.com (JIRA)

unread,
Oct 9, 2018, 2:51:04 AM10/9/18
to jenkinsc...@googlegroups.com
Sven Schubert edited a comment on Bug JENKINS-50975
This bug still exists with version 1.8.14 of the P4 plugin, and Pipeline 2.5:  We have one shared library configured in Jenkins. If the Jenkins tries to start multiple jobs (A, B, C) at the same time (e.g. because an upstream project has been built and multiple jobs depend on this project), only the first job (A) will be able to load the shared library. The other jobs (B, C) get confused about the workspace when trying to load the shared library, e.g. job B tries to find the shared library (pipeline-commons) within the workspace of another job and not within his own workspace ([^error_logs.txt]).  

( If the jobs are not started at the same time, everything works fine.  

Environment:
* Pipeline: 2.5
* Pipeline: Groovy (workflow-cps
) : 2.54 
* Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.9

sven.schubert.eso@gmail.com (JIRA)

unread,
Oct 9, 2018, 2:54:01 AM10/9/18
to jenkinsc...@googlegroups.com
Sven Schubert updated an issue
Change By: Sven Schubert
Comment:
This bug still exists with version 1.8.14 of the P4 plugin, and Pipeline 2.5:  We have one shared library configured in Jenkins. If the Jenkins tries to start multiple jobs (A, B, C) at the same time (e.g. because an upstream project has been built and multiple jobs depend on this project), only the first job (A) will be able to load the shared library. The other jobs (B, C) get confused about the workspace when trying to load the shared library, e.g. job B tries to find the shared library (pipeline-commons) within the workspace of another job and not within his own workspace ([^error_logs.txt]).

If the jobs are not started at the same time, everything works fine. 

Environment:
* Pipeline: 2.5
* Pipeline: Groovy (workflow-cps): 2.54 

* Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.9

sven.schubert.eso@gmail.com (JIRA)

unread,
Oct 9, 2018, 2:55:02 AM10/9/18
to jenkinsc...@googlegroups.com
Sven Schubert reopened an issue
 

This bug still exists with version 1.8.14 of the P4 plugin, and Pipeline 2.5:  We have one shared library configured in Jenkins. If the Jenkins tries to start multiple jobs (A, B, C) at the same time (e.g. because an upstream project has been built and multiple jobs depend on this project), only the first job (A) will be able to load the shared library. The other jobs (B, C) get confused about the workspace when trying to load the shared library, e.g. job B tries to find the shared library (pipeline-commons) within the workspace of another job and not within his own workspace (error_logs.txt).

If the jobs are not started at the same time, everything works fine. 

Environment:

  • Pipeline: 2.5
  • Pipeline: Groovy (workflow-cps): 2.54 
  • Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.9
Change By: Sven Schubert
Resolution: Fixed
Status: Closed Reopened

pallen@perforce.com (JIRA)

unread,
Oct 18, 2018, 11:56:02 AM10/18/18
to jenkinsc...@googlegroups.com
Paul Allen commented on Bug JENKINS-50975
 
Re: Concurrent Groovy Shared Library syncs on different jobs use same workspace root

I have recently made changes to Global Library workspaces for multiple instances JENKINS-53922
https://ci.jenkins.io/job/Plugins/job/p4-plugin/job/master/281/

I may need to add ${EXECUTOR_NUMBER} to the Global Library Workspace name formatter:

In file GlobalLibraryScmScurce.java:43

setFormat("jenkins-lib-${NODE_NAME}-${JOB_NAME}" + id);
setFormat("jenkins-lib-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}" + id);

Did you want to try this out and build it yourself or I can submit a change for you to try on the nightly build?

pallen@perforce.com (JIRA)

unread,
Oct 18, 2018, 11:59:02 AM10/18/18
to jenkinsc...@googlegroups.com
Paul Allen edited a comment on Bug JENKINS-50975
I have recently made changes to Global Library workspaces for multiple instances JENKINS-53922
[ https://ci.jenkins.io/job/Plugins/job/p4-plugin/job/master/281/ ]

I may need to add {{$
\
{EXECUTOR_NUMBER}}} to the Global Library Workspace name formatter:

In file GlobalLibraryScmScurce.java:43
{code:java}

setFormat("jenkins-lib-${NODE_NAME}-${JOB_NAME}" + id);
{code}

{code:java}

setFormat("jenkins-lib-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}" + id);
{code}

Did you want to try this out and build it yourself or I can submit a change for you to try on the nightly build?


 

(Just re-reading your comment and logs - Build 281 might be enough without the EXECUTOR_NUMBER)

premgangana@gmail.com (JIRA)

unread,
Oct 18, 2018, 4:55:02 PM10/18/18
to jenkinsc...@googlegroups.com

Hi Paul Allen,

I have tested this with the build 281. It is using 


"jenkins-lib-${NODE_NAME}-${JOB_NAME}" + id

as client name now, but i would suggest replacing / with _ in id instead of . (dot). So this is unique across different jobs due to JOB_NAME in there.

However, the issue will surface if a given job has two builds triggered at / almost at the same time, since the jenkins appends @<number> to the build workspace, and so the client root dirs will be different. I would suggest adding WORKSPACE base name some where in there.

 

pallen@perforce.com (JIRA)

unread,
Oct 19, 2018, 5:17:01 AM10/19/18
to jenkinsc...@googlegroups.com

Happy to change . to {{_ }}but was curious to know why?  Do you have dots in your JOB_NAME?

If the same Job is started concurrently then the EXECUTOR_NUMBER should be enough, this is the @1 or @2 that Jenkins adds to the directory. I think WORKSPACE would also work, but we could end up with very long names.  I had considered using a SHA1 of the NODE_NAME + JOB_NAME + depot path + EXECUTOR, but it becomes harder to debug if problems appear later on.

pallen@perforce.com (JIRA)

unread,
Oct 19, 2018, 8:27:02 AM10/19/18
to jenkinsc...@googlegroups.com

Turns out EXECUTOR_NUMBER and WORKSPACE are not define when the client workspace is created, Jenkins sets these variables later.

Plan B is to use a UUID and clean up (delete the client) after syncing the Library.

premgangana@gmail.com (JIRA)

unread,
Oct 19, 2018, 1:00:02 PM10/19/18
to jenkinsc...@googlegroups.com

> Having - is for uniformity sake since you were having jenkins-lib${NODE_NAME}-${JOB_ NAME}. 

-> Deleting the client is actually what i was going to ask you about. Cleaning up the client is needed here, otherwise the clients are going to get stacked up.

-> How about using BUILD_NUMBER ?

 

premgangana@gmail.com (JIRA)

unread,
Oct 19, 2018, 1:01:01 PM10/19/18
to jenkinsc...@googlegroups.com
Prem Gangana edited a comment on Bug JENKINS-50975
- > Having - is for uniformity sake since you were having jenkins-lib - ${NODE_NAME}-${JOB_ NAME}. 


-> Deleting the client is actually what i was going to ask you about. Cleaning up the client is needed here, otherwise the clients are going to get stacked up.

-> How about using BUILD_NUMBER ?

 

pallen@perforce.com (JIRA)

unread,
Oct 23, 2018, 4:17:02 AM10/23/18
to jenkinsc...@googlegroups.com

I have tried 'Plan B' a UUID and delete. 

I have also exposed the delete option for Manual Workspaces (visible in FreeStyle and Pipeline 'checkout' steps).  Not too sure if I will release it like this as deleting the client for normal 'checkout' steps may effect polling.

https://ci.jenkins.io/job/Plugins/job/p4-plugin/job/master/284/

pallen@perforce.com (JIRA)

unread,
Oct 23, 2018, 10:16:02 AM10/23/18
to jenkinsc...@googlegroups.com
Paul Allen updated an issue
 
Change By: Paul Allen
Labels: P4_A root shared-groovy-libraries sync

pallen@perforce.com (JIRA)

unread,
Oct 23, 2018, 10:17:04 AM10/23/18
to jenkinsc...@googlegroups.com
Paul Allen updated Bug JENKINS-50975
 

Ready for release.

Change By: Paul Allen
Status: Reopened Fixed but Unreleased
Resolution: Fixed

pallen@perforce.com (JIRA)

unread,
Oct 30, 2018, 11:12:02 AM10/30/18
to jenkinsc...@googlegroups.com

j.spang@gmail.com (JIRA)

unread,
Aug 9, 2019, 5:43:04 PM8/9/19
to jenkinsc...@googlegroups.com
Jay Spang reopened an issue
 

I know this issue is old, but I think it's repro'ing for me on Jenkins 2.164.3 with P4 Plugin 1.9.7.

The key difference is that I'm loading a DYNAMIC library. I don't know if that's relevant. Near the top of my pipeline, I load a library with this snippet:

String stream_name = 'stream_name'
String client_name = "jenkins-${env.JOB_NAME}-${stream_name}-library"
String library_source = "//depot/${stream_name}/Jenkinsfile/lib/..."

library(
    identifier: stream_name + '-library@now', 
    retriever: legacySCM(
        [
            $class: 'PerforceScm', credential: 'p4creds',
            populate: [
                $class: 'AutoCleanImpl', 
                delete: true, 
                modtime: false, 
                parallel: [
                    enable: false, 
                    minbytes: '1024', 
                    minfiles: '1', 
                    threads: '4'
                ], 
                pin: '', 
                quiet: true, 
                replace: true, 
                tidy: false
            ], 
            workspace: [
                $class: 'ManualWorkspaceImpl', 
                charset: 'none', 
                name: client_name,
                pinHost: false, 
                spec: [
                    allwrite: true, 
                    clobber: false, 
                    compress: false, 
                    line: 'LOCAL', 
                    locked: false, 
                    modtime: false, 
                    rmdir: false, 
                    streamName: '', 
                    view: "${library_source} //${client_name}/..."
                ]
            ]
        ]
    )
)

// rest of the pipeline

This works great in isolation. But my job is failing intermittently with this error. It seems to happen most (but not always!) when multiple builds are triggered at the same time.

P4 Task: reverting all pending and shelved revisions.
... p4 revert d:\JenkinsHome\jobs\stream_name\workspac___
 -
p4 revert d:\JenkinsHome\jobs\stream_name\jobs\stream_name-build\workspace_libs\stream_name-library_2/...

Path 'd:\JenkinsHome\jobs\stream_name\workspace_libs\stream_name-library_2/...' is not under client's root 'd:\JenkinsHome\jobs\stream_name\workspace_libs\stream_name-library'.

I can't figure out what logic it uses to append "_2" to the folder name, nor why it's failing to update the p4 client accordingly. It also seems like a bug that one path ends with "/..." while the other does not.

Change By: Jay Spang
Resolution: Fixed
Status: Closed Reopened

j.spang@gmail.com (JIRA)

unread,
Aug 9, 2019, 5:56:04 PM8/9/19
to jenkinsc...@googlegroups.com
I know this issue is old, but I think it's repro'ing for me on Jenkins 2.164.3 with P4 Plugin 1.9.7.

The key difference is that I'm loading a DYNAMIC library. I don't know if that's relevant. Near the top of my pipeline, I load a library with this snippet:
{code:java}
{code}

This works great in isolation. But my job is failing intermittently with this error. It seems to happen most (but not always!) when multiple builds are triggered at the same time.

{code :xml }

P4 Task: reverting all pending and shelved revisions.
... p4 revert d:\JenkinsHome\jobs\stream_name\workspac___
-
p4 revert d:\JenkinsHome\jobs\stream_name\jobs\stream_name-build\workspace_libs\stream_name-library_2/...

ERROR: P4: Task Exception: hudson.AbortException: P4JAVA: Error(s):
Path 'd:\JenkinsHome\jobs\stream_name\workspace_libs\stream_name-library_2/...' is not under client's root 'd:\JenkinsHome\jobs\stream_name\workspace_libs\stream_name-library'.
{code}


I can't figure out what logic it uses to append "_2" to the folder name, nor why it's failing to update the p4 client accordingly. It also seems like a bug that one path ends with "/..." while the other does not.

j.spang@gmail.com (JIRA)

unread,
Aug 9, 2019, 7:27:02 PM8/9/19
to jenkinsc...@googlegroups.com
Jay Spang edited a comment on Bug JENKINS-50975
I know this issue is old, but I think it's repro'ing for me on Jenkins 2.164.3 with P4 Plugin 1.9.7.

The key difference is that I'm loading both a global library and a DYNAMIC library. I don't know if that's relevant. Near the top of my pipeline, I load a library with this snippet:
{code:java}
library 'global-library@now'

The global library works fine and never seems to have a problem. The 'stream library' that I dynamically load occasionally throws an error syncing on the master.

I
can't figure out what logic it uses to append "_2" to the folder name, nor why it's failing to update the p4 client accordingly. It also seems like a bug that one path ends with "/..." while the other does not.

j.spang@gmail.com (JIRA)

unread,
Aug 12, 2019, 12:55:05 PM8/12/19
to jenkinsc...@googlegroups.com
Jay Spang edited a comment on Bug JENKINS-50975
I know this issue is old, but I think it's repro'ing for me on Jenkins 2.164.3 with P4 Plugin 1.9.7.

The key difference Here is an example pipeline that I'm loading both a global library and a DYNAMIC library repros the error . I don't know if Note that 's relevant. Near the top of my pipeline, I load it DYNAMICALLY loads a library with this snippet: pipeline.
{code:java}
library
'global-library@now'


String stream_name = 'stream_name'
String client_name = "jenkins-${env.JOB_NAME}-${stream_name}-library"
String library_source = "//depot/${stream_name}/Jenkinsfile/lib/..."

library
(
    identifier:
stream_name + ' dynamically - loaded- library@now',
    retriever: legacySCM(
        [
            $class: 'PerforceScm', credential: 'p4creds',
            populate: [
                $class: 'AutoCleanImpl',
                delete: true,
                modtime: false,
                parallel: [

                    enable: false,
                    minbytes: '1024',
                    minfiles: '1',
                    threads: '4'
                ],
                pin: '',
                quiet: true,
                replace: true,
                tidy: false
            ],
            workspace: [
                $class: 'ManualWorkspaceImpl',
                charset: 'none',
                name: client_name "jenkins-${env.JOB_NAME}-dynamically-loaded-library" ,
                pinHost: false,
                spec: [

                    allwrite: true,
                    clobber: false,
                    compress: false,
                    line: 'LOCAL',
                    locked: false,
                    modtime: false,
                    rmdir: false,
                    streamName: '',
                    view: "
${library_source} // depot/build-library/... //jenkins- ${ client_name env.JOB_NAME } -dynamically-loaded-library /..."
                ]
            ]
        ]
    )
)

// rest of the pipeline node() {
    stage('Echo') {
        echo("If you hit 'Build Now' rapidly enough, a build wiill fail before it even gets here.")
    }
}

{
code}
This If you hit "Build Now" once, it works great in isolation . But my job is failing intermittently If you rapidly hit it a few times, several builds will fail with this error. It seems to happen most (but not always!) when multiple builds are triggered at the same time.

{code:xml}
P4 Task: reverting all pending and shelved revisions.
... p4 revert d:\JenkinsHome\jobs\stream_name\workspac___
-
p4 revert d:\JenkinsHome\jobs\stream_name\jobs\stream_name -build \workspace_libs\ stream_name dynamically - loaded- library_2/...


ERROR: P4: Task Exception: hudson.AbortException: P4JAVA: Error(s):
Path 'd:\JenkinsHome\jobs\stream_name\workspace_libs\ stream_name dynamically - loaded- library_2/...' is not under client's root 'd:\JenkinsHome\jobs\stream_name\workspace_libs\ stream_name dynamically - loaded- library'.
{code}

The global library works fine and never It seems to have like when two builds are run in a problem. The 'stream library' that I dynamically load occasionally throws an error syncing on VERY close timeframe, the master.

I can't figure out what logic it uses to
plugin will append "_2" to the folder name, nor why it library 's failing to update foldername, but re-uses the old p4 client accordingly workspace . It also seems like a bug that one path ends

EDIT: Edited my original comment to be more concise,
with "/ a working example . .." while the other does not.

msmeeth@perforce.com (JIRA)

unread,
Aug 14, 2019, 12:33:02 PM8/14/19
to jenkinsc...@googlegroups.com

Hi Jay, 

I'm currently trying to reproduce your issue, so far I have been unable to. If possible could you share your library file?

Thanks,

Matthew

j.spang@gmail.com (JIRA)

unread,
Aug 14, 2019, 2:25:03 PM8/14/19
to jenkinsc...@googlegroups.com

To be extra clear, it only repros for me with dynamically loaded libraries.

If you import your library with...

library 'static-global-library@now'

it doesn't happen. But if you load your library with...

library(identifier: 'dynamically-loaded-library@now', retriever: legacySCM( // ... ) )`

it happens. Just click "Build Now" as rapidly as you you can 4-5 times.

I can still send you a library if you'd like, but the failure happens before it even loads, so its contents don't matter.

j.spang@gmail.com (JIRA)

unread,
Aug 14, 2019, 2:25:04 PM8/14/19
to jenkinsc...@googlegroups.com
Jay Spang edited a comment on Bug JENKINS-50975
To be extra clear, it only repros for me with _dynamically_ loaded libraries.

If you import your library with... {code}library 'static-global-library@now'{code} it doesn't happen. But if you load your library with... {code}library(identifier: 'dynamically-loaded-library@now', retriever: legacySCM( // ... ) )
` {code} it happens. Just click "Build Now" as rapidly as you you can 4-5 times.


I can still send you a library if you'd like, but the failure happens before it even loads, so its contents don't matter.

j.spang@gmail.com (JIRA)

unread,
Aug 19, 2019, 1:03:03 PM8/19/19
to jenkinsc...@googlegroups.com

Hey Matthew Smeeth, I was wondering if you had any luck repro'ing the error? The key is to use "dynamically loaded" libraries (where you specify the `retriever` property). You also have to hit "Build Now" very rapidly to get it to repro.

I hypothesize that p4groovy isn't changing the workspace root when it decides to append "_2" to the folder name. Since it works properly when specifying global libraries (that were previously loaded into the Jenkins global config), maybe it uses a slightly different code path.

Reply all
Reply to author
Forward
0 new messages