[JIRA] (JENKINS-59657) Concurrent runs of a pipeline job that uses a custom workspaces cause p4sync to fail occassionally

13 views
Skip to first unread message

vsiakka@gmail.com (JIRA)

unread,
Oct 4, 2019, 9:26:03 AM10/4/19
to jenkinsc...@googlegroups.com
Vasiliki Siakka created an issue
 
Jenkins / Bug JENKINS-59657
Concurrent runs of a pipeline job that uses a custom workspaces cause p4sync to fail occassionally
Issue Type: Bug Bug
Assignee: Unassigned
Components: p4-plugin, workflow-durable-task-step-plugin
Created: 2019-10-04 13:25
Environment: p4-plugin: 1.9.6
workflow-durable-task-step: 2.29
Jenkins: 2.168
Priority: Minor Minor
Reporter: Vasiliki Siakka

I have a number of jobs that are named differently, and do similar but different operations, but have the same perforce mapping and run the same setup.

I wanted to share a workspace amongst all those jobs, so that i can benefit from the p4sync and setup caching. 

I wrote my jobs like this:

node('myNodeLabel') {
    ws('C:\\MyCustomWorkspace') {
        stage('Prepare node') {
          def p4WsName = 'custom-workspace-${NODE_NAME}'
          p4sync credential: 'myCred',
           changelog: false,
           populate: autoClean(replace: true, quiet: true, delete:false),
            workspace: [
              $class: 'ManualWorkspaceImpl',
              charset: 'utf8',
              pinHost: false,
              name: p4WsName,
              spec: [
                allwrite: true,
                backup: false,
                clobber: true,
                compress: false,
                line: 'LOCAL',
                locked: false,
                modtime: false,
                rmdir: true,
                streamName: '',
                type: 'WRITABLE',
                view: """\
                  //depot/... //${p4WsName}/..."""
              ]
            ]
        }
        // Do stuff
    }
}
 

When 2 instances of the job run on the same node, jenkins correctly allocates second workspace at C:\MyCustomWorkspace@2, syncs it etc.

After the concurrent runs have finished and I run the job again, there is a high chance that the job will fail during the p4sync step with an error message that's indicating that the p4 root is set incorrectly:
> Path 'C:\MyCustomWorkspace/...' is not under client's root 'C:\MyCustomWorkspace%402'.
More complete console log of the p4 operations:

P4 Task: reverting all pending and shelved revisions.
... p4 revert C:\MyCustomWorkspace/...
 +
... rm [abandoned files]
duration: (15ms)

P4 Task: cleaning workspace to match have list.
... p4 reconcile -w -f -e -d C:\MyCustomWorkspace/...
 +
duration: 0m 10s

P4 Task: syncing files at change: XXXXXXX
... p4 sync -q C:\MyCustomWorkspace/...@XXXXXXX
 +
P4 Task: attempt: 1
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // ws
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
[BFA] Scanning build for known causes...
[BFA] No failure causes found
[BFA] Done. 0s
ERROR: P4: Task Exception: com.perforce.p4java.exception.P4JavaException: com.perforce.p4java.exception.P4JavaException: hudson.AbortException: P4JAVA: Error(s):
Path 'C:\MyCustomWorkspace/...' is not under client's root 'C:\MyCustomWorkspace%402'.

Repro steps:
1. Create a pipeline job using the code I pasted above and pin it on a single node.
2. Change the //Do stuff to something like a 1 min sleep
3. Start 2 runs of the job, which should run on the same node, and verify that jenkins had created 2 different workspaces. You might want to wait until the p4 sync of the first run has finished before starting the second one.
4. After the 2 runs have finished, start a 3rd run
5. The 3rd run will probably fail.

Note: I tried creating my custom workspace inside my jenkins workspace root, and experienced the same issue.

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

vsiakka@gmail.com (JIRA)

unread,
Oct 4, 2019, 9:30:01 AM10/4/19
to jenkinsc...@googlegroups.com

kwirth@perforce.com (JIRA)

unread,
Oct 4, 2019, 10:08:02 AM10/4/19
to jenkinsc...@googlegroups.com

kwirth@perforce.com (JIRA)

unread,
Oct 4, 2019, 10:11:03 AM10/4/19
to jenkinsc...@googlegroups.com

kwirth@perforce.com (JIRA)

unread,
Oct 4, 2019, 10:19:06 AM10/4/19
to jenkinsc...@googlegroups.com
Karl Wirth commented on Bug JENKINS-59657
 
Re: Concurrent runs of a pipeline job that uses a custom workspaces cause p4sync to fail occassionally

Hi Vasiliki Siakka - Thanks for raisining this.

Sharing a workspace between jobs is not supported because the root will change for each when job when Jenkins decides to use a different location as you have found. We recommend that the workspace naming convention is 'jenkins-${NODE_NAME}${JOB_NAME}${EXECUTOR_NUMBER}':

  https://www.perforce.com/manuals/jenkins/Content/P4Jenkins/variable-expansion.html#Workspace_name

This ties that one workspace to the job, executor and node it's running on so in theory is unique.

Please also note that Jenkins has a bug in EXECUTOR_NUMBER (JENKINS-48882) that will sometime cause multiple concurrent jobs on the same node and different executors to fail. We have defensive code to try and work around the bug but sometimes we cannot avoid that failure. 

  

 

 

kwirth@perforce.com (JIRA)

unread,
Oct 4, 2019, 10:20:02 AM10/4/19
to jenkinsc...@googlegroups.com
Karl Wirth edited a comment on Bug JENKINS-59657
Hi [~vsiakka] - Thanks for raisining this.

Sharing a workspace between jobs is not supported because the root will change for each when job when Jenkins decides to use a different location as you have found. We recommend that the workspace naming convention is
' :
{code:java}
jenkins-${NODE_NAME} - ${JOB_NAME} - ${EXECUTOR_NUMBER} ': {code}

  [https://www.perforce.com/manuals/jenkins/Content/P4Jenkins/variable-expansion.html#Workspace_name]


This ties that one workspace to the job, executor and node it's running on so in theory is unique.

Please also note that Jenkins has a bug in EXECUTOR_NUMBER (JENKINS-48882) that will sometime cause multiple concurrent jobs on the same node and different executors to fail. We have defensive code to try and work around the bug but sometimes we cannot avoid that failure. 

  

 

 

vsiakka@gmail.com (JIRA)

unread,
Oct 8, 2019, 6:54:02 AM10/8/19
to jenkinsc...@googlegroups.com

Thank you for the reply Karl Wirth. Would you consider supporting this in the future, should I change this to a feature request?

kwirth@perforce.com (JIRA)

unread,
Nov 12, 2019, 5:38:02 AM11/12/19
to jenkinsc...@googlegroups.com

Hi Vasiliki Siakka

Sorry for the delay. I was going through old tickets and noticed I missed your update.

What part would you like to be an enhancement request? The fact that two build jobs cannot use the same workspace concurrently is fundamental to the way Perforce tracks what is in the workspace (it keeps a 'have' list). Also you wouldn't be able to trust the build results if potentially it had been built from different versions of the software.

Thanks in advance,

Karl

Reply all
Reply to author
Forward
0 new messages