pipeline : slave throws an exception on trying to create a tmp folder, despite creating the workspace and a subfolder just fine

[Jonathan Hodgson]

Hi,

I've been experiementing trying to get some information which might give a clue to the issues I'm having with my Windows slave.

It's rather confusing, since to start with it worked fine. and the errors aren't consistent (looking for past logs I see an error with unstashing which I don't seem to be getting now).

Anyway, I created a pipeline with this groovy script

@NonCPS def printException(err)

{

    message = err.getMessage()

    stacktrace = err.getStackTrace()

    stacktrace.each { stackElem ->

    message += "\n" + stackElem.toString()

    }

    print message

}

def batch(platform)

{

    try {

        if (isUnix())

        {

            sh script:"echo running shell script"

        }

        else

        {

            bat script:"echo running batch script"

        }

        echo "Batch file ran ok"

    }

    catch (Exception err)

    {

        echo "Failed to run batch file on $platform"

        printException(err)

    }    

}

def dirStep(platform)

{

    try {

        dir ("mySubDirectory")

        {

            echo "In subdirectory"

            writeFile file:"mySubFile.txt", text:"this is a file in $platform"

        }

    }

    catch (Exception err)

    {

        echo "Failed enter subdirectory on $platform"

        printException(err)

    }     

}

def writeFileStep(platform)

{

    try {

        writeFile file:"myFile.txt", text:"this is a file in $platform"

        echo "wrote File on $platform"

    }

    catch (Exception err)

    {

        echo "Failed to create file on $platform"

        printException(err)

    }   

}

def testUnstash(platform)

{

     try {

        unstash "myStash"

        echo "instashed on $platform"

    }

    catch (Exception err)

    {

        echo "Failed to unstash on $platform"

        printException(err)

    }      

}

def runTests(nodeID)

{

    node (nodeID)

    {

        batch(nodeID)

        writeFileStep(nodeID)

        dirStep(nodeID)

        testUnstash(nodeID)

    }

}

stage("testing")

{

    node ("master")

    {

        writeFile file:"toStash.txt",text:"Testing stashing and unstashing"

        stash name:"myStash", include:"toStash.txt"

    }

    runTests("master")

    runTests("XCode")

    runTests("VS2013")

}

And when I run it, everything seems to work fine, except for the batch file step on windows. which gives an error related to MkDir

Started by user Jonathan Hodgson
[Pipeline] stage
[Pipeline] { (testing)
[Pipeline] node
Running on master in /var/lib/jenkins/workspace/pipeline-test2
[Pipeline] {
[Pipeline] writeFile
[Pipeline] stash
Stashed 3 file(s)
[Pipeline] }
[Pipeline] // node
[Pipeline] node
Running on master in /var/lib/jenkins/workspace/pipeline-test2
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
[pipeline-test2] Running shell script
+ echo running shell script
running shell script
[Pipeline] echo
Batch file ran ok
[Pipeline] writeFile
[Pipeline] echo
wrote File on master
[Pipeline] dir
Running in /var/lib/jenkins/workspace/pipeline-test2/mySubDirectory
[Pipeline] {
[Pipeline] echo
In subdirectory
[Pipeline] writeFile
[Pipeline] }
[Pipeline] // dir
[Pipeline] unstash
[Pipeline] echo
instahsed on master
[Pipeline] }
[Pipeline] // node
[Pipeline] node
Running on iMac in /Users/jon/jenkins/workspace/pipeline-test2
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
[pipeline-test2] Running shell script
+ echo running shell script
running shell script
[Pipeline] echo
Batch file ran ok
[Pipeline] writeFile
[Pipeline] echo
wrote File on XCode
[Pipeline] dir
Running in /Users/jon/jenkins/workspace/pipeline-test2/mySubDirectory
[Pipeline] {
[Pipeline] echo
In subdirectory
[Pipeline] writeFile
[Pipeline] }
[Pipeline] // dir
[Pipeline] unstash
[Pipeline] echo
instahsed on XCode
[Pipeline] }
[Pipeline] // node
[Pipeline] node
Running on Asus K51 in C:\Jenkins\workspace\pipeline-test2
[Pipeline] {
[Pipeline] isUnix
[Pipeline] bat
[Pipeline] echo
Failed to run batch file on VS2013
[Pipeline] echo
Failed to mkdirs: C:\Jenkins\workspace\pipeline-test2@tmp\durable-97381953
hudson.FilePath.mkdirs(FilePath.java:1169)
org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.<init>(FileMonitoringTask.java:101)
org.jenkinsci.plugins.durabletask.WindowsBatchScript$BatchController.<init>(WindowsBatchScript.java:94)
org.jenkinsci.plugins.durabletask.WindowsBatchScript$BatchController.<init>(WindowsBatchScript.java:92)
org.jenkinsci.plugins.durabletask.WindowsBatchScript.doLaunch(WindowsBatchScript.java:60)
org.jenkinsci.plugins.durabletask.FileMonitoringTask.launchWithCookie(FileMonitoringTask.java:66)
org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:61)
org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:158)
org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:184)
org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:126)
org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:18)
WorkflowScript.batch(WorkflowScript:20)
WorkflowScript.runTests(WorkflowScript:77)
___cps.transform___(Native Method)
com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
sun.reflect.GeneratedMethodAccessor365.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55)
com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.item(CollectionLiteralBlock.java:45)
sun.reflect.GeneratedMethodAccessor385.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
com.cloudbees.groovy.cps.Next.step(Next.java:58)
com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:163)
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:324)
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:78)
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:236)
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:224)
org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:63)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
[Pipeline] writeFile
[Pipeline] echo
wrote File on VS2013
[Pipeline] dir
Running in C:\Jenkins\workspace\pipeline-test2\mySubDirectory
[Pipeline] {
[Pipeline] echo
In subdirectory
[Pipeline] writeFile
[Pipeline] }
[Pipeline] // dir
[Pipeline] unstash
[Pipeline] echo
instahsed on VS2013
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
Finished: SUCCESS

I tried deleteing the pipeline-test2 folder, and it gets recreated fine, as does its subdirectory, but the tmp folder breaks everything.

If anyone has any suggestions they would be gratefully received, since Jenkins is largely useless to me until I get this working again, and I've invested a LOT of time in getting this far.

[Jonathan Hodgson]

I tried a couple of experiments today.

Launching a cmd windows as an administrator and running the slave from there... no difference.

Manually created the pipeline-test2@tmp directory - build was successful

Deleted both pipeline-test2 and pipeline-test2@tmp and ran build - jenkins recreated pipeline-test2 but failed to create pipeline-test2@tmp

Tried running the slave on a different windows machine (running WIndows 10, it's my development laptop) - build was successful

This has me confused. The fact that it's different on two different machines makes me think permisssions.

But if permissions allow the creation of pipeline-test2 by the slave, they should also allow the creation of pipeline-test2@tmp right next to it surely?

[Jonathan Hodgson]

Ok, adding to the confusion...

I just tried changing the slave configuration to give a compleely new  root folder (C:\Jenkins2 rather than C:\Jenkins)

Ran the same same job

Failed to mkdirs: C:\Jenkins2\workspace\pipeline-test2
hudson.FilePath.mkdirs(FilePath.java:1169)
org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.<init>(FileMonitoringTask.java:99)

org.jenkinsci.plugins.durabletask.WindowsBatchScript$BatchController.<init>(WindowsBatchScript.java:94)
org.jenkinsci.plugins.durabletask.WindowsBatchScript$BatchController.<init>(WindowsBatchScript.java:92)
org.jenkinsci.plugins.durabletask.WindowsBatchScript.doLaunch(WindowsBatchScript.java:60)
org.jenkinsci.plugins.durabletask.FileMonitoringTask.launchWithCookie(FileMonitoringTask.java:66)
org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:61)

As you can see, now it's failing to create the job workspace folder, in fact looking on the slave I see it has failed to create the Jenkins2 filder so I just tried the following...

1) manually created C:\Jenkins2

- Job failed with same stack trace

2) Manually created C:\Jenkins2\workspace

- Job failed with different stackl trace

Failed to mkdirs: C:\Jenkins2\workspace\pipeline-test2@tmp\durable-e94accb0
hudson.FilePath.mkdirs(FilePath.java:1169)

org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.<init>(FileMonitoringTask.java:101)
org.jenkinsci.plugins.durabletask.WindowsBatchScript$BatchController.<init>(WindowsBatchScript.java:94)
org.jenkinsci.plugins.durabletask.WindowsBatchScript$BatchController.<init>(WindowsBatchScript.java:92)
org.jenkinsci.plugins.durabletask.WindowsBatchScript.doLaunch(WindowsBatchScript.java:60)
org.jenkinsci.plugins.durabletask.FileMonitoringTask.launchWithCookie(FileMonitoringTask.java:66)
org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:61)
org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:158)

And when I look, I see it has created pipeline-test2!

So, I can only conclude that the slave creates pipeline-test2 via a different mechanism to either C:\Jenkins2\workspace or pipeline-test2@tmp

[Daniel Beck]

Is this a file system that doesn't like the @ character?

Set the system property hudson.slaves.WorkspaceList to e.g. _ to use a different separator as described on https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/6233493b-0b4e-44df-98ff-df7ca47f8f46%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Jonathan Hodgson]

On Sunday, December 11, 2016 at 4:36:24 PM UTC, Daniel Beck wrote:

Is this a file system that doesn't like the @ character?

Seems unlikely, since

1) I created the folder manually without issue (and then things worked correctly)

2) Jenkins previously created those folders just fine, this issue seems to have popped up out of nowhere

3) Jenkins also failed to create C:\Jenkins2\worspace  (despite having previously created C:\Jenkins'workspace just fine)

4) If I run the slave on another windows system (albeit 10 rather than 7) it works as it should.

[Jonathan Hodgson]

Further to the above.

I just forced it to run multiple builds simultaneously, and it generates new folders for them as pipeline-test2@2 and pipelin-test2@3.

So the @ symbol is most certainly not the issue.